Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan <jo...@confluent.io.INVALID>]

Hey folks, just wanted to share another quick update with KIP-890.

Making the change to bump epoch after every transaction means that we no
longer need to call InitProducerId during the life of the producer to fence
requests from previous transactions. With KIP-890 part 2, the new client
will no longer call InitProducerId except on startup to fence a previous
instance.

With this change, some other calls to InitProducerId were inspected
including the call after receiving an InvalidPidMappingException. This
exception was changed to abortable as part of KIP-360: Improve reliability
of idempotent/transactional producer
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820>.
However, this change means that we can violate EOS guarantees. As an
example:

Consider an application that is copying data from one partition to another

   - Application instance A processes to offset 4
   - Application instance B comes up and fences application instance A
   - Application instance B processes to offset 5
   - Application instances A and B are idle for transaction.id.expiration.ms,
   transaction id expires on server
   - Application instance A attempts to process offset 5 (since in its
   view, that is next) -- if we recover from invalid pid mapping, we can
   duplicate this processing

Thus, INVALID_PID_MAPPING should be fatal to the producer and as part of
KIP-890 the new client in 4.0 will treat the exception as fatal.

This is consistent with KIP-1050: Consistent error handling for Transactions
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions>
where
errors that are fatal to the producer are in the "application recoverable"
category. This is a grouping that indicates to the client that the producer
needs to restart and recovery on the application side is necessary.
KIP-1050 is approved so we are consistent with that decision.

Let me know if there are any questions. I've updated the KIP to reflect
this change.

Justine

On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid> wrote:

> Sounds good, Justine. It would be useful to document that in the KIP.
>
> Thanks,
>
> Jun
>
> On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> <jo...@confluent.io.invalid>
> wrote:
>
> > The client will send the newest EndTxn request version if and only if
> both
> > the client and the server support kip-890 part 2.
> > We set the value in the record based on the EndTxn version.
> >
> > Justine
> >
> > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Hi, Justine,
> > >
> > > Thanks for the reply.
> > >
> > > 120. If the broker sends TV Y for the finalized version in
> > > ApiVersionResponse, but the client doesn't support Y, how does the
> broker
> > > know the TV that the client supports?
> > >
> > > Jun
> > >
> > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hey Jun,
> > > >
> > > > No worries. Work on this KIP has been blocked for a bit anyways --
> > > catching
> > > > up and rereading what I wrote :)
> > > >
> > > > 120. ClientTransactionProtocolVersion is the transaction version as
> > > defined
> > > > by the highest transaction version (feature version value) supported
> by
> > > the
> > > > client and the server. This works by the broker sending an
> > > > ApiVersionsRequest to the client with the finalized version. Assuming
> > > > kip-890 part 2 is enabled by transaction version Y, if this request
> > > > contains finalized version Y and the client has the logic to set this
> > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2 not
> > enable)
> > > > the client will send Y - 1, even though the client has the ability to
> > > > support kip-890 part 2.
> > > >
> > > > 121. You are correct that this is not needed. However, currently that
> > > field
> > > > is already being set in memory -- just not written to disk. I think
> it
> > is
> > > > ok to write it to disk though. Let me know if you think otherwise.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Hi, Justine,
> > > > >
> > > > > Thanks for the update and sorry for the late reply.
> > > > >
> > > > > 120. I am wondering what value is used for
> > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > EndTxnRequest?
> > > > >
> > > > > 121. Earlier, you made the change to set lastProducerId in PREPARE
> to
> > > > > indicate that the marker is written for the new client. With the
> new
> > > > > ClientTransactionProtocolVersion field, it seems this is no longer
> > > > > necessary.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > <jo...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi there -- another update!
> > > > > >
> > > > > > When looking into the implementation for the safe epoch bumps I
> > > > realized
> > > > > > that we are already populating previousProducerID in memory as
> part
> > > of
> > > > > > KIP-360.
> > > > > > If we are to start using flexible fields, it is better to always
> > use
> > > > this
> > > > > > information and have an explicit (tagged) field to indicate
> whether
> > > the
> > > > > > client supports KIP-890 part 2.
> > > > > >
> > > > > > I've included the extra field and how it is set in the KIP. I've
> > also
> > > > > > updated the KIP to explain that we will be setting the tagged
> > fields
> > > > when
> > > > > > they are available for all transitions.
> > > > > >
> > > > > > Finally, I added clearer text about the transaction protocol
> > versions
> > > > > > included with this KIP. 1 for flexible transaction state records
> > and
> > > 2
> > > > > for
> > > > > > KIP-890 part 2 enablement.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > jolshan@confluent.io
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey there -- small update to the KIP,
> > > > > > >
> > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > > TxnOffsetCommit
> > > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > AbortableTransactionException to match the pattern we had for
> > other
> > > > > > errors.
> > > > > > > I also mentioned bumping all 6 transactional APIs so we can
> > future
> > > > > > > proof/support the error on the client going forward. If a
> future
> > > > change
> > > > > > > wants to have an error scenario that requires us to abort the
> > > > > > transaction,
> > > > > > > we can rely on the 3.8+ clients to support it. We ran into
> issues
> > > > > finding
> > > > > > > good/generic error codes that older clients could support while
> > > > working
> > > > > > on
> > > > > > > this KIP, so this should help in the future.
> > > > > > >
> > > > > > > The features discussion is still ongoing in KIP-1022. Will
> update
> > > > again
> > > > > > > here when that concludes.
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > jolshan@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> I don't think AddPartitions is a good example since we
> currenly
> > > > don't
> > > > > > >> gate the version on TV or MV. (We only set a different flag
> > > > depending
> > > > > on
> > > > > > >> the TV)
> > > > > > >>
> > > > > > >> Even if we did want to gate it on TV, I think the idea is to
> > move
> > > > away
> > > > > > >> from MV gating inter broker protocols. Ideally we can get to a
> > > state
> > > > > > where
> > > > > > >> MV is just used for metadata changes.
> > > > > > >>
> > > > > > >> I think some of this discussion might fit more with the
> feature
> > > > > version
> > > > > > >> KIP, so I can try to open that up soon. Until we settle that,
> > some
> > > > of
> > > > > > the
> > > > > > >> work in KIP-890 is blocked.
> > > > > > >>
> > > > > > >> Justine
> > > > > > >>
> > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> <jun@confluent.io.invalid
> > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >>> Hi, Justine,
> > > > > > >>>
> > > > > > >>> Thanks for the reply.
> > > > > > >>>
> > > > > > >>> Since AddPartitions is an inter broker request, will its
> > version
> > > be
> > > > > > gated
> > > > > > >>> only by TV or other features like MV too? For example, if we
> > need
> > > > to
> > > > > > >>> change
> > > > > > >>> the protocol for AddPartitions for reasons other than txn
> > > > > verification
> > > > > > in
> > > > > > >>> the future, will the new version be gated by a new MV? If so,
> > > does
> > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > >>>
> > > > > > >>> Jun
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> > One TV gates the flexible feature version (no rpcs
> involved,
> > > only
> > > > > the
> > > > > > >>> > transactional records that should only be gated by TV)
> > > > > > >>> > Another TV gates the ability to turn on kip-890 part 2.
> This
> > > > would
> > > > > > >>> gate the
> > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > transactions),
> > > > > and
> > > > > > >>> > specifies a flag in AddPartitionsToTxn though the version
> is
> > > > > already
> > > > > > >>> used
> > > > > > >>> > without TV.
> > > > > > >>> >
> > > > > > >>> > I think the only concern is the Produce request and we
> could
> > > > > consider
> > > > > > >>> work
> > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > >>> >
> > > > > > >>> > Justine
> > > > > > >>> >
> > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > <jun@confluent.io.invalid
> > > > >
> > > > > > >>> wrote:
> > > > > > >>> >
> > > > > > >>> > > Hi, Justine,
> > > > > > >>> > >
> > > > > > >>> > > Which PRC/record protocols will TV guard? Going forward,
> > will
> > > > > those
> > > > > > >>> > > PRC/record protocols only be guarded by TV and not by
> other
> > > > > > features
> > > > > > >>> like
> > > > > > >>> > > MV?
> > > > > > >>> > >
> > > > > > >>> > > Thanks,
> > > > > > >>> > >
> > > > > > >>> > > Jun
> > > > > > >>> > >
> > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > >>> > > >
> > > > > > >>> > > wrote:
> > > > > > >>> > >
> > > > > > >>> > > > Hi Jun,
> > > > > > >>> > > >
> > > > > > >>> > > > Sorry I think I misunderstood your question or answered
> > > > > > >>> incorrectly.
> > > > > > >>> > The
> > > > > > >>> > > TV
> > > > > > >>> > > > version should ideally be fully independent from MV.
> > > > > > >>> > > > At least for the changes I proposed, TV should not
> affect
> > > MV
> > > > > and
> > > > > > MV
> > > > > > >>> > > should
> > > > > > >>> > > > not affect TV/
> > > > > > >>> > > >
> > > > > > >>> > > > I think if we downgrade TV, only that feature should
> > > > downgrade.
> > > > > > >>> > Likewise
> > > > > > >>> > > > the same with MV. The finalizedFeatures should just
> > reflect
> > > > the
> > > > > > >>> feature
> > > > > > >>> > > > downgrade we made.
> > > > > > >>> > > >
> > > > > > >>> > > > I also plan to write a new KIP for managing the disk
> > format
> > > > and
> > > > > > >>> upgrade
> > > > > > >>> > > > tool as we will need new flags to support these
> features.
> > > > That
> > > > > > >>> should
> > > > > > >>> > > help
> > > > > > >>> > > > clarify some things.
> > > > > > >>> > > >
> > > > > > >>> > > > Justine
> > > > > > >>> > > >
> > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > >>> > > wrote:
> > > > > > >>> > > >
> > > > > > >>> > > > > Hi, Justine,
> > > > > > >>> > > > >
> > > > > > >>> > > > > Thanks for the reply.
> > > > > > >>> > > > >
> > > > > > >>> > > > > So, if we downgrade TV, we could implicitly downgrade
> > > > another
> > > > > > >>> feature
> > > > > > >>> > > > (say
> > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would we
> > return
> > > > for
> > > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in
> that
> > > > case?
> > > > > > >>> > > > >
> > > > > > >>> > > > > Thanks,
> > > > > > >>> > > > >
> > > > > > >>> > > > > Jun
> > > > > > >>> > > > >
> > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > >>> > > > > >
> > > > > > >>> > > > > wrote:
> > > > > > >>> > > > >
> > > > > > >>> > > > > > Hey Jun,
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> (transaction
> > > > > > version)
> > > > > > >>> we
> > > > > > >>> > > will
> > > > > > >>> > > > > stop
> > > > > > >>> > > > > > using the add partitions to txn optimization and
> stop
> > > > > writing
> > > > > > >>> the
> > > > > > >>> > > > > flexible
> > > > > > >>> > > > > > feature version of the log.
> > > > > > >>> > > > > > In the compatibility section I included some
> > > explanations
> > > > > on
> > > > > > >>> how
> > > > > > >>> > this
> > > > > > >>> > > > is
> > > > > > >>> > > > > > done.
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > Thanks,
> > > > > > >>> > > > > > Justine
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > >>> > > > > wrote:
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > > Hi, Justine,
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > Thanks for the update.
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > If we ever downgrade the transaction feature, any
> > > > feature
> > > > > > >>> > depending
> > > > > > >>> > > > on
> > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > changes
> > > > > > made
> > > > > > >>> in
> > > > > > >>> > > > KIP-890
> > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > Jun
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > >>> > > > > > > wrote:
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > > Hey Jun,
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > I wanted to get back to you about your
> questions
> > > > about
> > > > > > >>> MV/IBP.
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > Looking at the options, I think it makes the
> most
> > > > sense
> > > > > > to
> > > > > > >>> > > create a
> > > > > > >>> > > > > > > > separate feature for transactions and use that
> to
> > > > > version
> > > > > > >>> gate
> > > > > > >>> > > the
> > > > > > >>> > > > > > > features
> > > > > > >>> > > > > > > > we need to version gate (flexible transactional
> > > state
> > > > > > >>> records
> > > > > > >>> > and
> > > > > > >>> > > > > using
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > new protocol)
> > > > > > >>> > > > > > > > I've updated the KIP to include this change.
> > > > Hopefully
> > > > > > >>> that's
> > > > > > >>> > > > > > everything
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > Justine
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine Olshan
> <
> > > > > > >>> > > > jolshan@confluent.io
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > > > wrote:
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > I will update the KIP with the prev field for
> > > > prepare
> > > > > > as
> > > > > > >>> > well.
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > PREPARE
> > > > > > >>> > > > > > > > > producerId: x
> > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty or z if
> y
> > > will
> > > > > > >>> overflow
> > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > COMPLETE
> > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > Thanks again,
> > > > > > >>> > > > > > > > > Justine
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > >>> > > > >
> > > > > > >>> > > > > > > > wrote:
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > >>> > > > > > > > >> (1) My point was that the coordinator could
> > fail
> > > > > right
> > > > > > >>> after
> > > > > > >>> > > > > writing
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> prepare marker. When the new txn coordinator
> > > > > generates
> > > > > > >>> the
> > > > > > >>> > > > > complete
> > > > > > >>> > > > > > > > marker
> > > > > > >>> > > > > > > > >> after the failover, it needs some field from
> > the
> > > > > > prepare
> > > > > > >>> > > marker
> > > > > > >>> > > > to
> > > > > > >>> > > > > > > > >> determine whether it's written by the new
> > > client.
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> (2) The changing of the behavior sounds good
> > to
> > > > me.
> > > > > We
> > > > > > >>> only
> > > > > > >>> > > want
> > > > > > >>> > > > > to
> > > > > > >>> > > > > > > > return
> > > > > > >>> > > > > > > > >> success if the prepare state is written by
> the
> > > new
> > > > > > >>> client.
> > > > > > >>> > So,
> > > > > > >>> > > > in
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> non-overflow case, it seems that we also
> need
> > > sth
> > > > in
> > > > > > the
> > > > > > >>> > > prepare
> > > > > > >>> > > > > > > marker
> > > > > > >>> > > > > > > > to
> > > > > > >>> > > > > > > > >> tell us whether it's written by the new
> > client.
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That sounds
> > > good
> > > > to
> > > > > > me.
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> Jun
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine
> > Olshan
> > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually have two
> > > > > questions.
> > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to
> > > write
> > > > > the
> > > > > > >>> > > previous
> > > > > > >>> > > > > > > produce
> > > > > > >>> > > > > > > > Id
> > > > > > >>> > > > > > > > >> > tagged field in the end maker so that we
> > know
> > > if
> > > > > the
> > > > > > >>> > marker
> > > > > > >>> > > is
> > > > > > >>> > > > > > from
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> new
> > > > > > >>> > > > > > > > >> > client. Since the end maker is derived
> from
> > > the
> > > > > > >>> prepare
> > > > > > >>> > > > marker,
> > > > > > >>> > > > > > > should
> > > > > > >>> > > > > > > > >> we
> > > > > > >>> > > > > > > > >> > write the previous produce Id in the
> prepare
> > > > > marker
> > > > > > >>> field
> > > > > > >>> > > too?
> > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > >>> > > > > > > > >> > we will lose this information when
> deriving
> > > the
> > > > > end
> > > > > > >>> > marker.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > The "previous" producer ID is in the
> normal
> > > > > producer
> > > > > > >>> ID
> > > > > > >>> > > field.
> > > > > > >>> > > > > So
> > > > > > >>> > > > > > > yes,
> > > > > > >>> > > > > > > > >> we
> > > > > > >>> > > > > > > > >> > need it in prepare and that was always the
> > > plan.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> > enumerate
> > > > the
> > > > > > >>> fields
> > > > > > >>> > and
> > > > > > >>> > > > add
> > > > > > >>> > > > > > > them
> > > > > > >>> > > > > > > > to
> > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch y.
> When
> > we
> > > > > > >>> overflow
> > > > > > >>> > > epoch
> > > > > > >>> > > > y
> > > > > > >>> > > > > we
> > > > > > >>> > > > > > > get
> > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field):
> > empty
> > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty or z
> > if y
> > > > > will
> > > > > > >>> > overflow
> > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field): x
> > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we overflowed
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry and
> > see
> > > > > epoch
> > > > > > -
> > > > > > >>> 1 +
> > > > > > >>> > ID
> > > > > > >>> > > > in
> > > > > > >>> > > > > > last
> > > > > > >>> > > > > > > > >> seen
> > > > > > >>> > > > > > > > >> > fields and are issuing the same command
> (ie
> > > > commit
> > > > > > not
> > > > > > >>> > > abort),
> > > > > > >>> > > > > we
> > > > > > >>> > > > > > > > return
> > > > > > >>> > > > > > > > >> > success. The logic before KIP-890 seems to
> > > > return
> > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > >>> > > > > > > > >> > in this case. Are we intentionally making
> > this
> > > > > > change?
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer if the
> > > epoch
> > > > is
> > > > > > >>> bumped
> > > > > > >>> > > and
> > > > > > >>> > > > we
> > > > > > >>> > > > > > > get a
> > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally
> > > adding
> > > > > this
> > > > > > >>> to
> > > > > > >>> > > > prevent
> > > > > > >>> > > > > > > > fencing.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > 112. We already merged the code that adds
> > the
> > > > > > >>> VerifyOnly
> > > > > > >>> > > field
> > > > > > >>> > > > > in
> > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an
> inter
> > > > > broker
> > > > > > >>> > request.
> > > > > > >>> > > > It
> > > > > > >>> > > > > > > seems
> > > > > > >>> > > > > > > > >> that
> > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do you
> > > know
> > > > > why?
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > We no longer need IBP for all interbroker
> > > > requests
> > > > > > as
> > > > > > >>> > > > > ApiVersions
> > > > > > >>> > > > > > > > should
> > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > >>> > > > > > > > >> > We also handle unsupported version errors
> > > > > correctly
> > > > > > >>> if we
> > > > > > >>> > > > > receive
> > > > > > >>> > > > > > > them
> > > > > > >>> > > > > > > > >> in
> > > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > Justine
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > > >> wrote:
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually have
> two
> > > > > > questions.
> > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to
> > > write
> > > > > the
> > > > > > >>> > > previous
> > > > > > >>> > > > > > > produce
> > > > > > >>> > > > > > > > Id
> > > > > > >>> > > > > > > > >> > > tagged field in the end maker so that we
> > > know
> > > > if
> > > > > > the
> > > > > > >>> > > marker
> > > > > > >>> > > > is
> > > > > > >>> > > > > > > from
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > new
> > > > > > >>> > > > > > > > >> > > client. Since the end maker is derived
> > from
> > > > the
> > > > > > >>> prepare
> > > > > > >>> > > > > marker,
> > > > > > >>> > > > > > > > >> should we
> > > > > > >>> > > > > > > > >> > > write the previous produce Id in the
> > prepare
> > > > > > marker
> > > > > > >>> > field
> > > > > > >>> > > > too?
> > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > >>> > > > > > > > >> > > we will lose this information when
> > deriving
> > > > the
> > > > > > end
> > > > > > >>> > > marker.
> > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we retry
> and
> > > see
> > > > > > epoch
> > > > > > >>> - 1
> > > > > > >>> > +
> > > > > > >>> > > ID
> > > > > > >>> > > > > in
> > > > > > >>> > > > > > > last
> > > > > > >>> > > > > > > > >> seen
> > > > > > >>> > > > > > > > >> > > fields and are issuing the same command
> > (ie
> > > > > commit
> > > > > > >>> not
> > > > > > >>> > > > abort),
> > > > > > >>> > > > > > we
> > > > > > >>> > > > > > > > >> return
> > > > > > >>> > > > > > > > >> > > success. The logic before KIP-890 seems
> to
> > > > > return
> > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > >>> > > > > > > > >> > > in this case. Are we intentionally
> making
> > > this
> > > > > > >>> change?
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > 112. We already merged the code that
> adds
> > > the
> > > > > > >>> VerifyOnly
> > > > > > >>> > > > field
> > > > > > >>> > > > > > in
> > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an
> > inter
> > > > > > broker
> > > > > > >>> > > request.
> > > > > > >>> > > > > It
> > > > > > >>> > > > > > > > seems
> > > > > > >>> > > > > > > > >> > that
> > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do
> you
> > > > know
> > > > > > why?
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > Jun
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM Justine
> > > Olshan
> > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to
> > "current
> > > > > > >>> producer id
> > > > > > >>> > > and
> > > > > > >>> > > > > > > epoch"
> > > > > > >>> > > > > > > > if
> > > > > > >>> > > > > > > > >> > that
> > > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am still
> > > > > finalizing
> > > > > > >>> the
> > > > > > >>> > > > design.
> > > > > > >>> > > > > I
> > > > > > >>> > > > > > > will
> > > > > > >>> > > > > > > > >> > update
> > > > > > >>> > > > > > > > >> > > > the KIP once I get the information
> > > > finalized.
> > > > > > >>> Sorry
> > > > > > >>> > for
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > delays.
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun
> Rao
> > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case, the
> > > > previous
> > > > > > ID
> > > > > > >>> is
> > > > > > >>> > the
> > > > > > >>> > > > > same
> > > > > > >>> > > > > > as
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > produce
> > > > > > >>> > > > > > > > >> > > > > ID for the complete marker too, but
> we
> > > set
> > > > > the
> > > > > > >>> > > previous
> > > > > > >>> > > > ID
> > > > > > >>> > > > > > in
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > complete
> > > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned that
> > this
> > > is
> > > > > to
> > > > > > >>> know
> > > > > > >>> > > that
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> marker is
> > > > > > >>> > > > > > > > >> > > > > written by the new client so that we
> > > could
> > > > > > >>> return
> > > > > > >>> > > > success
> > > > > > >>> > > > > on
> > > > > > >>> > > > > > > > >> retried
> > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was trying to
> > > > > understand
> > > > > > >>> why
> > > > > > >>> > > this
> > > > > > >>> > > > is
> > > > > > >>> > > > > > not
> > > > > > >>> > > > > > > > >> needed
> > > > > > >>> > > > > > > > >> > > for
> > > > > > >>> > > > > > > > >> > > > > the prepare marker since retry can
> > > happen
> > > > in
> > > > > > the
> > > > > > >>> > > prepare
> > > > > > >>> > > > > > state
> > > > > > >>> > > > > > > > >> too.
> > > > > > >>> > > > > > > > >> > Is
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > reason that in the prepare state, we
> > > > return
> > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > >>> > > > > > > > >> > > > instead
> > > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> > requests?
> > > > If
> > > > > > so,
> > > > > > >>> > should
> > > > > > >>> > > > we
> > > > > > >>> > > > > > > change
> > > > > > >>> > > > > > > > >> "If
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in last
> > > seen
> > > > > > >>> fields and
> > > > > > >>> > > are
> > > > > > >>> > > > > > > issuing
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > same
> > > > > > >>> > > > > > > > >> > > > > command (ie commit not abort) we can
> > > > return
> > > > > > >>> (with
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > epoch)"
> > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a
> delayed
> > > > > endMaker
> > > > > > >>> > request
> > > > > > >>> > > > > > carries
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > same
> > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This can
> > > > > > >>> commit/abort a
> > > > > > >>> > > > future
> > > > > > >>> > > > > > > > >> transaction
> > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we
> have
> > > > seen
> > > > > > >>> this in
> > > > > > >>> > > > > practice
> > > > > > >>> > > > > > > > >> though.
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be useful
> > to
> > > > make
> > > > > > it
> > > > > > >>> > clear
> > > > > > >>> > > > that
> > > > > > >>> > > > > > we
> > > > > > >>> > > > > > > > can
> > > > > > >>> > > > > > > > >> now
> > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field from the
> > log
> > > > > > >>> reliably.
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > >>> > > > > since
> > > > > > >>> > > > > > > > it's
> > > > > > >>> > > > > > > > >> > > called
> > > > > > >>> > > > > > > > >> > > > > across brokers and we are changing
> its
> > > > > schema.
> > > > > > >>> Are
> > > > > > >>> > you
> > > > > > >>> > > > > > saying
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > >> > don't
> > > > > > >>> > > > > > > > >> > > > need
> > > > > > >>> > > > > > > > >> > > > > it any more? I thought that we
> already
> > > > > > >>> implemented
> > > > > > >>> > the
> > > > > > >>> > > > > > server
> > > > > > >>> > > > > > > > side
> > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > >>> > > > > across
> > > > > > >>> > > > > > > > >> brokers.
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM
> > Justine
> > > > > Olshan
> > > > > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the previous ID
> > in
> > > > the
> > > > > > >>> Prepare
> > > > > > >>> > > > field
> > > > > > >>> > > > > > > since
> > > > > > >>> > > > > > > > we
> > > > > > >>> > > > > > > > >> > > don't
> > > > > > >>> > > > > > > > >> > > > > need
> > > > > > >>> > > > > > > > >> > > > > > it. It is the same producer ID as
> > the
> > > > main
> > > > > > >>> > producer
> > > > > > >>> > > ID
> > > > > > >>> > > > > > > field.
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to reread
> > your
> > > > > > message
> > > > > > >>> > about
> > > > > > >>> > > > > > delayed
> > > > > > >>> > > > > > > > >> > markers.
> > > > > > >>> > > > > > > > >> > > > If
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn marker
> > after
> > > > the
> > > > > > >>> > > transaction
> > > > > > >>> > > > is
> > > > > > >>> > > > > > > > already
> > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > transaction
> > > > > early
> > > > > > >>> > without
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > fixes
> > > > > > >>> > > > > > > > >> in
> > > > > > >>> > > > > > > > >> > > part
> > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was
> used
> > > in
> > > > a
> > > > > > >>> previous
> > > > > > >>> > > KIP
> > > > > > >>> > > > > and
> > > > > > >>> > > > > > > > never
> > > > > > >>> > > > > > > > >> > > > > > implemented it in the log -- only
> in
> > > > > memory
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> > protocol
> > > > are
> > > > > > you
> > > > > > >>> > > > referring
> > > > > > >>> > > > > > to?
> > > > > > >>> > > > > > > I
> > > > > > >>> > > > > > > > am
> > > > > > >>> > > > > > > > >> > > > working
> > > > > > >>> > > > > > > > >> > > > > on
> > > > > > >>> > > > > > > > >> > > > > > the design for the work to remove
> > the
> > > > > extra
> > > > > > >>> add
> > > > > > >>> > > > > partitions
> > > > > > >>> > > > > > > > call
> > > > > > >>> > > > > > > > >> > and I
> > > > > > >>> > > > > > > > >> > > > > right
> > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I have
> yet
> > to
> > > > > > update
> > > > > > >>> that
> > > > > > >>> > > > > section
> > > > > > >>> > > > > > > as
> > > > > > >>> > > > > > > > I
> > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > >>> > > > > > > > >> > > > > > the design so please stay tuned.
> Was
> > > > there
> > > > > > >>> > anything
> > > > > > >>> > > > else
> > > > > > >>> > > > > > you
> > > > > > >>> > > > > > > > >> > thought
> > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM
> Jun
> > > Rao
> > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > I don't see this create any
> issue.
> > > It
> > > > > just
> > > > > > >>> makes
> > > > > > >>> > > it
> > > > > > >>> > > > a
> > > > > > >>> > > > > > bit
> > > > > > >>> > > > > > > > >> hard to
> > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged produce id
> > > field
> > > > > > >>> means. We
> > > > > > >>> > > are
> > > > > > >>> > > > > > > > >> essentially
> > > > > > >>> > > > > > > > >> > > > trying
> > > > > > >>> > > > > > > > >> > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > combine two actions (completing
> a
> > > txn
> > > > > and
> > > > > > >>> init a
> > > > > > >>> > > new
> > > > > > >>> > > > > > > produce
> > > > > > >>> > > > > > > > >> Id)
> > > > > > >>> > > > > > > > >> > > in a
> > > > > > >>> > > > > > > > >> > > > > > > single record. But, this may be
> > fine
> > > > > too.
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > A few other follow up comments.
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that we
> > > only
> > > > > set
> > > > > > >>> the
> > > > > > >>> > > > previous
> > > > > > >>> > > > > > > > >> produce id
> > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > >>> > > > > > > > >> > > > > > > field in the complete marker,
> but
> > > not
> > > > in
> > > > > > the
> > > > > > >>> > > prepare
> > > > > > >>> > > > > > > marker,
> > > > > > >>> > > > > > > > >> is
> > > > > > >>> > > > > > > > >> > > that
> > > > > > >>> > > > > > > > >> > > > in
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always return
> > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > >>> > > > > > on
> > > > > > >>> > > > > > > > >> > retried
> > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your second
> point
> > is
> > > > > > >>> mentioned
> > > > > > >>> > in
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > KIP. I
> > > > > > >>> > > > > > > > >> can
> > > > > > >>> > > > > > > > >> > > add
> > > > > > >>> > > > > > > > >> > > > > more
> > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > >>> > > > > > > > >> > > > > > > > The delayed message case can
> > also
> > > > > > violate
> > > > > > >>> EOS
> > > > > > >>> > if
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > delayed
> > > > > > >>> > > > > > > > >> > > > message
> > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > addPartitionsToTxn
> > > > > > >>> > request
> > > > > > >>> > > > > comes
> > > > > > >>> > > > > > > in.
> > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> previous
> > > > > > (aborted)
> > > > > > >>> > > > > transaction
> > > > > > >>> > > > > > > > become
> > > > > > >>> > > > > > > > >> > part
> > > > > > >>> > > > > > > > >> > > > of
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > The above is the case when a
> > delayed
> > > > > > >>> message is
> > > > > > >>> > > > > appended
> > > > > > >>> > > > > > > to
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > data
> > > > > > >>> > > > > > > > >> > > > > > > partition. What I mentioned is a
> > > > > slightly
> > > > > > >>> > > different
> > > > > > >>> > > > > case
> > > > > > >>> > > > > > > > when
> > > > > > >>> > > > > > > > >> a
> > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > > transaction
> > > > > log
> > > > > > >>> > > partition.
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we move
> > past
> > > > the
> > > > > > >>> Prepare
> > > > > > >>> > > and
> > > > > > >>> > > > > > > > Complete
> > > > > > >>> > > > > > > > >> > > states,
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about
> lastSeen
> > > > > fields
> > > > > > >>> and
> > > > > > >>> > > clear
> > > > > > >>> > > > > > them,
> > > > > > >>> > > > > > > > just
> > > > > > >>> > > > > > > > >> > > handle
> > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is the
> > > > lastSeen
> > > > > > >>> field
> > > > > > >>> > the
> > > > > > >>> > > > > same
> > > > > > >>> > > > > > as
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > > >>> TransactionLogValue?
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> > > > > > inter-broker
> > > > > > >>> > > > protocol,
> > > > > > >>> > > > > > > should
> > > > > > >>> > > > > > > > >> we
> > > > > > >>> > > > > > > > >> > > bump
> > > > > > >>> > > > > > > > >> > > > up
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this feature
> > only
> > > > for
> > > > > > the
> > > > > > >>> > KRaft
> > > > > > >>> > > > > mode?
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at 11:13 AM
> > > > Justine
> > > > > > >>> Olshan
> > > > > > >>> > > > > > > > >> > > > > > > <jo...@confluent.io.invalid>
> > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> > > > convergence
> > > > > > on
> > > > > > >>> the
> > > > > > >>> > > > > design.
> > > > > > >>> > > > > > :)
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > While I understand it seems a
> > > little
> > > > > > >>> "weird".
> > > > > > >>> > > I'm
> > > > > > >>> > > > > not
> > > > > > >>> > > > > > > sure
> > > > > > >>> > > > > > > > >> what
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra record to
> > the
> > > > log.
> > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> > describe
> > > > > > >>> transactions
> > > > > > >>> > > > won't
> > > > > > >>> > > > > > > work
> > > > > > >>> > > > > > > > >> (ie,
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > >>> > > > > > > > >> > > > > > > > state is needed to calculate
> the
> > > > time
> > > > > > >>> since
> > > > > > >>> > the
> > > > > > >>> > > > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like this,
> > it
> > > is
> > > > > > >>> enough to
> > > > > > >>> > > > > > convince
> > > > > > >>> > > > > > > me
> > > > > > >>> > > > > > > > >> we
> > > > > > >>> > > > > > > > >> > > need
> > > > > > >>> > > > > > > > >> > > > > such
> > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems like it
> > > would
> > > > > be
> > > > > > >>> > > replacing
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > record
> > > > > > >>> > > > > > > > >> > > > written
> > > > > > >>> > > > > > > > >> > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this
> correct?
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at
> 5:14 PM
> > > Jun
> > > > > Rao
> > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the explanation.
> I
> > > > > > >>> understand the
> > > > > > >>> > > > > > intention
> > > > > > >>> > > > > > > > >> now.
> > > > > > >>> > > > > > > > >> > In
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the non-tagged
> > > field
> > > > to
> > > > > > >>> the old
> > > > > > >>> > > pid
> > > > > > >>> > > > > > (and
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> max
> > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that we
> > could
> > > > > > >>> correctly
> > > > > > >>> > > write
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> marker to
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> > > > downgrades.
> > > > > > When
> > > > > > >>> > > writing
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> complete
> > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has already
> > been
> > > > > > >>> written to
> > > > > > >>> > > the
> > > > > > >>> > > > > data
> > > > > > >>> > > > > > > > >> > partition.
> > > > > > >>> > > > > > > > >> > > > We
> > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the new
> > pid
> > > to
> > > > > > avoid
> > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to work.
> It's
> > > > just a
> > > > > > bit
> > > > > > >>> > > > > > inconsistent
> > > > > > >>> > > > > > > > for
> > > > > > >>> > > > > > > > >> a
> > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker to use
> > > > > different
> > > > > > >>> pids
> > > > > > >>> > in
> > > > > > >>> > > > > this
> > > > > > >>> > > > > > > > >> special
> > > > > > >>> > > > > > > > >> > > case.
> > > > > > >>> > > > > > > > >> > > > > If
> > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the complete
> > > > marker,
> > > > > it
> > > > > > >>> seems
> > > > > > >>> > > > that
> > > > > > >>> > > > > we
> > > > > > >>> > > > > > > > will
> > > > > > >>> > > > > > > > >> > never
> > > > > > >>> > > > > > > > >> > > > be
> > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > write the complete marker
> with
> > > the
> > > > > old
> > > > > > >>> pid.
> > > > > > >>> > > Not
> > > > > > >>> > > > > sure
> > > > > > >>> > > > > > > if
> > > > > > >>> > > > > > > > it
> > > > > > >>> > > > > > > > >> > > causes
> > > > > > >>> > > > > > > > >> > > > > any
> > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit
> > weird.
> > > > > > >>> Instead of
> > > > > > >>> > > > > writing
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> > > complete
> > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid, could we
> > write
> > > > two
> > > > > > >>> > records:
> > > > > > >>> > > a
> > > > > > >>> > > > > > > complete
> > > > > > >>> > > > > > > > >> > marker
> > > > > > >>> > > > > > > > >> > > > > with
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > > >>> TransactionLogValue
> > > > > > >>> > with
> > > > > > >>> > > > the
> > > > > > >>> > > > > > new
> > > > > > >>> > > > > > > > pid
> > > > > > >>> > > > > > > > >> > and
> > > > > > >>> > > > > > > > >> > > an
> > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make the two
> > > > records
> > > > > > in
> > > > > > >>> the
> > > > > > >>> > > same
> > > > > > >>> > > > > > batch
> > > > > > >>> > > > > > > > so
> > > > > > >>> > > > > > > > >> > that
> > > > > > >>> > > > > > > > >> > > > they
> > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > >>> > > > > > > > >> > > > > > > > > added to the log atomically.
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at
> > 5:40 PM
> > > > > > Justine
> > > > > > >>> > Olshan
> > > > > > >>> > > > > > > > >> > > > > > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker is
> > > > written,
> > > > > > >>> but the
> > > > > > >>> > > > > endTxn
> > > > > > >>> > > > > > > > >> response
> > > > > > >>> > > > > > > > >> > is
> > > > > > >>> > > > > > > > >> > > > not
> > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when the
> > server
> > > > > > >>> downgrades
> > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker is
> > > > > written,
> > > > > > >>> the
> > > > > > >>> > > endTxn
> > > > > > >>> > > > > > > > response
> > > > > > >>> > > > > > > > >> is
> > > > > > >>> > > > > > > > >> > > > > received
> > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > client when the server
> > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a
> little
> > > > > > confused.
> > > > > > >>> In
> > > > > > >>> > > both
> > > > > > >>> > > > of
> > > > > > >>> > > > > > > these
> > > > > > >>> > > > > > > > >> > cases,
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has the
> old
> > > > > producer
> > > > > > >>> ID.
> > > > > > >>> > We
> > > > > > >>> > > > > don't
> > > > > > >>> > > > > > > > write
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > new
> > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare marker's
> non
> > > > tagged
> > > > > > >>> fields.
> > > > > > >>> > > > > > > > >> > > > > > > > > > If the server downgrades
> > now,
> > > it
> > > > > > would
> > > > > > >>> > read
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > records
> > > > > > >>> > > > > > > > >> not
> > > > > > >>> > > > > > > > >> > > in
> > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the complete
> > marker
> > > > > will
> > > > > > >>> also
> > > > > > >>> > > have
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > old
> > > > > > >>> > > > > > > > >> > > producer
> > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the new
> > > producer
> > > > > ID,
> > > > > > >>> we
> > > > > > >>> > > would
> > > > > > >>> > > > > not
> > > > > > >>> > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since the
> > producer
> > > > id
> > > > > > >>> doesn't
> > > > > > >>> > > > match
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> > > transaction
> > > > > > >>> > > > > > > > >> > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be correct
> > on
> > > > the
> > > > > > data
> > > > > > >>> > > > > partition.)
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow case, I'd
> > > expect
> > > > > the
> > > > > > >>> > > following
> > > > > > >>> > > > > to
> > > > > > >>> > > > > > > > >> happen on
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry EndTxn
> > --
> > > it
> > > > > is
> > > > > > >>> the
> > > > > > >>> > same
> > > > > > >>> > > > > > > producer
> > > > > > >>> > > > > > > > ID
> > > > > > >>> > > > > > > > >> > and
> > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry
> > > EndTxn
> > > > > and
> > > > > > >>> use
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > >> producer id
> > > > > > >>> > > > > > > > >> > > > which
> > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > InvalidPidMappingException
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have special
> > > > handling
> > > > > > for
> > > > > > >>> > when
> > > > > > >>> > > a
> > > > > > >>> > > > > > server
> > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could get an
> > API
> > > > > > version
> > > > > > >>> > > request
> > > > > > >>> > > > > > > showing
> > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In that
> case,
> > > we
> > > > > can
> > > > > > >>> call
> > > > > > >>> > > > > > > > initProducerId
> > > > > > >>> > > > > > > > >> to
> > > > > > >>> > > > > > > > >> > > > abort
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> > overflow
> > > > > case,
> > > > > > >>> this
> > > > > > >>> > > > > correctly
> > > > > > >>> > > > > > > > gives
> > > > > > >>> > > > > > > > >> > us a
> > > > > > >>> > > > > > > > >> > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the corresponding
> > case
> > > > > would
> > > > > > >>> be
> > > > > > >>> > > where
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> *complete
> > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn is
> > not
> > > > > > >>> received by
> > > > > > >>> > > the
> > > > > > >>> > > > > > client
> > > > > > >>> > > > > > > > and
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > server
> > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would
> > result
> > > in
> > > > > the
> > > > > > >>> > > > transaction
> > > > > > >>> > > > > > > > >> > coordinator
> > > > > > >>> > > > > > > > >> > > > > having
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old one.
> If
> > > the
> > > > > > client
> > > > > > >>> > > > retries,
> > > > > > >>> > > > > it
> > > > > > >>> > > > > > > > will
> > > > > > >>> > > > > > > > >> > > receive
> > > > > > >>> > > > > > > > >> > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> InvalidPidMappingException.
> > > The
> > > > > > >>> > > InitProducerId
> > > > > > >>> > > > > > > > scenario
> > > > > > >>> > > > > > > > >> > above
> > > > > > >>> > > > > > > > >> > > > > would
> > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > compatibility
> > > > > story
> > > > > > is
> > > > > > >>> > meant
> > > > > > >>> > > > to
> > > > > > >>> > > > > > > > support
> > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > > transactional
> > > > > > >>> > > correctness.
> > > > > > >>> > > > > > > Keeping
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > > client
> > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not the
> > > > > priority.
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can
> also
> > > add
> > > > > text
> > > > > > >>> in
> > > > > > >>> > the
> > > > > > >>> > > > KIP
> > > > > > >>> > > > > > > about
> > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes some edge
> > > > cases.
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at
> > > 4:10 PM
> > > > > Jun
> > > > > > >>> Rao
> > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we don't
> need
> > > to
> > > > > > >>> optimize
> > > > > > >>> > for
> > > > > > >>> > > > > > fencing
> > > > > > >>> > > > > > > > >> during
> > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding consistency,
> > there
> > > > are
> > > > > > two
> > > > > > >>> > > > possible
> > > > > > >>> > > > > > > cases:
> > > > > > >>> > > > > > > > >> (1)
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but the
> endTxn
> > > > > > response
> > > > > > >>> is
> > > > > > >>> > not
> > > > > > >>> > > > > > > received
> > > > > > >>> > > > > > > > by
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > client
> > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)
> > the
> > > > > > prepare
> > > > > > >>> > marker
> > > > > > >>> > > > is
> > > > > > >>> > > > > > > > written,
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the
> client
> > > when
> > > > > the
> > > > > > >>> > server
> > > > > > >>> > > > > > > > downgrades.
> > > > > > >>> > > > > > > > >> In
> > > > > > >>> > > > > > > > >> > > (1),
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old
> produce
> > Id
> > > > and
> > > > > > in
> > > > > > >>> (2),
> > > > > > >>> > > the
> > > > > > >>> > > > > > > client
> > > > > > >>> > > > > > > > >> will
> > > > > > >>> > > > > > > > >> > > have
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> > downgrade
> > > > > right
> > > > > > >>> after
> > > > > > >>> > > the
> > > > > > >>> > > > > > > prepare
> > > > > > >>> > > > > > > > >> > marker,
> > > > > > >>> > > > > > > > >> > > > we
> > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both (1)
> and
> > > (2)
> > > > > > >>> since we
> > > > > > >>> > > can
> > > > > > >>> > > > > only
> > > > > > >>> > > > > > > put
> > > > > > >>> > > > > > > > >> one
> > > > > > >>> > > > > > > > >> > > > value
> > > > > > >>> > > > > > > > >> > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id
> field.
> > > > It's
> > > > > > >>> also not
> > > > > > >>> > > > clear
> > > > > > >>> > > > > > > which
> > > > > > >>> > > > > > > > >> case
> > > > > > >>> > > > > > > > >> > > is
> > > > > > >>> > > > > > > > >> > > > > more
> > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could probably be
> > > > > consistent
> > > > > > >>> with
> > > > > > >>> > > > either
> > > > > > >>> > > > > > > case.
> > > > > > >>> > > > > > > > >> By
> > > > > > >>> > > > > > > > >> > > > putting
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the
> prepare
> > > > > marker,
> > > > > > >>> we
> > > > > > >>> > are
> > > > > > >>> > > > > > > consistent
> > > > > > >>> > > > > > > > >> with
> > > > > > >>> > > > > > > > >> > > > case
> > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the slight
> > benefit
> > > > that
> > > > > > the
> > > > > > >>> > > produce
> > > > > > >>> > > > > > field
> > > > > > >>> > > > > > > > in
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker are
> > > consistent
> > > > > in
> > > > > > >>> the
> > > > > > >>> > > > overflow
> > > > > > >>> > > > > > > case.
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024 at
> > > > 3:11 PM
> > > > > > >>> Justine
> > > > > > >>> > > > Olshan
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> > describe,
> > > we
> > > > > > would
> > > > > > >>> > need
> > > > > > >>> > > to
> > > > > > >>> > > > > > have
> > > > > > >>> > > > > > > a
> > > > > > >>> > > > > > > > >> > delayed
> > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn,
> and a
> > > > > > >>> successful
> > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > >>> > > > > > > > >> > > and
> > > > > > >>> > > > > > > > >> > > > > then
> > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn request
> > go
> > > > > > through
> > > > > > >>> for
> > > > > > >>> > a
> > > > > > >>> > > > > given
> > > > > > >>> > > > > > > > >> producer.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure
> out
> > > if
> > > > it
> > > > > > is
> > > > > > >>> > > possible
> > > > > > >>> > > > > for
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> > client
> > > > > > >>> > > > > > > > >> > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous request is
> > > > delayed
> > > > > > >>> > somewhere.
> > > > > > >>> > > > But
> > > > > > >>> > > > > > > yes,
> > > > > > >>> > > > > > > > in
> > > > > > >>> > > > > > > > >> > this
> > > > > > >>> > > > > > > > >> > > > > case
> > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the
> client.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow
> > case.
> > > > In
> > > > > > the
> > > > > > >>> > > overflow
> > > > > > >>> > > > > > case,
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > producer
> > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are different on
> > the
> > > > > > marker
> > > > > > >>> and
> > > > > > >>> > on
> > > > > > >>> > > > the
> > > > > > >>> > > > > > new
> > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use the
> > max
> > > > > epoch
> > > > > > >>> but
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > >> > transaction
> > > > > > >>> > > > > > > > >> > > > > should
> > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and epoch 0
> > in
> > > > the
> > > > > > >>> > > > transactional
> > > > > > >>> > > > > > > state.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> downgrade
> > > > case,
> > > > > we
> > > > > > >>> want
> > > > > > >>> > to
> > > > > > >>> > > > see
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> > producer
> > > > > > >>> > > > > > > > >> > > > ID
> > > > > > >>> > > > > > > > >> > > > > as
> > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client will
> > have.
> > > > If
> > > > > we
> > > > > > >>> > > complete
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> commit,
> > > > > > >>> > > > > > > > >> > and
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded, we
> > need
> > > > the
> > > > > > new
> > > > > > >>> > > > producer
> > > > > > >>> > > > > ID
> > > > > > >>> > > > > > > in
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > state
> > > > > > >>> > > > > > > > >> > > > > so
> > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid producer ID
> > > > > mapping.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server downgrade
> > cases
> > > > are
> > > > > > >>> > > considering
> > > > > > >>> > > > > > > > >> > transactional
> > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing from
> > > previous
> > > > > > >>> behavior
> > > > > > >>> > --
> > > > > > >>> > > > and
> > > > > > >>> > > > > > are
> > > > > > >>> > > > > > > > not
> > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the safety
> > from
> > > > > > fencing
> > > > > > >>> > > retries
> > > > > > >>> > > > > (as
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > >> have
> > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> support).
> > > > > Perhaps
> > > > > > >>> this
> > > > > > >>> > is
> > > > > > >>> > > a
> > > > > > >>> > > > > > trade
> > > > > > >>> > > > > > > > off,
> > > > > > >>> > > > > > > > >> > but
> > > > > > >>> > > > > > > > >> > > I
> > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > downgrades,
> > > > it
> > > > > > will
> > > > > > >>> > have
> > > > > > >>> > > > > > > restarted
> > > > > > >>> > > > > > > > >> and
> > > > > > >>> > > > > > > > >> > it
> > > > > > >>> > > > > > > > >> > > is
> > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new producer ID
> > > too).
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 11:42 AM
> > > > > > >>> Jun
> > > > > > >>> > Rao
> > > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the
> reply.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the marker
> > is
> > > > > > written
> > > > > > >>> by
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > >> client, we
> > > > > > >>> > > > > > > > >> > > can
> > > > > > >>> > > > > > > > >> > > > > as
> > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> > guarantee
> > > > > that
> > > > > > >>> any
> > > > > > >>> > > EndTxn
> > > > > > >>> > > > > > > > requests
> > > > > > >>> > > > > > > > >> > with
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same
> producer
> > > and
> > > > > the
> > > > > > >>> same
> > > > > > >>> > > > > > > transaction.
> > > > > > >>> > > > > > > > >> Then
> > > > > > >>> > > > > > > > >> > > we
> > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced
> error
> > > but
> > > > > can
> > > > > > >>> handle
> > > > > > >>> > > > > > > gracefully
> > > > > > >>> > > > > > > > as
> > > > > > >>> > > > > > > > >> > > > > described
> > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed
> EndTnx
> > > > > request
> > > > > > is
> > > > > > >>> > > > > processed,
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> txn
> > > > > > >>> > > > > > > > >> > > state
> > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next txn. I
> > > guess
> > > > in
> > > > > > >>> this
> > > > > > >>> > case
> > > > > > >>> > > > we
> > > > > > >>> > > > > > > still
> > > > > > >>> > > > > > > > >> > return
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed request?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> > question
> > > > was
> > > > > > >>> > > inaccurate.
> > > > > > >>> > > > > What
> > > > > > >>> > > > > > > you
> > > > > > >>> > > > > > > > >> > > > described
> > > > > > >>> > > > > > > > >> > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > > compatibility I
> > > > > > >>> > mention
> > > > > > >>> > > > is
> > > > > > >>> > > > > > that
> > > > > > >>> > > > > > > > we
> > > > > > >>> > > > > > > > >> > keep
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the
> main
> > > > > > >>> (non-tagged)
> > > > > > >>> > > > fields
> > > > > > >>> > > > > as
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > >> did
> > > > > > >>> > > > > > > > >> > > > before
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If we
> > want
> > > > to
> > > > > do
> > > > > > >>> this,
> > > > > > >>> > > it
> > > > > > >>> > > > > > seems
> > > > > > >>> > > > > > > > >> that we
> > > > > > >>> > > > > > > > >> > > > > should
> > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current produce Id
> and
> > > max
> > > > > > >>> epoch in
> > > > > > >>> > > the
> > > > > > >>> > > > > > > existing
> > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch fields
> > for
> > > > > both
> > > > > > >>> the
> > > > > > >>> > > > prepare
> > > > > > >>> > > > > > and
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > complete
> > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can
> > happen
> > > > > after
> > > > > > >>> the
> > > > > > >>> > > > > complete
> > > > > > >>> > > > > > > > >> marker is
> > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described, the
> > > > > downgraded
> > > > > > >>> > > > coordinator
> > > > > > >>> > > > > > will
> > > > > > >>> > > > > > > > see
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the old
> > one.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 10:44 AM
> > > > > > >>> > > Justine
> > > > > > >>> > > > > > > Olshan
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > <jolshan@confluent.io.invalid
> > > > > > >
> > > > > > >>> > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> > > > > > description.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your
> > second
> > > > > point
> > > > > > is
> > > > > > >>> > > > mentioned
> > > > > > >>> > > > > > in
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > KIP.
> > > > > > >>> > > > > > > > >> > > I
> > > > > > >>> > > > > > > > >> > > > > can
> > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is
> > helpful.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed
> > message
> > > > case
> > > > > > can
> > > > > > >>> > also
> > > > > > >>> > > > > > violate
> > > > > > >>> > > > > > > > EOS
> > > > > > >>> > > > > > > > >> if
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after the
> > > next
> > > > > > >>> > > > > addPartitionsToTxn
> > > > > > >>> > > > > > > > >> request
> > > > > > >>> > > > > > > > >> > > > comes
> > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a message
> > > from a
> > > > > > >>> previous
> > > > > > >>> > > > > > (aborted)
> > > > > > >>> > > > > > > > >> > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next transaction.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is
> > > written
> > > > > by
> > > > > > >>> the
> > > > > > >>> > new
> > > > > > >>> > > > > > client,
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > >> can
> > > > > > >>> > > > > > > > >> > > as I
> > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> guarantee
> > > > that
> > > > > > any
> > > > > > >>> > EndTxn
> > > > > > >>> > > > > > > requests
> > > > > > >>> > > > > > > > >> with
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same producer
> > and
> > > > the
> > > > > > same
> > > > > > >>> > > > > > transaction.
> > > > > > >>> > > > > > > > >> Then we
> > > > > > >>> > > > > > > > >> > > > don't
> > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but
> can
> > > > > handle
> > > > > > >>> > > gracefully
> > > > > > >>> > > > > as
> > > > > > >>> > > > > > > > >> described
> > > > > > >>> > > > > > > > >> > > in
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a
> > > boolean
> > > > is
> > > > > > >>> useful
> > > > > > >>> > > > since
> > > > > > >>> > > > > it
> > > > > > >>> > > > > > > is
> > > > > > >>> > > > > > > > >> > > directly
> > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or lack
> of
> > > the
> > > > > > >>> tagged
> > > > > > >>> > > field
> > > > > > >>> > > > > > being
> > > > > > >>> > > > > > > > >> > written.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare
> > marker
> > > we
> > > > > > will
> > > > > > >>> have
> > > > > > >>> > > the
> > > > > > >>> > > > > > same
> > > > > > >>> > > > > > > > >> > producer
> > > > > > >>> > > > > > > > >> > > ID
> > > > > > >>> > > > > > > > >> > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the
> > Complete
> > > > > state
> > > > > > >>> we
> > > > > > >>> > may
> > > > > > >>> > > > not.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why
> the
> > > > > ongoing
> > > > > > >>> state
> > > > > > >>> > > > > matters
> > > > > > >>> > > > > > > for
> > > > > > >>> > > > > > > > >> this
> > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > >>> > > > > > > > >> > > > > It
> > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure what
> > you
> > > > are
> > > > > > >>> > referring
> > > > > > >>> > > to
> > > > > > >>> > > > > > about
> > > > > > >>> > > > > > > > >> > writing
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the prepare
> > > > marker.
> > > > > > >>> This is
> > > > > > >>> > > not
> > > > > > >>> > > > in
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> KIP.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow
> > case,
> > > we
> > > > > > >>> write the
> > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > >>> > > > > > > > >> > in
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we know
> > > what
> > > > we
> > > > > > >>> > assigned
> > > > > > >>> > > > when
> > > > > > >>> > > > > > we
> > > > > > >>> > > > > > > > >> reload
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we complete,
> we
> > > > > > >>> transition
> > > > > > >>> > this
> > > > > > >>> > > > ID
> > > > > > >>> > > > > to
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> main
> > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the previous
> > > > producer
> > > > > > ID
> > > > > > >>> > field
> > > > > > >>> > > > > filled
> > > > > > >>> > > > > > > in.
> > > > > > >>> > > > > > > > >> This
> > > > > > >>> > > > > > > > >> > > is
> > > > > > >>> > > > > > > > >> > > > so
> > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case
> the
> > > > > > operation
> > > > > > >>> > > > completed
> > > > > > >>> > > > > > > > >> > successfully
> > > > > > >>> > > > > > > > >> > > > and
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The
> > > downgrade
> > > > > > >>> > > compatibility
> > > > > > >>> > > > I
> > > > > > >>> > > > > > > > mention
> > > > > > >>> > > > > > > > >> is
> > > > > > >>> > > > > > > > >> > > that
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and
> > epoch
> > > in
> > > > > the
> > > > > > >>> main
> > > > > > >>> > > > > > > (non-tagged)
> > > > > > >>> > > > > > > > >> > fields
> > > > > > >>> > > > > > > > >> > > > as
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the server
> > > side.
> > > > > If
> > > > > > >>> the
> > > > > > >>> > > server
> > > > > > >>> > > > > > > > >> downgrades,
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > are
> > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses
> both
> > > the
> > > > > > >>> prepare
> > > > > > >>> > and
> > > > > > >>> > > > > > complete
> > > > > > >>> > > > > > > > >> state
> > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > >>> 10:21 AM
> > > > > > >>> > Jun
> > > > > > >>> > > > Rao
> > > > > > >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the
> > > reply.
> > > > > > Sorry
> > > > > > >>> for
> > > > > > >>> > > the
> > > > > > >>> > > > > > > delay. I
> > > > > > >>> > > > > > > > >> > have a
> > > > > > >>> > > > > > > > >> > > > few
> > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think the
> > > > > > motivation
> > > > > > >>> > > section
> > > > > > >>> > > > > > could
> > > > > > >>> > > > > > > be
> > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations
> listed
> > > by
> > > > > the
> > > > > > >>> KIP is
> > > > > > >>> > > > "This
> > > > > > >>> > > > > > can
> > > > > > >>> > > > > > > > >> happen
> > > > > > >>> > > > > > > > >> > > > when
> > > > > > >>> > > > > > > > >> > > > > a
> > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due
> to
> > > > > > networking
> > > > > > >>> > > issues
> > > > > > >>> > > > > or a
> > > > > > >>> > > > > > > > >> network
> > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and then
> > the
> > > > > > delayed
> > > > > > >>> > > message
> > > > > > >>> > > > > > > finally
> > > > > > >>> > > > > > > > >> > comes
> > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate.
> > > Without
> > > > > > >>> KIP-890,
> > > > > > >>> > > > > > currently,
> > > > > > >>> > > > > > > > if
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an
> ongoing
> > > > > > >>> transaction,
> > > > > > >>> > it
> > > > > > >>> > > > > > already
> > > > > > >>> > > > > > > > >> bumps
> > > > > > >>> > > > > > > > >> > up
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents
> the
> > > > > delayed
> > > > > > >>> > produce
> > > > > > >>> > > > > > message
> > > > > > >>> > > > > > > > >> from
> > > > > > >>> > > > > > > > >> > > being
> > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition. What
> > can
> > > > > cause
> > > > > > a
> > > > > > >>> > > hanging
> > > > > > >>> > > > > > > > >> transaction
> > > > > > >>> > > > > > > > >> > is
> > > > > > >>> > > > > > > > >> > > > that
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes
> (either
> > > > aborts
> > > > > > or
> > > > > > >>> > > > commits) a
> > > > > > >>> > > > > > > > >> > transaction
> > > > > > >>> > > > > > > > >> > > > > before
> > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack
> on
> > > > > messages
> > > > > > >>> > > published
> > > > > > >>> > > > > in
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> same
> > > > > > >>> > > > > > > > >> > > txn.
> > > > > > >>> > > > > > > > >> > > > > In
> > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for the
> > > > delayed
> > > > > > >>> message
> > > > > > >>> > > to
> > > > > > >>> > > > be
> > > > > > >>> > > > > > > > >> appended
> > > > > > >>> > > > > > > > >> > to
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker, causing
> a
> > > > > > >>> transaction to
> > > > > > >>> > > > hang.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar issue
> > (not
> > > > > > >>> mentioned
> > > > > > >>> > in
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > >> motivation)
> > > > > > >>> > > > > > > > >> > > > could
> > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> > > > > > coordinator's
> > > > > > >>> log.
> > > > > > >>> > > For
> > > > > > >>> > > > > > > > example,
> > > > > > >>> > > > > > > > >> > it's
> > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest to
> > be
> > > > > > delayed
> > > > > > >>> on
> > > > > > >>> > the
> > > > > > >>> > > > > > > > >> coordinator.
> > > > > > >>> > > > > > > > >> > By
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest is
> > > > > > processed,
> > > > > > >>> it's
> > > > > > >>> > > > > > possible
> > > > > > >>> > > > > > > > that
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed and a
> > new
> > > > txn
> > > > > > has
> > > > > > >>> > > started.
> > > > > > >>> > > > > > > > >> Currently,
> > > > > > >>> > > > > > > > >> > > since
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on every
> > txn,
> > > > the
> > > > > > >>> delayed
> > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > >>> > > > > > > > >> > will
> > > > > > >>> > > > > > > > >> > > > add
> > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker
> > (and
> > > > > > >>> eventually a
> > > > > > >>> > > > > > complete
> > > > > > >>> > > > > > > > >> marker)
> > > > > > >>> > > > > > > > >> > > to
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause the
> > > > > > transaction
> > > > > > >>> to
> > > > > > >>> > > hang,
> > > > > > >>> > > > > but
> > > > > > >>> > > > > > > it
> > > > > > >>> > > > > > > > >> will
> > > > > > >>> > > > > > > > >> > > > break
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal in
> > this
> > > > KIP
> > > > > > >>> will
> > > > > > >>> > > > address
> > > > > > >>> > > > > > this
> > > > > > >>> > > > > > > > >> issue
> > > > > > >>> > > > > > > > >> > > too.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101. "However, I
> > was
> > > > > > >>> writing it
> > > > > > >>> > so
> > > > > > >>> > > > > that
> > > > > > >>> > > > > > we
> > > > > > >>> > > > > > > > can
> > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients
> where
> > we
> > > > > don't
> > > > > > >>> have
> > > > > > >>> > > the
> > > > > > >>> > > > > > > ability
> > > > > > >>> > > > > > > > do
> > > > > > >>> > > > > > > > >> > this
> > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that
> can.
> > > (Old
> > > > > > >>> clients
> > > > > > >>> > > don't
> > > > > > >>> > > > > > bump
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > epoch
> > > > > > >>> > > > > > > > >> > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure the
> > > write
> > > > > > >>> belongs
> > > > > > >>> > to
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > given
> > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> > wondering
> > > > why
> > > > > > we
> > > > > > >>> need
> > > > > > >>> > > to
> > > > > > >>> > > > > > > > >> distinguish
> > > > > > >>> > > > > > > > >> > > > whether
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by the
> old
> > > and
> > > > > the
> > > > > > >>> new
> > > > > > >>> > > > client.
> > > > > > >>> > > > > > > Could
> > > > > > >>> > > > > > > > >> you
> > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently if
> we
> > > know
> > > > > the
> > > > > > >>> > marker
> > > > > > >>> > > is
> > > > > > >>> > > > > > > written
> > > > > > >>> > > > > > > > >> by
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do
> > need
> > > a
> > > > > way
> > > > > > to
> > > > > > >>> > > > > distinguish
> > > > > > >>> > > > > > > > >> whether
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and the
> > new
> > > > > > client.
> > > > > > >>> > Would
> > > > > > >>> > > it
> > > > > > >>> > > > > be
> > > > > > >>> > > > > > > > >> simpler
> > > > > > >>> > > > > > > > >> > to
> > > > > > >>> > > > > > > > >> > > > just
> > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field instead of
> > > > > > indirectly
> > > > > > >>> > > through
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> previous
> > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not
> > clear
> > > > to
> > > > > me
> > > > > > >>> why
> > > > > > >>> > we
> > > > > > >>> > > > only
> > > > > > >>> > > > > > add
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete
> > marker,
> > > > but
> > > > > > >>> not in
> > > > > > >>> > > the
> > > > > > >>> > > > > > > prepare
> > > > > > >>> > > > > > > > >> > marker.
> > > > > > >>> > > > > > > > >> > > > If
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a marker
> > is
> > > > > > written
> > > > > > >>> by
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > client
> > > > > > >>> > > > > > > > >> or
> > > > > > >>> > > > > > > > >> > > not,
> > > > > > >>> > > > > > > > >> > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > > consistently
> > > > > > for
> > > > > > >>> all
> > > > > > >>> > > > > markers.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What about
> > the
> > > > > > >>> > > > > TransactionLogValue
> > > > > > >>> > > > > > > > >> record
> > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should we
> > > also
> > > > > > >>> > distinguish
> > > > > > >>> > > > > > whether
> > > > > > >>> > > > > > > > it's
> > > > > > >>> > > > > > > > >> > > > written
> > > > > > >>> > > > > > > > >> > > > > by
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the
> > overflow
> > > > > case,
> > > > > > >>> it's
> > > > > > >>> > > > still
> > > > > > >>> > > > > > not
> > > > > > >>> > > > > > > > >> clear
> > > > > > >>> > > > > > > > >> > to
> > > > > > >>> > > > > > > > >> > > me
> > > > > > >>> > > > > > > > >> > > > > why
> > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous produce
> > Id
> > > in
> > > > > the
> > > > > > >>> > prepare
> > > > > > >>> > > > > > marker
> > > > > > >>> > > > > > > > >> while
> > > > > > >>> > > > > > > > >> > > > writing
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the complete
> > > > marker.
> > > > > > You
> > > > > > >>> > > > mentioned
> > > > > > >>> > > > > > that
> > > > > > >>> > > > > > > > >> it's
> > > > > > >>> > > > > > > > >> > for
> > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> downgrade
> > > > with
> > > > > > >>> either
> > > > > > >>> > the
> > > > > > >>> > > > > > prepare
> > > > > > >>> > > > > > > > >> marker
> > > > > > >>> > > > > > > > >> > > or
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either case,
> > the
> > > > > > >>> downgraded
> > > > > > >>> > > > > > coordinator
> > > > > > >>> > > > > > > > >> should
> > > > > > >>> > > > > > > > >> > > see
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the
> > > previous
> > > > > > >>> produce
> > > > > > >>> > > Id),
> > > > > > >>> > > > > > right?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec 20,
> > 2023
> > > > at
> > > > > > >>> 6:00 PM
> > > > > > >>> > > > > Justine
> > > > > > >>> > > > > > > > Olshan
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for
> > taking
> > > a
> > > > > look
> > > > > > >>> at
> > > > > > >>> > the
> > > > > > >>> > > > KIP
> > > > > > >>> > > > > > > again.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the
> > epoch
> > > > > > >>> overflow
> > > > > > >>> > > case,
> > > > > > >>> > > > > only
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> > marker
> > > > > > >>> > > > > > > > >> > > > > will
> > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the
> > behavior
> > > > of
> > > > > > the
> > > > > > >>> rest
> > > > > > >>> > > of
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> markers
> > > > > > >>> > > > > > > > >> > > where
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the
> > > > > transaction
> > > > > > >>> > > records +
> > > > > > >>> > > > > 1.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are
> > > correct
> > > > > > that
> > > > > > >>> we
> > > > > > >>> > > don't
> > > > > > >>> > > > > > need
> > > > > > >>> > > > > > > to
> > > > > > >>> > > > > > > > >> > write
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same.
> > > > However,
> > > > > I
> > > > > > >>> was
> > > > > > >>> > > > writing
> > > > > > >>> > > > > it
> > > > > > >>> > > > > > > so
> > > > > > >>> > > > > > > > >> that
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > can
> > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients
> > where
> > > we
> > > > > > don't
> > > > > > >>> > have
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > ability
> > > > > > >>> > > > > > > > >> do
> > > > > > >>> > > > > > > > >> > > this
> > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that
> > can.
> > > > (Old
> > > > > > >>> clients
> > > > > > >>> > > > don't
> > > > > > >>> > > > > > > bump
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > epoch
> > > > > > >>> > > > > > > > >> > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure
> the
> > > > write
> > > > > > >>> belongs
> > > > > > >>> > > to
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > given
> > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn request
> > > from
> > > > a
> > > > > > new
> > > > > > >>> > > client,
> > > > > > >>> > > > we
> > > > > > >>> > > > > > > will
> > > > > > >>> > > > > > > > >> fill
> > > > > > >>> > > > > > > > >> > > this
> > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any
> EndTxn
> > > > > requests
> > > > > > >>> with
> > > > > > >>> > > the
> > > > > > >>> > > > > same
> > > > > > >>> > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > are
> > > > > > >>> > > > > > > > >> > > > from
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > > transaction.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In
> prepare
> > > > phase,
> > > > > > we
> > > > > > >>> have
> > > > > > >>> > > the
> > > > > > >>> > > > > > same
> > > > > > >>> > > > > > > > >> > producer
> > > > > > >>> > > > > > > > >> > > ID
> > > > > > >>> > > > > > > > >> > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is the
> > > > > producer
> > > > > > >>> ID and
> > > > > > >>> > > > epoch
> > > > > > >>> > > > > > > that
> > > > > > >>> > > > > > > > >> are
> > > > > > >>> > > > > > > > >> > on
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we stay
> > the
> > > > > same
> > > > > > >>> unless
> > > > > > >>> > > it
> > > > > > >>> > > > is
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> > overflow
> > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the
> producer
> > > ID
> > > > to
> > > > > > >>> the new
> > > > > > >>> > > one
> > > > > > >>> > > > > we
> > > > > > >>> > > > > > > > >> generated
> > > > > > >>> > > > > > > > >> > > and
> > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete. This
> > is
> > > > for
> > > > > > >>> > downgrade
> > > > > > >>> > > > > > > > >> compatibility.
> > > > > > >>> > > > > > > > >> > > The
> > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety guards
> > for
> > > > > > retries
> > > > > > >>> and
> > > > > > >>> > > > > > failovers.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare
> phase
> > > for
> > > > > > epoch
> > > > > > >>> > > > overflow
> > > > > > >>> > > > > > case
> > > > > > >>> > > > > > > > >> only
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > > store
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is
> for
> > > the
> > > > > case
> > > > > > >>> where
> > > > > > >>> > > we
> > > > > > >>> > > > > > reload
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare state.
> > > Once
> > > > > the
> > > > > > >>> > > > transaction
> > > > > > >>> > > > > is
> > > > > > >>> > > > > > > > >> > committed,
> > > > > > >>> > > > > > > > >> > > > we
> > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the client
> > > > already
> > > > > is
> > > > > > >>> > using.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit
> phase,
> > > we
> > > > > > store
> > > > > > >>> the
> > > > > > >>> > > > > previous
> > > > > > >>> > > > > > > > >> producer
> > > > > > >>> > > > > > > > >> > > ID
> > > > > > >>> > > > > > > > >> > > > in
> > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it is
> > > easier
> > > > > to
> > > > > > >>> think
> > > > > > >>> > of
> > > > > > >>> > > > it
> > > > > > >>> > > > > as
> > > > > > >>> > > > > > > > just
> > > > > > >>> > > > > > > > >> how
> > > > > > >>> > > > > > > > >> > > we
> > > > > > >>> > > > > > > > >> > > > > were
> > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch
> > before,
> > > > with
> > > > > > >>> some
> > > > > > >>> > > extra
> > > > > > >>> > > > > > > > bookeeping
> > > > > > >>> > > > > > > > >> > and
> > > > > > >>> > > > > > > > >> > > > edge
> > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged fields.
> > We
> > > > have
> > > > > > to
> > > > > > >>> do
> > > > > > >>> > it
> > > > > > >>> > > > this
> > > > > > >>> > > > > > way
> > > > > > >>> > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next
> > producer
> > > > ID
> > > > > is
> > > > > > >>> for
> > > > > > >>> > > > prepare
> > > > > > >>> > > > > > > > status
> > > > > > >>> > > > > > > > >> and
> > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> complete.
> > > The
> > > > > > >>> reason why
> > > > > > >>> > > we
> > > > > > >>> > > > > need
> > > > > > >>> > > > > > > two
> > > > > > >>> > > > > > > > >> > > separate
> > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > > > > compatibility.
> > > > > > >>> We
> > > > > > >>> > need
> > > > > > >>> > > > to
> > > > > > >>> > > > > > keep
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > same
> > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> field
> > > in
> > > > > case
> > > > > > >>> we
> > > > > > >>> > > > > downgrade.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set
> the
> > > > fields
> > > > > > as
> > > > > > >>> we
> > > > > > >>> > do
> > > > > > >>> > > in
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> > > > > > compatibility
> > > > > > >>> --
> > > > > > >>> > if
> > > > > > >>> > > we
> > > > > > >>> > > > > > > > >> downgrade,
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > will
> > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > fields)
> > > > It
> > > > > > >>> will be
> > > > > > >>> > > the
> > > > > > >>> > > > > old
> > > > > > >>> > > > > > > > >> producer
> > > > > > >>> > > > > > > > >> > ID
> > > > > > >>> > > > > > > > >> > > > and
> > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this
> helps.
> > > Let
> > > > > me
> > > > > > >>> know
> > > > > > >>> > if
> > > > > > >>> > > > you
> > > > > > >>> > > > > > have
> > > > > > >>> > > > > > > > >> > further
> > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec
> 20,
> > > 2023
> > > > > at
> > > > > > >>> > 3:33 PM
> > > > > > >>> > > > Jun
> > > > > > >>> > > > > > Rao
> > > > > > >>> > > > > > > > >> > > > > > > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems
> that
> > > you
> > > > > have
> > > > > > >>> made
> > > > > > >>> > > some
> > > > > > >>> > > > > > > changes
> > > > > > >>> > > > > > > > >> to
> > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > particular,
> we
> > > are
> > > > > > >>> changing
> > > > > > >>> > > the
> > > > > > >>> > > > > > format
> > > > > > >>> > > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments
> > related
> > > > to
> > > > > > >>> that.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just to
> > be
> > > > > clear.
> > > > > > >>> The
> > > > > > >>> > > > > overflow
> > > > > > >>> > > > > > > case
> > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated)
> is
> > > when
> > > > > the
> > > > > > >>> > current
> > > > > > >>> > > > > epoch
> > > > > > >>> > > > > > > > >> equals
> > > > > > >>> > > > > > > > >> > to
> > > > > > >>> > > > > > > > >> > > > max
> > > > > > >>> > > > > > > > >> > > > > -
> > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For the
> > > "not
> > > > > > epoch
> > > > > > >>> > > > overflow"
> > > > > > >>> > > > > > > case,
> > > > > > >>> > > > > > > > we
> > > > > > >>> > > > > > > > >> > > write
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged field
> > in
> > > > the
> > > > > > >>> complete
> > > > > > >>> > > > > phase.
> > > > > > >>> > > > > > Do
> > > > > > >>> > > > > > > > we
> > > > > > >>> > > > > > > > >> > need
> > > > > > >>> > > > > > > > >> > > to
> > > > > > >>> > > > > > > > >> > > > > do
> > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't
> change
> > > in
> > > > > this
> > > > > > >>> case?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It
> seems
> > > that
> > > > > the
> > > > > > >>> > meaning
> > > > > > >>> > > > for
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > TransactionLogValue
> > > > > > >>> changes
> > > > > > >>> > > > > > depending
> > > > > > >>> > > > > > > on
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > TransactionStatus
> > > > > > is
> > > > > > >>> > > > ongoing,
> > > > > > >>> > > > > > they
> > > > > > >>> > > > > > > > >> > > represent
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the current
> > > > > > >>> ProducerEpoch.
> > > > > > >>> > > When
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > >>> > > they
> > > > > > >>> > > > > > > > represent
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > ProducerEpoch.
> > > > > > >>> When the
> > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > >>> > > > > > > > >> > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further
> depend
> > > on
> > > > > > >>> whether
> > > > > > >>> > the
> > > > > > >>> > > > > epoch
> > > > > > >>> > > > > > > > >> overflows
> > > > > > >>> > > > > > > > >> > > or
> > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow,
> they
> > > > > > represent
> > > > > > >>> > the
> > > > > > >>> > > > > > current
> > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > >>> > > > > > > > >> > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ProducerEpoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > > Otherwise,
> > > > > they
> > > > > > >>> > > represent
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > newly
> > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> ProducerEpoch
> > of
> > > > 0.
> > > > > Is
> > > > > > >>> that
> > > > > > >>> > > > right?
> > > > > > >>> > > > > > > This
> > > > > > >>> > > > > > > > >> seems
> > > > > > >>> > > > > > > > >> > > not
> > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide
> > some
> > > > > > examples
> > > > > > >>> > like
> > > > > > >>> > > > what
> > > > > > >>> > > > > > > Artem
> > > > > > >>> > > > > > > > >> has
> > > > > > >>> > > > > > > > >> > > done
> > > > > > >>> > > > > > > > >> > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > considered a
> > > > simpler
> > > > > > >>> design
> > > > > > >>> > > > where
> > > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same
> value
> > > > (e.g.
> > > > > > >>> for the
> > > > > > >>> > > > > current
> > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > TransactionStatus
> > > > > and
> > > > > > >>> epoch
> > > > > > >>> > > > > > overflow?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's
> not
> > > > clear
> > > > > to
> > > > > > >>> me
> > > > > > >>> > why
> > > > > > >>> > > we
> > > > > > >>> > > > > > need
> > > > > > >>> > > > > > > 3
> > > > > > >>> > > > > > > > >> > fields:
> > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> PrevProducerId,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > NextProducerId.
> > > > > Could
> > > > > > we
> > > > > > >>> > just
> > > > > > >>> > > > have
> > > > > > >>> > > > > > > > >> ProducerId
> > > > > > >>> > > > > > > > >> > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > >>> > > > > if
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> > > producer
> > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the
> > > > > producerId
> > > > > > >>> and
> > > > > > >>> > the
> > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > >
> > > > > > >>> > > > >
> > > > > > >>> > > >
> > > > > > >>> > >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan <jo...@confluent.io.INVALID>]

Thanks Calvin,

The changes make sense to me. Can we add the defaults to the KIP page as
well? Thanks!

Justine

On Tue, Feb 11, 2025 at 3:27 PM Calvin Liu <ca...@confluent.io.invalid>
wrote:

> Hi Jun,
> Thanks for the feedback, will update the name as suggested.
> The default values are
> add.partitions.to.txn.retry.backoff.ms=20
> -- This is because the client side AddPartitionsToTxn retry backoff is
> 20ms.
> add.partitions.to.txn.retry.backoff.max.ms=100
> -- We want this value to be small to avoid client side timeout and retry,
> but also long enough for the markers to propagate.
>
>
> On Tue, Feb 11, 2025 at 2:36 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Hi, Calvin,
> >
> > Thanks for the update. The change looks good to me. Some minor comments.
> >
> > For retry timeout, our convention is to use retry.backoff.max.ms and
> > retry.backoff.ms. So, to be consistent, the new configurations should be
> > add.partitions.to.txn.retry.backoff.max.ms and
> > add.partitions.to.txn.retry.backoff.ms.
> >
> > Also, could you specify the defaults for these two values?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> >
> > On Mon, Feb 10, 2025 at 2:45 PM Calvin Liu <ca...@confluent.io.invalid>
> > wrote:
> >
> > > Hey team,
> > > In order to address the performance issue in the
> > > https://issues.apache.org/jira/browse/KAFKA-18654. We are making a
> > change
> > > to retry the AddPartitionToTxn requests on the server side when hitting
> > the
> > > CONCURRENT_TRANSACTION error. This error is common after the
> transaction
> > is
> > > committed but before the markers are fully propagated to all the
> involved
> > > partitions. With the server side retry, the client can avoid the
> frequent
> > > and expensive retry for the produce request. In order to better control
> > the
> > > retry behavior, we proposed to add 2 following configs.
> > >
> > >    - *add.partitions.to.txn.max.timeout.ms
> > >    <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum
> > > retry
> > >    timeout when the server attempts to add the partition to the
> > > transaction.
> > >    - *add.partitions.to.txn.retry.backoff.ms
> > >    <http://add.partitions.to.txn.retry.backoff.ms>* defines how
> > frequently
> > >    the server will retry the AddPartitionToTxn.
> > >
> > > Thanks!
> > >
> > > On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hey folks, just wanted to share another quick update with KIP-890.
> > > >
> > > > Making the change to bump epoch after every transaction means that we
> > no
> > > > longer need to call InitProducerId during the life of the producer to
> > > fence
> > > > requests from previous transactions. With KIP-890 part 2, the new
> > client
> > > > will no longer call InitProducerId except on startup to fence a
> > previous
> > > > instance.
> > > >
> > > > With this change, some other calls to InitProducerId were inspected
> > > > including the call after receiving an InvalidPidMappingException.
> This
> > > > exception was changed to abortable as part of KIP-360: Improve
> > > reliability
> > > > of idempotent/transactional producer
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> > > > >.
> > > > However, this change means that we can violate EOS guarantees. As an
> > > > example:
> > > >
> > > > Consider an application that is copying data from one partition to
> > > another
> > > >
> > > >    - Application instance A processes to offset 4
> > > >    - Application instance B comes up and fences application instance
> A
> > > >    - Application instance B processes to offset 5
> > > >    - Application instances A and B are idle for
> > > > transaction.id.expiration.ms,
> > > >    transaction id expires on server
> > > >    - Application instance A attempts to process offset 5 (since in
> its
> > > >    view, that is next) -- if we recover from invalid pid mapping, we
> > can
> > > >    duplicate this processing
> > > >
> > > > Thus, INVALID_PID_MAPPING should be fatal to the producer and as part
> > of
> > > > KIP-890 the new client in 4.0 will treat the exception as fatal.
> > > >
> > > > This is consistent with KIP-1050: Consistent error handling for
> > > > Transactions
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> > > > >
> > > > where
> > > > errors that are fatal to the producer are in the "application
> > > recoverable"
> > > > category. This is a grouping that indicates to the client that the
> > > producer
> > > > needs to restart and recovery on the application side is necessary.
> > > > KIP-1050 is approved so we are consistent with that decision.
> > > >
> > > > Let me know if there are any questions. I've updated the KIP to
> reflect
> > > > this change.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Sounds good, Justine. It would be useful to document that in the
> KIP.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > > > > <jo...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > The client will send the newest EndTxn request version if and
> only
> > if
> > > > > both
> > > > > > the client and the server support kip-890 part 2.
> > > > > > We set the value in the record based on the EndTxn version.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <jun@confluent.io.invalid
> >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Justine,
> > > > > > >
> > > > > > > Thanks for the reply.
> > > > > > >
> > > > > > > 120. If the broker sends TV Y for the finalized version in
> > > > > > > ApiVersionResponse, but the client doesn't support Y, how does
> > the
> > > > > broker
> > > > > > > know the TV that the client supports?
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Jun,
> > > > > > > >
> > > > > > > > No worries. Work on this KIP has been blocked for a bit
> anyways
> > > --
> > > > > > > catching
> > > > > > > > up and rereading what I wrote :)
> > > > > > > >
> > > > > > > > 120. ClientTransactionProtocolVersion is the transaction
> > version
> > > as
> > > > > > > defined
> > > > > > > > by the highest transaction version (feature version value)
> > > > supported
> > > > > by
> > > > > > > the
> > > > > > > > client and the server. This works by the broker sending an
> > > > > > > > ApiVersionsRequest to the client with the finalized version.
> > > > Assuming
> > > > > > > > kip-890 part 2 is enabled by transaction version Y, if this
> > > request
> > > > > > > > contains finalized version Y and the client has the logic to
> > set
> > > > this
> > > > > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2
> > not
> > > > > > enable)
> > > > > > > > the client will send Y - 1, even though the client has the
> > > ability
> > > > to
> > > > > > > > support kip-890 part 2.
> > > > > > > >
> > > > > > > > 121. You are correct that this is not needed. However,
> > currently
> > > > that
> > > > > > > field
> > > > > > > > is already being set in memory -- just not written to disk. I
> > > think
> > > > > it
> > > > > > is
> > > > > > > > ok to write it to disk though. Let me know if you think
> > > otherwise.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Justine,
> > > > > > > > >
> > > > > > > > > Thanks for the update and sorry for the late reply.
> > > > > > > > >
> > > > > > > > > 120. I am wondering what value is used for
> > > > > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > > > > EndTxnRequest?
> > > > > > > > >
> > > > > > > > > 121. Earlier, you made the change to set lastProducerId in
> > > > PREPARE
> > > > > to
> > > > > > > > > indicate that the marker is written for the new client.
> With
> > > the
> > > > > new
> > > > > > > > > ClientTransactionProtocolVersion field, it seems this is no
> > > > longer
> > > > > > > > > necessary.
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi there -- another update!
> > > > > > > > > >
> > > > > > > > > > When looking into the implementation for the safe epoch
> > > bumps I
> > > > > > > > realized
> > > > > > > > > > that we are already populating previousProducerID in
> memory
> > > as
> > > > > part
> > > > > > > of
> > > > > > > > > > KIP-360.
> > > > > > > > > > If we are to start using flexible fields, it is better to
> > > > always
> > > > > > use
> > > > > > > > this
> > > > > > > > > > information and have an explicit (tagged) field to
> indicate
> > > > > whether
> > > > > > > the
> > > > > > > > > > client supports KIP-890 part 2.
> > > > > > > > > >
> > > > > > > > > > I've included the extra field and how it is set in the
> KIP.
> > > > I've
> > > > > > also
> > > > > > > > > > updated the KIP to explain that we will be setting the
> > tagged
> > > > > > fields
> > > > > > > > when
> > > > > > > > > > they are available for all transitions.
> > > > > > > > > >
> > > > > > > > > > Finally, I added clearer text about the transaction
> > protocol
> > > > > > versions
> > > > > > > > > > included with this KIP. 1 for flexible transaction state
> > > > records
> > > > > > and
> > > > > > > 2
> > > > > > > > > for
> > > > > > > > > > KIP-890 part 2 enablement.
> > > > > > > > > >
> > > > > > > > > > Justine
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > > > > jolshan@confluent.io
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey there -- small update to the KIP,
> > > > > > > > > > >
> > > > > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and
> bumping
> > > > > > > > > TxnOffsetCommit
> > > > > > > > > > > and Produce requests. I've changed the name in the KIP
> to
> > > > > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > > > > AbortableTransactionException to match the pattern we
> had
> > > for
> > > > > > other
> > > > > > > > > > errors.
> > > > > > > > > > > I also mentioned bumping all 6 transactional APIs so we
> > can
> > > > > > future
> > > > > > > > > > > proof/support the error on the client going forward.
> If a
> > > > > future
> > > > > > > > change
> > > > > > > > > > > wants to have an error scenario that requires us to
> abort
> > > the
> > > > > > > > > > transaction,
> > > > > > > > > > > we can rely on the 3.8+ clients to support it. We ran
> > into
> > > > > issues
> > > > > > > > > finding
> > > > > > > > > > > good/generic error codes that older clients could
> support
> > > > while
> > > > > > > > working
> > > > > > > > > > on
> > > > > > > > > > > this KIP, so this should help in the future.
> > > > > > > > > > >
> > > > > > > > > > > The features discussion is still ongoing in KIP-1022.
> > Will
> > > > > update
> > > > > > > > again
> > > > > > > > > > > here when that concludes.
> > > > > > > > > > >
> > > > > > > > > > > Justine
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > > > > jolshan@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> I don't think AddPartitions is a good example since we
> > > > > currenly
> > > > > > > > don't
> > > > > > > > > > >> gate the version on TV or MV. (We only set a different
> > > flag
> > > > > > > > depending
> > > > > > > > > on
> > > > > > > > > > >> the TV)
> > > > > > > > > > >>
> > > > > > > > > > >> Even if we did want to gate it on TV, I think the idea
> > is
> > > to
> > > > > > move
> > > > > > > > away
> > > > > > > > > > >> from MV gating inter broker protocols. Ideally we can
> > get
> > > > to a
> > > > > > > state
> > > > > > > > > > where
> > > > > > > > > > >> MV is just used for metadata changes.
> > > > > > > > > > >>
> > > > > > > > > > >> I think some of this discussion might fit more with
> the
> > > > > feature
> > > > > > > > > version
> > > > > > > > > > >> KIP, so I can try to open that up soon. Until we
> settle
> > > > that,
> > > > > > some
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> work in KIP-890 is blocked.
> > > > > > > > > > >>
> > > > > > > > > > >> Justine
> > > > > > > > > > >>
> > > > > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Hi, Justine,
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Since AddPartitions is an inter broker request, will
> > its
> > > > > > version
> > > > > > > be
> > > > > > > > > > gated
> > > > > > > > > > >>> only by TV or other features like MV too? For
> example,
> > if
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > > > >>> change
> > > > > > > > > > >>> the protocol for AddPartitions for reasons other than
> > txn
> > > > > > > > > verification
> > > > > > > > > > in
> > > > > > > > > > >>> the future, will the new version be gated by a new
> MV?
> > If
> > > > so,
> > > > > > > does
> > > > > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Jun
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > > > > involved,
> > > > > > > only
> > > > > > > > > the
> > > > > > > > > > >>> > transactional records that should only be gated by
> > TV)
> > > > > > > > > > >>> > Another TV gates the ability to turn on kip-890
> part
> > 2.
> > > > > This
> > > > > > > > would
> > > > > > > > > > >>> gate the
> > > > > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > > > > transactions),
> > > > > > > > > and
> > > > > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the
> > > version
> > > > > is
> > > > > > > > > already
> > > > > > > > > > >>> used
> > > > > > > > > > >>> > without TV.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > I think the only concern is the Produce request and
> > we
> > > > > could
> > > > > > > > > consider
> > > > > > > > > > >>> work
> > > > > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Justine
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > Hi, Justine,
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> > > > forward,
> > > > > > will
> > > > > > > > > those
> > > > > > > > > > >>> > > PRC/record protocols only be guarded by TV and
> not
> > by
> > > > > other
> > > > > > > > > > features
> > > > > > > > > > >>> like
> > > > > > > > > > >>> > > MV?
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Jun
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > > Hi Jun,
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Sorry I think I misunderstood your question or
> > > > answered
> > > > > > > > > > >>> incorrectly.
> > > > > > > > > > >>> > The
> > > > > > > > > > >>> > > TV
> > > > > > > > > > >>> > > > version should ideally be fully independent
> from
> > > MV.
> > > > > > > > > > >>> > > > At least for the changes I proposed, TV should
> > not
> > > > > affect
> > > > > > > MV
> > > > > > > > > and
> > > > > > > > > > MV
> > > > > > > > > > >>> > > should
> > > > > > > > > > >>> > > > not affect TV/
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > I think if we downgrade TV, only that feature
> > > should
> > > > > > > > downgrade.
> > > > > > > > > > >>> > Likewise
> > > > > > > > > > >>> > > > the same with MV. The finalizedFeatures should
> > just
> > > > > > reflect
> > > > > > > > the
> > > > > > > > > > >>> feature
> > > > > > > > > > >>> > > > downgrade we made.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > I also plan to write a new KIP for managing the
> > > disk
> > > > > > format
> > > > > > > > and
> > > > > > > > > > >>> upgrade
> > > > > > > > > > >>> > > > tool as we will need new flags to support these
> > > > > features.
> > > > > > > > That
> > > > > > > > > > >>> should
> > > > > > > > > > >>> > > help
> > > > > > > > > > >>> > > > clarify some things.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Justine
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > > > > <jun@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > > Hi, Justine,
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> > > > downgrade
> > > > > > > > another
> > > > > > > > > > >>> feature
> > > > > > > > > > >>> > > > (say
> > > > > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What
> would
> > we
> > > > > > return
> > > > > > > > for
> > > > > > > > > > >>> > > > > FinalizedFeatures for MV in
> ApiVersionsResponse
> > > in
> > > > > that
> > > > > > > > case?
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Thanks,
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Jun
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > wrote:
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > > > > (transaction
> > > > > > > > > > version)
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > will
> > > > > > > > > > >>> > > > > stop
> > > > > > > > > > >>> > > > > > using the add partitions to txn
> optimization
> > > and
> > > > > stop
> > > > > > > > > writing
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > flexible
> > > > > > > > > > >>> > > > > > feature version of the log.
> > > > > > > > > > >>> > > > > > In the compatibility section I included
> some
> > > > > > > explanations
> > > > > > > > > on
> > > > > > > > > > >>> how
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > done.
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > Thanks,
> > > > > > > > > > >>> > > > > > Justine
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > wrote:
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > If we ever downgrade the transaction
> > feature,
> > > > any
> > > > > > > > feature
> > > > > > > > > > >>> > depending
> > > > > > > > > > >>> > > > on
> > > > > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > > > > >>> > > > > > >
> > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > > > > changes
> > > > > > > > > > made
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > > > KIP-890
> > > > > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > Jun
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine
> > > Olshan
> > > > > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > > > > questions
> > > > > > > > about
> > > > > > > > > > >>> MV/IBP.
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > Looking at the options, I think it
> makes
> > > the
> > > > > most
> > > > > > > > sense
> > > > > > > > > > to
> > > > > > > > > > >>> > > create a
> > > > > > > > > > >>> > > > > > > > separate feature for transactions and
> use
> > > > that
> > > > > to
> > > > > > > > > version
> > > > > > > > > > >>> gate
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > features
> > > > > > > > > > >>> > > > > > > > we need to version gate (flexible
> > > > transactional
> > > > > > > state
> > > > > > > > > > >>> records
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > > using
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > new protocol)
> > > > > > > > > > >>> > > > > > > > I've updated the KIP to include this
> > > change.
> > > > > > > > Hopefully
> > > > > > > > > > >>> that's
> > > > > > > > > > >>> > > > > > everything
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> > > > Olshan
> > > > > <
> > > > > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > I will update the KIP with the prev
> > field
> > > > for
> > > > > > > > prepare
> > > > > > > > > > as
> > > > > > > > > > >>> > well.
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> > field): x
> > > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > or z
> > > > if
> > > > > y
> > > > > > > will
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> > field): x
> > > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we
> > > overflowed
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > > > > >>> > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun
> Rao
> > > > > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > > > > >>> > > > > > > > >> (1) My point was that the
> coordinator
> > > > could
> > > > > > fail
> > > > > > > > > right
> > > > > > > > > > >>> after
> > > > > > > > > > >>> > > > > writing
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> > > > coordinator
> > > > > > > > > generates
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > complete
> > > > > > > > > > >>> > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> after the failover, it needs some
> > field
> > > > from
> > > > > > the
> > > > > > > > > > prepare
> > > > > > > > > > >>> > > marker
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > > > >> determine whether it's written by
> the
> > > new
> > > > > > > client.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> (2) The changing of the behavior
> > sounds
> > > > good
> > > > > > to
> > > > > > > > me.
> > > > > > > > > We
> > > > > > > > > > >>> only
> > > > > > > > > > >>> > > want
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> success if the prepare state is
> > written
> > > by
> > > > > the
> > > > > > > new
> > > > > > > > > > >>> client.
> > > > > > > > > > >>> > So,
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we
> > also
> > > > > need
> > > > > > > sth
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > > marker
> > > > > > > > > > >>> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> tell us whether it's written by the
> > new
> > > > > > client.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation.
> That
> > > > sounds
> > > > > > > good
> > > > > > > > to
> > > > > > > > > > me.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> Jun
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM
> > Justine
> > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid>
> wrote:
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually
> > have
> > > > two
> > > > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > > need
> > > > to
> > > > > > > write
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > tagged field in the end maker so
> > that
> > > we
> > > > > > know
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > is
> > > > > > > > > > >>> > > > > > from
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> new
> > > > > > > > > > >>> > > > > > > > >> > client. Since the end maker is
> > derived
> > > > > from
> > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > > > marker,
> > > > > > > > > > >>> > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > write the previous produce Id in
> the
> > > > > prepare
> > > > > > > > > marker
> > > > > > > > > > >>> field
> > > > > > > > > > >>> > > too?
> > > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > > >>> > > > > > > > >> > we will lose this information when
> > > > > deriving
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > >>> > marker.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in
> the
> > > > > normal
> > > > > > > > > producer
> > > > > > > > > > >>> ID
> > > > > > > > > > >>> > > field.
> > > > > > > > > > >>> > > > > So
> > > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > need it in prepare and that was
> > always
> > > > the
> > > > > > > plan.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I
> will
> > > > > > enumerate
> > > > > > > > the
> > > > > > > > > > >>> fields
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > add
> > > > > > > > > > >>> > > > > > > them
> > > > > > > > > > >>> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > > > > >>> > > > > > > > >> > Say we have producer ID x and
> epoch
> > y.
> > > > > When
> > > > > > we
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > epoch
> > > > > > > > > > >>> > > > y
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > get
> > > > > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > > field):
> > > > > > empty
> > > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field):
> empty
> > > or
> > > > z
> > > > > > if y
> > > > > > > > > will
> > > > > > > > > > >>> > overflow
> > > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > > field):
> > > > x
> > > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field):
> empty
> > > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> > > > overflowed
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we
> > retry
> > > > and
> > > > > > see
> > > > > > > > > epoch
> > > > > > > > > > -
> > > > > > > > > > >>> 1 +
> > > > > > > > > > >>> > ID
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > last
> > > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > > >>> > > > > > > > >> > fields and are issuing the same
> > > command
> > > > > (ie
> > > > > > > > commit
> > > > > > > > > > not
> > > > > > > > > > >>> > > abort),
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890
> > > seems
> > > > to
> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> > > > making
> > > > > > this
> > > > > > > > > > change?
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer
> > if
> > > > the
> > > > > > > epoch
> > > > > > > > is
> > > > > > > > > > >>> bumped
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > get a
> > > > > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are
> > > intentionally
> > > > > > > adding
> > > > > > > > > this
> > > > > > > > > > >>> to
> > > > > > > > > > >>> > > > prevent
> > > > > > > > > > >>> > > > > > > > fencing.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > 112. We already merged the code
> that
> > > > adds
> > > > > > the
> > > > > > > > > > >>> VerifyOnly
> > > > > > > > > > >>> > > field
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which
> is
> > an
> > > > > inter
> > > > > > > > > broker
> > > > > > > > > > >>> > request.
> > > > > > > > > > >>> > > > It
> > > > > > > > > > >>> > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for
> that.
> > Do
> > > > you
> > > > > > > know
> > > > > > > > > why?
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> > > > interbroker
> > > > > > > > requests
> > > > > > > > > > as
> > > > > > > > > > >>> > > > > ApiVersions
> > > > > > > > > > >>> > > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> > > > errors
> > > > > > > > > correctly
> > > > > > > > > > >>> if we
> > > > > > > > > > >>> > > > > receive
> > > > > > > > > > >>> > > > > > > them
> > > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > > >>> > > > > > > > >> > edge cases like
> upgrades/downgrades.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM
> Jun
> > > Rao
> > > > > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually
> > > have
> > > > > two
> > > > > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > > need
> > > > to
> > > > > > > write
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so
> > > that
> > > > we
> > > > > > > know
> > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > >>> > > marker
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > new
> > > > > > > > > > >>> > > > > > > > >> > > client. Since the end maker is
> > > derived
> > > > > > from
> > > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> should we
> > > > > > > > > > >>> > > > > > > > >> > > write the previous produce Id in
> > the
> > > > > > prepare
> > > > > > > > > > marker
> > > > > > > > > > >>> > field
> > > > > > > > > > >>> > > > too?
> > > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > > >>> > > > > > > > >> > > we will lose this information
> when
> > > > > > deriving
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > >>> > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we
> > > retry
> > > > > and
> > > > > > > see
> > > > > > > > > > epoch
> > > > > > > > > > >>> - 1
> > > > > > > > > > >>> > +
> > > > > > > > > > >>> > > ID
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > last
> > > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> > > > command
> > > > > > (ie
> > > > > > > > > commit
> > > > > > > > > > >>> not
> > > > > > > > > > >>> > > > abort),
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> return
> > > > > > > > > > >>> > > > > > > > >> > > success. The logic before
> KIP-890
> > > > seems
> > > > > to
> > > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > > in this case. Are we
> intentionally
> > > > > making
> > > > > > > this
> > > > > > > > > > >>> change?
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > 112. We already merged the code
> > that
> > > > > adds
> > > > > > > the
> > > > > > > > > > >>> VerifyOnly
> > > > > > > > > > >>> > > > field
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which
> > is
> > > an
> > > > > > inter
> > > > > > > > > > broker
> > > > > > > > > > >>> > > request.
> > > > > > > > > > >>> > > > > It
> > > > > > > > > > >>> > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for
> > that.
> > > Do
> > > > > you
> > > > > > > > know
> > > > > > > > > > why?
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> > > > Justine
> > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen"
> > to
> > > > > > "current
> > > > > > > > > > >>> producer id
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > > > > epoch"
> > > > > > > > > > >>> > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > > was the part that was
> confusing
> > > > > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am
> > > still
> > > > > > > > > finalizing
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > design.
> > > > > > > > > > >>> > > > > I
> > > > > > > > > > >>> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > update
> > > > > > > > > > >>> > > > > > > > >> > > > the KIP once I get the
> > information
> > > > > > > > finalized.
> > > > > > > > > > >>> Sorry
> > > > > > > > > > >>> > for
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > delays.
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at
> 10:50 AM
> > > Jun
> > > > > Rao
> > > > > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow
> > case,
> > > > the
> > > > > > > > previous
> > > > > > > > > > ID
> > > > > > > > > > >>> is
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > as
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker
> > too,
> > > > but
> > > > > we
> > > > > > > set
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > ID
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you
> mentioned
> > > that
> > > > > > this
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > > that
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > > >>> > > > > > > > >> > > > > written by the new client so
> > > that
> > > > we
> > > > > > > could
> > > > > > > > > > >>> return
> > > > > > > > > > >>> > > > success
> > > > > > > > > > >>> > > > > on
> > > > > > > > > > >>> > > > > > > > >> retried
> > > > > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was
> > trying
> > > > to
> > > > > > > > > understand
> > > > > > > > > > >>> why
> > > > > > > > > > >>> > > this
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > not
> > > > > > > > > > >>> > > > > > > > >> needed
> > > > > > > > > > >>> > > > > > > > >> > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > the prepare marker since
> retry
> > > can
> > > > > > > happen
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > state
> > > > > > > > > > >>> > > > > > > > >> too.
> > > > > > > > > > >>> > > > > > > > >> > Is
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare
> > > state,
> > > > we
> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > > > > >>> > > > > > > > >> > > > > of success on retried
> endMaker
> > > > > > requests?
> > > > > > > > If
> > > > > > > > > > so,
> > > > > > > > > > >>> > should
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > change
> > > > > > > > > > >>> > > > > > > > >> "If
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID
> > in
> > > > last
> > > > > > > seen
> > > > > > > > > > >>> fields and
> > > > > > > > > > >>> > > are
> > > > > > > > > > >>> > > > > > > issuing
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > command (ie commit not
> abort)
> > we
> > > > can
> > > > > > > > return
> > > > > > > > > > >>> (with
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > epoch)"
> > > > > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP,
> a
> > > > > delayed
> > > > > > > > > endMaker
> > > > > > > > > > >>> > request
> > > > > > > > > > >>> > > > > > carries
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced.
> > This
> > > > can
> > > > > > > > > > >>> commit/abort a
> > > > > > > > > > >>> > > > future
> > > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure
> if
> > > we
> > > > > have
> > > > > > > > seen
> > > > > > > > > > >>> this in
> > > > > > > > > > >>> > > > > practice
> > > > > > > > > > >>> > > > > > > > >> though.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would
> be
> > > > useful
> > > > > > to
> > > > > > > > make
> > > > > > > > > > it
> > > > > > > > > > >>> > clear
> > > > > > > > > > >>> > > > that
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> now
> > > > > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field
> > from
> > > > the
> > > > > > log
> > > > > > > > > > >>> reliably.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > > >>> > > > > since
> > > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > called
> > > > > > > > > > >>> > > > > > > > >> > > > > across brokers and we are
> > > changing
> > > > > its
> > > > > > > > > schema.
> > > > > > > > > > >>> Are
> > > > > > > > > > >>> > you
> > > > > > > > > > >>> > > > > > saying
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > don't
> > > > > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that
> we
> > > > > already
> > > > > > > > > > >>> implemented
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > server
> > > > > > > > > > >>> > > > > > > > side
> > > > > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > > >>> > > > > across
> > > > > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at
> > 5:05 PM
> > > > > > Justine
> > > > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the
> > > previous
> > > > ID
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> Prepare
> > > > > > > > > > >>> > > > field
> > > > > > > > > > >>> > > > > > > since
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > > it. It is the same
> producer
> > ID
> > > > as
> > > > > > the
> > > > > > > > main
> > > > > > > > > > >>> > producer
> > > > > > > > > > >>> > > ID
> > > > > > > > > > >>> > > > > > > field.
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> > > > reread
> > > > > > your
> > > > > > > > > > message
> > > > > > > > > > >>> > about
> > > > > > > > > > >>> > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn
> > > marker
> > > > > > after
> > > > > > > > the
> > > > > > > > > > >>> > > transaction
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > > already
> > > > > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > > > > transaction
> > > > > > > > > early
> > > > > > > > > > >>> > without
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > fixes
> > > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > > >>> > > > > > > > >> > > part
> > > > > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this
> terminology
> > > was
> > > > > used
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > >>> previous
> > > > > > > > > > >>> > > KIP
> > > > > > > > > > >>> > > > > and
> > > > > > > > > > >>> > > > > > > > never
> > > > > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log
> --
> > > > only
> > > > > in
> > > > > > > > > memory
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which
> interbroker
> > > > > > protocol
> > > > > > > > are
> > > > > > > > > > you
> > > > > > > > > > >>> > > > referring
> > > > > > > > > > >>> > > > > > to?
> > > > > > > > > > >>> > > > > > > I
> > > > > > > > > > >>> > > > > > > > am
> > > > > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> > > > remove
> > > > > > the
> > > > > > > > > extra
> > > > > > > > > > >>> add
> > > > > > > > > > >>> > > > > partitions
> > > > > > > > > > >>> > > > > > > > call
> > > > > > > > > > >>> > > > > > > > >> > and I
> > > > > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I
> > > have
> > > > > yet
> > > > > > to
> > > > > > > > > > update
> > > > > > > > > > >>> that
> > > > > > > > > > >>> > > > > section
> > > > > > > > > > >>> > > > > > > as
> > > > > > > > > > >>> > > > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > > > > >>> > > > > > > > >> > > > > > the design so please stay
> > > tuned.
> > > > > Was
> > > > > > > > there
> > > > > > > > > > >>> > anything
> > > > > > > > > > >>> > > > else
> > > > > > > > > > >>> > > > > > you
> > > > > > > > > > >>> > > > > > > > >> > thought
> > > > > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at
> > > 3:07 PM
> > > > > Jun
> > > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create
> > any
> > > > > issue.
> > > > > > > It
> > > > > > > > > just
> > > > > > > > > > >>> makes
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > a
> > > > > > > > > > >>> > > > > > bit
> > > > > > > > > > >>> > > > > > > > >> hard to
> > > > > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged
> > produce
> > > > id
> > > > > > > field
> > > > > > > > > > >>> means. We
> > > > > > > > > > >>> > > are
> > > > > > > > > > >>> > > > > > > > >> essentially
> > > > > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> > > > (completing
> > > > > a
> > > > > > > txn
> > > > > > > > > and
> > > > > > > > > > >>> init a
> > > > > > > > > > >>> > > new
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> Id)
> > > > > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this
> > may
> > > > be
> > > > > > fine
> > > > > > > > > too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> > > > comments.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason
> > > that
> > > > we
> > > > > > > only
> > > > > > > > > set
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > previous
> > > > > > > > > > >>> > > > > > > > >> produce id
> > > > > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > field in the complete
> > > marker,
> > > > > but
> > > > > > > not
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> > > > return
> > > > > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > retried
> > > > > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your
> > second
> > > > > point
> > > > > > is
> > > > > > > > > > >>> mentioned
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > KIP. I
> > > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > > >>> > > > > > > > >> > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message
> case
> > > can
> > > > > > also
> > > > > > > > > > violate
> > > > > > > > > > >>> EOS
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > > > > addPartitionsToTxn
> > > > > > > > > > >>> > request
> > > > > > > > > > >>> > > > > comes
> > > > > > > > > > >>> > > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > > > > previous
> > > > > > > > > > (aborted)
> > > > > > > > > > >>> > > > > transaction
> > > > > > > > > > >>> > > > > > > > become
> > > > > > > > > > >>> > > > > > > > >> > part
> > > > > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > The above is the case
> > when a
> > > > > > delayed
> > > > > > > > > > >>> message is
> > > > > > > > > > >>> > > > > appended
> > > > > > > > > > >>> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > data
> > > > > > > > > > >>> > > > > > > > >> > > > > > > partition. What I
> > mentioned
> > > > is a
> > > > > > > > > slightly
> > > > > > > > > > >>> > > different
> > > > > > > > > > >>> > > > > case
> > > > > > > > > > >>> > > > > > > > when
> > > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to
> the
> > > > > > > transaction
> > > > > > > > > log
> > > > > > > > > > >>> > > partition.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once
> we
> > > > move
> > > > > > past
> > > > > > > > the
> > > > > > > > > > >>> Prepare
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > > > > > Complete
> > > > > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry
> about
> > > > > lastSeen
> > > > > > > > > fields
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > > clear
> > > > > > > > > > >>> > > > > > them,
> > > > > > > > > > >>> > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.".
> > Is
> > > > the
> > > > > > > > lastSeen
> > > > > > > > > > >>> field
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > as
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field
> in
> > > > > > > > > > >>> TransactionLogValue?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip
> changes
> > > the
> > > > > > > > > > inter-broker
> > > > > > > > > > >>> > > > protocol,
> > > > > > > > > > >>> > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> > > > feature
> > > > > > only
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > >>> > KRaft
> > > > > > > > > > >>> > > > > mode?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> > > > 11:13 AM
> > > > > > > > Justine
> > > > > > > > > > >>> Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are
> getting
> > to
> > > > > > > > convergence
> > > > > > > > > > on
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > design.
> > > > > > > > > > >>> > > > > > :)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it
> > > seems
> > > > a
> > > > > > > little
> > > > > > > > > > >>> "weird".
> > > > > > > > > > >>> > > I'm
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > > > sure
> > > > > > > > > > >>> > > > > > > > >> what
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra
> > record
> > > > to
> > > > > > the
> > > > > > > > log.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool
> to
> > > > > > describe
> > > > > > > > > > >>> transactions
> > > > > > > > > > >>> > > > won't
> > > > > > > > > > >>> > > > > > > work
> > > > > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to
> > > calculate
> > > > > the
> > > > > > > > time
> > > > > > > > > > >>> since
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason
> like
> > > > this,
> > > > > > it
> > > > > > > is
> > > > > > > > > > >>> enough to
> > > > > > > > > > >>> > > > > > convince
> > > > > > > > > > >>> > > > > > > me
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems
> > > like
> > > > it
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > >>> > > replacing
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > record
> > > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is
> this
> > > > > correct?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024
> at
> > > > > 5:14 PM
> > > > > > > Jun
> > > > > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> > > > explanation.
> > > > > I
> > > > > > > > > > >>> understand the
> > > > > > > > > > >>> > > > > > intention
> > > > > > > > > > >>> > > > > > > > >> now.
> > > > > > > > > > >>> > > > > > > > >> > In
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> > > > non-tagged
> > > > > > > field
> > > > > > > > to
> > > > > > > > > > >>> the old
> > > > > > > > > > >>> > > pid
> > > > > > > > > > >>> > > > > > (and
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> max
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so
> that
> > > we
> > > > > > could
> > > > > > > > > > >>> correctly
> > > > > > > > > > >>> > > write
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> marker to
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the
> > broker
> > > > > > > > downgrades.
> > > > > > > > > > When
> > > > > > > > > > >>> > > writing
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> > > > already
> > > > > > been
> > > > > > > > > > >>> written to
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > data
> > > > > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to
> > the
> > > > new
> > > > > > pid
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to
> > work.
> > > > > It's
> > > > > > > > just a
> > > > > > > > > > bit
> > > > > > > > > > >>> > > > > > inconsistent
> > > > > > > > > > >>> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete
> marker
> > to
> > > > use
> > > > > > > > > different
> > > > > > > > > > >>> pids
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > > this
> > > > > > > > > > >>> > > > > > > > >> special
> > > > > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> > > > complete
> > > > > > > > marker,
> > > > > > > > > it
> > > > > > > > > > >>> seems
> > > > > > > > > > >>> > > > that
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > never
> > > > > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete
> > > marker
> > > > > with
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > >>> pid.
> > > > > > > > > > >>> > > Not
> > > > > > > > > > >>> > > > > sure
> > > > > > > > > > >>> > > > > > > if
> > > > > > > > > > >>> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems
> a
> > > bit
> > > > > > weird.
> > > > > > > > > > >>> Instead of
> > > > > > > > > > >>> > > > > writing
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid,
> > could
> > > we
> > > > > > write
> > > > > > > > two
> > > > > > > > > > >>> > records:
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by
> a
> > > > > > > > > > >>> TransactionLogValue
> > > > > > > > > > >>> > with
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > new
> > > > > > > > > > >>> > > > > > > > pid
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make
> > the
> > > > two
> > > > > > > > records
> > > > > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > same
> > > > > > > > > > >>> > > > > > batch
> > > > > > > > > > >>> > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> > > > atomically.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 5:40 PM
> > > > > > > > > > Justine
> > > > > > > > > > >>> > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > <jolshan@confluent.io.invalid
> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare
> > marker
> > > > is
> > > > > > > > written,
> > > > > > > > > > >>> but the
> > > > > > > > > > >>> > > > > endTxn
> > > > > > > > > > >>> > > > > > > > >> response
> > > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when
> > the
> > > > > > server
> > > > > > > > > > >>> downgrades
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare
> > > marker
> > > > is
> > > > > > > > > written,
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > endTxn
> > > > > > > > > > >>> > > > > > > > response
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the
> > server
> > > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am
> still a
> > > > > little
> > > > > > > > > > confused.
> > > > > > > > > > >>> In
> > > > > > > > > > >>> > > both
> > > > > > > > > > >>> > > > of
> > > > > > > > > > >>> > > > > > > these
> > > > > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log
> has
> > > the
> > > > > old
> > > > > > > > > producer
> > > > > > > > > > >>> ID.
> > > > > > > > > > >>> > We
> > > > > > > > > > >>> > > > > don't
> > > > > > > > > > >>> > > > > > > > write
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare
> > > marker's
> > > > > non
> > > > > > > > tagged
> > > > > > > > > > >>> fields.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server
> > > downgrades
> > > > > > now,
> > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > >>> > read
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > records
> > > > > > > > > > >>> > > > > > > > >> not
> > > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the
> > > complete
> > > > > > marker
> > > > > > > > > will
> > > > > > > > > > >>> also
> > > > > > > > > > >>> > > have
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > old
> > > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used
> the
> > > new
> > > > > > > producer
> > > > > > > > > ID,
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > would
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since
> > the
> > > > > > producer
> > > > > > > > id
> > > > > > > > > > >>> doesn't
> > > > > > > > > > >>> > > > match
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> > > > correct
> > > > > > on
> > > > > > > > the
> > > > > > > > > > data
> > > > > > > > > > >>> > > > > partition.)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow
> > case,
> > > > I'd
> > > > > > > expect
> > > > > > > > > the
> > > > > > > > > > >>> > > following
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > > >> happen on
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we
> retry
> > > > EndTxn
> > > > > > --
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > same
> > > > > > > > > > >>> > > > > > > producer
> > > > > > > > > > >>> > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the
> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't
> > > retry
> > > > > > > EndTxn
> > > > > > > > > and
> > > > > > > > > > >>> use
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> producer id
> > > > > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > > > > InvalidPidMappingException
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> > > > special
> > > > > > > > handling
> > > > > > > > > > for
> > > > > > > > > > >>> > when
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we
> could
> > > get
> > > > an
> > > > > > API
> > > > > > > > > > version
> > > > > > > > > > >>> > > request
> > > > > > > > > > >>> > > > > > > showing
> > > > > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In
> > that
> > > > > case,
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > >>> call
> > > > > > > > > > >>> > > > > > > > initProducerId
> > > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In
> the
> > > > > > overflow
> > > > > > > > > case,
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > > > > correctly
> > > > > > > > > > >>> > > > > > > > gives
> > > > > > > > > > >>> > > > > > > > >> > us a
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> > > > corresponding
> > > > > > case
> > > > > > > > > would
> > > > > > > > > > >>> be
> > > > > > > > > > >>> > > where
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> *complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the
> > endTxn
> > > > is
> > > > > > not
> > > > > > > > > > >>> received by
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > client
> > > > > > > > > > >>> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This
> > would
> > > > > > result
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old
> > > one.
> > > > > If
> > > > > > > the
> > > > > > > > > > client
> > > > > > > > > > >>> > > > retries,
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > InvalidPidMappingException.
> > > > > > > The
> > > > > > > > > > >>> > > InitProducerId
> > > > > > > > > > >>> > > > > > > > scenario
> > > > > > > > > > >>> > > > > > > > >> > above
> > > > > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > > > > compatibility
> > > > > > > > > story
> > > > > > > > > > is
> > > > > > > > > > >>> > meant
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > > > support
> > > > > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping
> the
> > > > > > > > transactional
> > > > > > > > > > >>> > > correctness.
> > > > > > > > > > >>> > > > > > > Keeping
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is
> > not
> > > > the
> > > > > > > > > priority.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I
> > can
> > > > > also
> > > > > > > add
> > > > > > > > > text
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > KIP
> > > > > > > > > > >>> > > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes
> > some
> > > > edge
> > > > > > > > cases.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 4:10 PM
> > > > > > > > > Jun
> > > > > > > > > > >>> Rao
> > > > > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the
> > > reply.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we
> > > don't
> > > > > need
> > > > > > > to
> > > > > > > > > > >>> optimize
> > > > > > > > > > >>> > for
> > > > > > > > > > >>> > > > > > fencing
> > > > > > > > > > >>> > > > > > > > >> during
> > > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding
> > > consistency,
> > > > > > there
> > > > > > > > are
> > > > > > > > > > two
> > > > > > > > > > >>> > > > possible
> > > > > > > > > > >>> > > > > > > cases:
> > > > > > > > > > >>> > > > > > > > >> (1)
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but
> > the
> > > > > endTxn
> > > > > > > > > > response
> > > > > > > > > > >>> is
> > > > > > > > > > >>> > not
> > > > > > > > > > >>> > > > > > > received
> > > > > > > > > > >>> > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > server
> downgrades;
> > > (2)
> > > > > > the
> > > > > > > > > > prepare
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > > written,
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by
> the
> > > > > client
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > >>> > server
> > > > > > > > > > >>> > > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> In
> > > > > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the
> old
> > > > > produce
> > > > > > Id
> > > > > > > > and
> > > > > > > > > > in
> > > > > > > > > > >>> (2),
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > > >>> > > > > > > > >> > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If
> we
> > > > > > downgrade
> > > > > > > > > right
> > > > > > > > > > >>> after
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to
> both
> > > (1)
> > > > > and
> > > > > > > (2)
> > > > > > > > > > >>> since we
> > > > > > > > > > >>> > > can
> > > > > > > > > > >>> > > > > only
> > > > > > > > > > >>> > > > > > > put
> > > > > > > > > > >>> > > > > > > > >> one
> > > > > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce
> > Id
> > > > > field.
> > > > > > > > It's
> > > > > > > > > > >>> also not
> > > > > > > > > > >>> > > > clear
> > > > > > > > > > >>> > > > > > > which
> > > > > > > > > > >>> > > > > > > > >> case
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could
> > probably
> > > > be
> > > > > > > > > consistent
> > > > > > > > > > >>> with
> > > > > > > > > > >>> > > > either
> > > > > > > > > > >>> > > > > > > case.
> > > > > > > > > > >>> > > > > > > > >> By
> > > > > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in
> the
> > > > > prepare
> > > > > > > > > marker,
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > are
> > > > > > > > > > >>> > > > > > > consistent
> > > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the
> > slight
> > > > > > benefit
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > >>> > > produce
> > > > > > > > > > >>> > > > > > field
> > > > > > > > > > >>> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker
> > are
> > > > > > > consistent
> > > > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > overflow
> > > > > > > > > > >>> > > > > > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > > 3:11 PM
> > > > > > > > > > >>> Justine
> > > > > > > > > > >>> > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case
> you
> > > > > > describe,
> > > > > > > we
> > > > > > > > > > would
> > > > > > > > > > >>> > need
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > have
> > > > > > > > > > >>> > > > > > > a
> > > > > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful
> > EndTxn,
> > > > > and a
> > > > > > > > > > >>> successful
> > > > > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> > > > request
> > > > > > go
> > > > > > > > > > through
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > a
> > > > > > > > > > >>> > > > > given
> > > > > > > > > > >>> > > > > > > > >> producer.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to
> > > figure
> > > > > out
> > > > > > > if
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > >>> > > possible
> > > > > > > > > > >>> > > > > for
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > client
> > > > > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous
> > request
> > > > is
> > > > > > > > delayed
> > > > > > > > > > >>> > somewhere.
> > > > > > > > > > >>> > > > But
> > > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > > >>> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence
> the
> > > > > client.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the
> > > overflow
> > > > > > case.
> > > > > > > > In
> > > > > > > > > > the
> > > > > > > > > > >>> > > overflow
> > > > > > > > > > >>> > > > > > case,
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are
> > > different
> > > > on
> > > > > > the
> > > > > > > > > > marker
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > on
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to
> > use
> > > > the
> > > > > > max
> > > > > > > > > epoch
> > > > > > > > > > >>> but
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> > > > epoch 0
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> > > > transactional
> > > > > > > > > > >>> > > > > > > state.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > > > > downgrade
> > > > > > > > case,
> > > > > > > > > we
> > > > > > > > > > >>> want
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > > > see
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the
> client
> > > will
> > > > > > have.
> > > > > > > > If
> > > > > > > > > we
> > > > > > > > > > >>> > > complete
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> commit,
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is
> > reloaded,
> > > > we
> > > > > > need
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > >>> > > > producer
> > > > > > > > > > >>> > > > > ID
> > > > > > > > > > >>> > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid
> > > producer
> > > > ID
> > > > > > > > > mapping.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server
> > > downgrade
> > > > > > cases
> > > > > > > > are
> > > > > > > > > > >>> > > considering
> > > > > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing
> > > from
> > > > > > > previous
> > > > > > > > > > >>> behavior
> > > > > > > > > > >>> > --
> > > > > > > > > > >>> > > > and
> > > > > > > > > > >>> > > > > > are
> > > > > > > > > > >>> > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> > > > safety
> > > > > > from
> > > > > > > > > > fencing
> > > > > > > > > > >>> > > retries
> > > > > > > > > > >>> > > > > (as
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> have
> > > > > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > > > > support).
> > > > > > > > > Perhaps
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > is
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > trade
> > > > > > > > > > >>> > > > > > > > off,
> > > > > > > > > > >>> > > > > > > > >> > but
> > > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > > > > downgrades,
> > > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > > > > > > restarted
> > > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > > >>> > > > > > > > >> > it
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new
> > > producer
> > > > ID
> > > > > > > too).
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan
> 12,
> > > 2024
> > > > > at
> > > > > > > > > 11:42 AM
> > > > > > > > > > >>> Jun
> > > > > > > > > > >>> > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for
> the
> > > > > reply.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If
> the
> > > > marker
> > > > > > is
> > > > > > > > > > written
> > > > > > > > > > >>> by
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> client, we
> > > > > > > > > > >>> > > > > > > > >> > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last
> email
> > > > > > guarantee
> > > > > > > > > that
> > > > > > > > > > >>> any
> > > > > > > > > > >>> > > EndTxn
> > > > > > > > > > >>> > > > > > > > requests
> > > > > > > > > > >>> > > > > > > > >> > with
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the
> same
> > > > > producer
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >>> same
> > > > > > > > > > >>> > > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> Then
> > > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a
> > fenced
> > > > > error
> > > > > > > but
> > > > > > > > > can
> > > > > > > > > > >>> handle
> > > > > > > > > > >>> > > > > > > gracefully
> > > > > > > > > > >>> > > > > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a
> delayed
> > > > > EndTnx
> > > > > > > > > request
> > > > > > > > > > is
> > > > > > > > > > >>> > > > > processed,
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> txn
> > > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next
> > > txn.
> > > > I
> > > > > > > guess
> > > > > > > > in
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > case
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > still
> > > > > > > > > > >>> > > > > > > > >> > return
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> > > > request?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry,
> my
> > > > > > question
> > > > > > > > was
> > > > > > > > > > >>> > > inaccurate.
> > > > > > > > > > >>> > > > > What
> > > > > > > > > > >>> > > > > > > you
> > > > > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The
> downgrade
> > > > > > > > > compatibility I
> > > > > > > > > > >>> > mention
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > that
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > keep
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in
> > the
> > > > > main
> > > > > > > > > > >>> (non-tagged)
> > > > > > > > > > >>> > > > fields
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> did
> > > > > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server
> side."
> > If
> > > > we
> > > > > > want
> > > > > > > > to
> > > > > > > > > do
> > > > > > > > > > >>> this,
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> that we
> > > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current
> > produce
> > > Id
> > > > > and
> > > > > > > max
> > > > > > > > > > >>> epoch in
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > existing
> > > > > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> producerEpoch
> > > > fields
> > > > > > for
> > > > > > > > > both
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > prepare
> > > > > > > > > > >>> > > > > > and
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The
> downgrade
> > > can
> > > > > > happen
> > > > > > > > > after
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > complete
> > > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you
> described,
> > > the
> > > > > > > > > downgraded
> > > > > > > > > > >>> > > > coordinator
> > > > > > > > > > >>> > > > > > will
> > > > > > > > > > >>> > > > > > > > see
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of
> the
> > > old
> > > > > > one.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan
> > 12,
> > > > 2024
> > > > > > at
> > > > > > > > > > 10:44 AM
> > > > > > > > > > >>> > > Justine
> > > > > > > > > > >>> > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can
> update
> > > the
> > > > > > > > > > description.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe
> > your
> > > > > > second
> > > > > > > > > point
> > > > > > > > > > is
> > > > > > > > > > >>> > > > mentioned
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it
> > is
> > > > > > helpful.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> > delayed
> > > > > > message
> > > > > > > > case
> > > > > > > > > > can
> > > > > > > > > > >>> > also
> > > > > > > > > > >>> > > > > > violate
> > > > > > > > > > >>> > > > > > > > EOS
> > > > > > > > > > >>> > > > > > > > >> if
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in
> > after
> > > > the
> > > > > > > next
> > > > > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > > > > >>> > > > > > > > >> request
> > > > > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> > > > message
> > > > > > > from a
> > > > > > > > > > >>> previous
> > > > > > > > > > >>> > > > > > (aborted)
> > > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the
> > marker
> > > is
> > > > > > > written
> > > > > > > > > by
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > new
> > > > > > > > > > >>> > > > > > client,
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > > > > guarantee
> > > > > > > > that
> > > > > > > > > > any
> > > > > > > > > > >>> > EndTxn
> > > > > > > > > > >>> > > > > > > requests
> > > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> > > > producer
> > > > > > and
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > >>> > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> Then we
> > > > > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced
> error
> > > but
> > > > > can
> > > > > > > > > handle
> > > > > > > > > > >>> > > gracefully
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > > >> described
> > > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't
> > think
> > > a
> > > > > > > boolean
> > > > > > > > is
> > > > > > > > > > >>> useful
> > > > > > > > > > >>> > > > since
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence
> or
> > > > lack
> > > > > of
> > > > > > > the
> > > > > > > > > > >>> tagged
> > > > > > > > > > >>> > > field
> > > > > > > > > > >>> > > > > > being
> > > > > > > > > > >>> > > > > > > > >> > written.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > prepare
> > > > > > marker
> > > > > > > we
> > > > > > > > > > will
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In
> > the
> > > > > > Complete
> > > > > > > > > state
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > may
> > > > > > > > > > >>> > > > not.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not
> sure
> > > why
> > > > > the
> > > > > > > > > ongoing
> > > > > > > > > > >>> state
> > > > > > > > > > >>> > > > > matters
> > > > > > > > > > >>> > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> this
> > > > > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not
> sure
> > > > what
> > > > > > you
> > > > > > > > are
> > > > > > > > > > >>> > referring
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > writing
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> > > > prepare
> > > > > > > > marker.
> > > > > > > > > > >>> This is
> > > > > > > > > > >>> > > not
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > > overflow
> > > > > > case,
> > > > > > > we
> > > > > > > > > > >>> write the
> > > > > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > > > > >>> > > > > > > > >> > in
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so
> > we
> > > > know
> > > > > > > what
> > > > > > > > we
> > > > > > > > > > >>> > assigned
> > > > > > > > > > >>> > > > when
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> reload
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> > > > complete,
> > > > > we
> > > > > > > > > > >>> transition
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > > ID
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> main
> > > > > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> > > > previous
> > > > > > > > producer
> > > > > > > > > > ID
> > > > > > > > > > >>> > field
> > > > > > > > > > >>> > > > > filled
> > > > > > > > > > >>> > > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> This
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry
> > > case
> > > > > the
> > > > > > > > > > operation
> > > > > > > > > > >>> > > > completed
> > > > > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer.
> > The
> > > > > > > downgrade
> > > > > > > > > > >>> > > compatibility
> > > > > > > > > > >>> > > > I
> > > > > > > > > > >>> > > > > > > > mention
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> ID
> > > and
> > > > > > epoch
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> main
> > > > > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > > > > >>> > > > > > > > >> > fields
> > > > > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on
> the
> > > > server
> > > > > > > side.
> > > > > > > > > If
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > server
> > > > > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > addresses
> > > > > both
> > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri,
> Jan
> > > 12,
> > > > > 2024
> > > > > > > at
> > > > > > > > > > >>> 10:21 AM
> > > > > > > > > > >>> > Jun
> > > > > > > > > > >>> > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi,
> > Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks
> for
> > > the
> > > > > > > reply.
> > > > > > > > > > Sorry
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > delay. I
> > > > > > > > > > >>> > > > > > > > >> > have a
> > > > > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I
> > think
> > > > the
> > > > > > > > > > motivation
> > > > > > > > > > >>> > > section
> > > > > > > > > > >>> > > > > > could
> > > > > > > > > > >>> > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > motivations
> > > > > listed
> > > > > > > by
> > > > > > > > > the
> > > > > > > > > > >>> KIP is
> > > > > > > > > > >>> > > > "This
> > > > > > > > > > >>> > > > > > can
> > > > > > > > > > >>> > > > > > > > >> happen
> > > > > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or
> delayed
> > > due
> > > > > to
> > > > > > > > > > networking
> > > > > > > > > > >>> > > issues
> > > > > > > > > > >>> > > > > or a
> > > > > > > > > > >>> > > > > > > > >> network
> > > > > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts,
> > and
> > > > then
> > > > > > the
> > > > > > > > > > delayed
> > > > > > > > > > >>> > > message
> > > > > > > > > > >>> > > > > > > finally
> > > > > > > > > > >>> > > > > > > > >> > comes
> > > > > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very
> > > accurate.
> > > > > > > Without
> > > > > > > > > > >>> KIP-890,
> > > > > > > > > > >>> > > > > > currently,
> > > > > > > > > > >>> > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts
> an
> > > > > ongoing
> > > > > > > > > > >>> transaction,
> > > > > > > > > > >>> > it
> > > > > > > > > > >>> > > > > > already
> > > > > > > > > > >>> > > > > > > > >> bumps
> > > > > > > > > > >>> > > > > > > > >> > up
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which
> > > prevents
> > > > > the
> > > > > > > > > delayed
> > > > > > > > > > >>> > produce
> > > > > > > > > > >>> > > > > > message
> > > > > > > > > > >>> > > > > > > > >> from
> > > > > > > > > > >>> > > > > > > > >> > > being
> > > > > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> partition.
> > > > What
> > > > > > can
> > > > > > > > > cause
> > > > > > > > > > a
> > > > > > > > > > >>> > > hanging
> > > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> completes
> > > > > (either
> > > > > > > > aborts
> > > > > > > > > > or
> > > > > > > > > > >>> > > > commits) a
> > > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> successful
> > > ack
> > > > > on
> > > > > > > > > messages
> > > > > > > > > > >>> > > published
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> same
> > > > > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible
> > for
> > > > the
> > > > > > > > delayed
> > > > > > > > > > >>> message
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > be
> > > > > > > > > > >>> > > > > > > > >> appended
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> > > > causing
> > > > > a
> > > > > > > > > > >>> transaction to
> > > > > > > > > > >>> > > > hang.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A
> similar
> > > > issue
> > > > > > (not
> > > > > > > > > > >>> mentioned
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker
> in
> > > the
> > > > > > > > > > coordinator's
> > > > > > > > > > >>> log.
> > > > > > > > > > >>> > > For
> > > > > > > > > > >>> > > > > > > > example,
> > > > > > > > > > >>> > > > > > > > >> > it's
> > > > > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > EndTxnRequest
> > > > to
> > > > > > be
> > > > > > > > > > delayed
> > > > > > > > > > >>> on
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > > > > >>> > > > > > > > >> > By
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > EndTxnRequest
> > > > is
> > > > > > > > > > processed,
> > > > > > > > > > >>> it's
> > > > > > > > > > >>> > > > > > possible
> > > > > > > > > > >>> > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> completed
> > > and
> > > > a
> > > > > > new
> > > > > > > > txn
> > > > > > > > > > has
> > > > > > > > > > >>> > > started.
> > > > > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > > > > >>> > > > > > > > >> > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped
> on
> > > > every
> > > > > > txn,
> > > > > > > > the
> > > > > > > > > > >>> delayed
> > > > > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > > > > >>> > > > > > > > >> > will
> > > > > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare
> > > marker
> > > > > > (and
> > > > > > > > > > >>> eventually a
> > > > > > > > > > >>> > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> marker)
> > > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't
> > cause
> > > > the
> > > > > > > > > > transaction
> > > > > > > > > > >>> to
> > > > > > > > > > >>> > > hang,
> > > > > > > > > > >>> > > > > but
> > > > > > > > > > >>> > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> > proposal
> > > > in
> > > > > > this
> > > > > > > > KIP
> > > > > > > > > > >>> will
> > > > > > > > > > >>> > > > address
> > > > > > > > > > >>> > > > > > this
> > > > > > > > > > >>> > > > > > > > >> issue
> > > > > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> > > > "However, I
> > > > > > was
> > > > > > > > > > >>> writing it
> > > > > > > > > > >>> > so
> > > > > > > > > > >>> > > > > that
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old
> > clients
> > > > > where
> > > > > > we
> > > > > > > > > don't
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > ability
> > > > > > > > > > >>> > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients
> > that
> > > > > can.
> > > > > > > (Old
> > > > > > > > > > >>> clients
> > > > > > > > > > >>> > > don't
> > > > > > > > > > >>> > > > > > bump
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for
> > sure
> > > > the
> > > > > > > write
> > > > > > > > > > >>> belongs
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > given
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I
> am
> > > > > > wondering
> > > > > > > > why
> > > > > > > > > > we
> > > > > > > > > > >>> need
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written
> by
> > > the
> > > > > old
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >>> new
> > > > > > > > > > >>> > > > client.
> > > > > > > > > > >>> > > > > > > Could
> > > > > > > > > > >>> > > > > > > > >> you
> > > > > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > differently
> > > if
> > > > > we
> > > > > > > know
> > > > > > > > > the
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > is
> > > > > > > > > > >>> > > > > > > written
> > > > > > > > > > >>> > > > > > > > >> by
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If
> > we
> > > do
> > > > > > need
> > > > > > > a
> > > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > >>> > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> whether
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old
> > and
> > > > the
> > > > > > new
> > > > > > > > > > client.
> > > > > > > > > > >>> > Would
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > > be
> > > > > > > > > > >>> > > > > > > > >> simpler
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field
> > > instead
> > > > of
> > > > > > > > > > indirectly
> > > > > > > > > > >>> > > through
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> previous
> > > > > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3
> It's
> > > not
> > > > > > clear
> > > > > > > > to
> > > > > > > > > me
> > > > > > > > > > >>> why
> > > > > > > > > > >>> > we
> > > > > > > > > > >>> > > > only
> > > > > > > > > > >>> > > > > > add
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > complete
> > > > > > marker,
> > > > > > > > but
> > > > > > > > > > >>> not in
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether
> a
> > > > marker
> > > > > > is
> > > > > > > > > > written
> > > > > > > > > > >>> by
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> or
> > > > > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do
> this
> > > > > > > > consistently
> > > > > > > > > > for
> > > > > > > > > > >>> all
> > > > > > > > > > >>> > > > > markers.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4
> What
> > > > about
> > > > > > the
> > > > > > > > > > >>> > > > > TransactionLogValue
> > > > > > > > > > >>> > > > > > > > >> record
> > > > > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state?
> > > Should
> > > > we
> > > > > > > also
> > > > > > > > > > >>> > distinguish
> > > > > > > > > > >>> > > > > > whether
> > > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new
> > client?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In
> > the
> > > > > > overflow
> > > > > > > > > case,
> > > > > > > > > > >>> it's
> > > > > > > > > > >>> > > > still
> > > > > > > > > > >>> > > > > > not
> > > > > > > > > > >>> > > > > > > > >> clear
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > me
> > > > > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> > > > produce
> > > > > > Id
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> > prepare
> > > > > > > > > > >>> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> while
> > > > > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> > > > complete
> > > > > > > > marker.
> > > > > > > > > > You
> > > > > > > > > > >>> > > > mentioned
> > > > > > > > > > >>> > > > > > that
> > > > > > > > > > >>> > > > > > > > >> it's
> > > > > > > > > > >>> > > > > > > > >> > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > > > > downgrade
> > > > > > > > with
> > > > > > > > > > >>> either
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> marker
> > > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In
> either
> > > > case,
> > > > > > the
> > > > > > > > > > >>> downgraded
> > > > > > > > > > >>> > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> should
> > > > > > > > > > >>> > > > > > > > >> > > see
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> (probably
> > > the
> > > > > > > previous
> > > > > > > > > > >>> produce
> > > > > > > > > > >>> > > Id),
> > > > > > > > > > >>> > > > > > right?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed,
> > Dec
> > > > 20,
> > > > > > 2023
> > > > > > > > at
> > > > > > > > > > >>> 6:00 PM
> > > > > > > > > > >>> > > > > Justine
> > > > > > > > > > >>> > > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey
> Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks
> > for
> > > > > > taking
> > > > > > > a
> > > > > > > > > look
> > > > > > > > > > >>> at
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > KIP
> > > > > > > > > > >>> > > > > > > again.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100.
> For
> > > the
> > > > > > epoch
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > case,
> > > > > > > > > > >>> > > > > only
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps
> > the
> > > > > > behavior
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >>> rest
> > > > > > > > > > >>> > > of
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> markers
> > > > > > > > > > >>> > > > > > > > >> > > where
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch
> of
> > > the
> > > > > > > > > transaction
> > > > > > > > > > >>> > > records +
> > > > > > > > > > >>> > > > > 1.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101.
> You
> > > are
> > > > > > > correct
> > > > > > > > > > that
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > don't
> > > > > > > > > > >>> > > > > > need
> > > > > > > > > > >>> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > write
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the
> > > same.
> > > > > > > > However,
> > > > > > > > > I
> > > > > > > > > > >>> was
> > > > > > > > > > >>> > > > writing
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old
> > > clients
> > > > > > where
> > > > > > > we
> > > > > > > > > > don't
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > ability
> > > > > > > > > > >>> > > > > > > > >> do
> > > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> clients
> > > that
> > > > > > can.
> > > > > > > > (Old
> > > > > > > > > > >>> clients
> > > > > > > > > > >>> > > > don't
> > > > > > > > > > >>> > > > > > > bump
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say
> for
> > > sure
> > > > > the
> > > > > > > > write
> > > > > > > > > > >>> belongs
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > given
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> > > > request
> > > > > > > from
> > > > > > > > a
> > > > > > > > > > new
> > > > > > > > > > >>> > > client,
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> fill
> > > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> guarantee
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that
> any
> > > > > EndTxn
> > > > > > > > > requests
> > > > > > > > > > >>> with
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > are
> > > > > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the
> same
> > > > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102.
> In
> > > > > prepare
> > > > > > > > phase,
> > > > > > > > > > we
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had.
> It
> > is
> > > > the
> > > > > > > > > producer
> > > > > > > > > > >>> ID and
> > > > > > > > > > >>> > > > epoch
> > > > > > > > > > >>> > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> are
> > > > > > > > > > >>> > > > > > > > >> > on
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase,
> > we
> > > > stay
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > >>> unless
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set
> the
> > > > > producer
> > > > > > > ID
> > > > > > > > to
> > > > > > > > > > >>> the new
> > > > > > > > > > >>> > > one
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > >> generated
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > complete.
> > > > This
> > > > > > is
> > > > > > > > for
> > > > > > > > > > >>> > downgrade
> > > > > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > > > > >>> > > > > > > > >> > > The
> > > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> > > > guards
> > > > > > for
> > > > > > > > > > retries
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > > > > > failovers.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> > prepare
> > > > > phase
> > > > > > > for
> > > > > > > > > > epoch
> > > > > > > > > > >>> > > > overflow
> > > > > > > > > > >>> > > > > > case
> > > > > > > > > > >>> > > > > > > > >> only
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID.
> This
> > > is
> > > > > for
> > > > > > > the
> > > > > > > > > case
> > > > > > > > > > >>> where
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > reload
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> prepare
> > > > state.
> > > > > > > Once
> > > > > > > > > the
> > > > > > > > > > >>> > > > transaction
> > > > > > > > > > >>> > > > > is
> > > > > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> > > > client
> > > > > > > > already
> > > > > > > > > is
> > > > > > > > > > >>> > using.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> > commit
> > > > > phase,
> > > > > > > we
> > > > > > > > > > store
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > previous
> > > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I
> think
> > it
> > > > is
> > > > > > > easier
> > > > > > > > > to
> > > > > > > > > > >>> think
> > > > > > > > > > >>> > of
> > > > > > > > > > >>> > > > it
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> how
> > > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > epoch
> > > > > > before,
> > > > > > > > with
> > > > > > > > > > >>> some
> > > > > > > > > > >>> > > extra
> > > > > > > > > > >>> > > > > > > > bookeeping
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> > > > fields.
> > > > > > We
> > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > >>> do
> > > > > > > > > > >>> > it
> > > > > > > > > > >>> > > > this
> > > > > > > > > > >>> > > > > > way
> > > > > > > > > > >>> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103.
> > Next
> > > > > > producer
> > > > > > > > ID
> > > > > > > > > is
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > > > prepare
> > > > > > > > > > >>> > > > > > > > status
> > > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > > > > complete.
> > > > > > > The
> > > > > > > > > > >>> reason why
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > need
> > > > > > > > > > >>> > > > > > > two
> > > > > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > backwards
> > > > > > > > > compatibility.
> > > > > > > > > > >>> We
> > > > > > > > > > >>> > need
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > keep
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > non-tagged
> > > > > field
> > > > > > > in
> > > > > > > > > case
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > > > downgrade.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104.
> We
> > > set
> > > > > the
> > > > > > > > fields
> > > > > > > > > > as
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > do
> > > > > > > > > > >>> > > in
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do
> this
> > > for
> > > > > > > > > > compatibility
> > > > > > > > > > >>> --
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > non-tagged
> > > > > > fields)
> > > > > > > > It
> > > > > > > > > > >>> will be
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > old
> > > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > > >>> > > > > > > > >> > ID
> > > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope
> > this
> > > > > helps.
> > > > > > > Let
> > > > > > > > > me
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > > you
> > > > > > > > > > >>> > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > further
> > > > > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On
> Wed,
> > > Dec
> > > > > 20,
> > > > > > > 2023
> > > > > > > > > at
> > > > > > > > > > >>> > 3:33 PM
> > > > > > > > > > >>> > > > Jun
> > > > > > > > > > >>> > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> > > > Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It
> > seems
> > > > > that
> > > > > > > you
> > > > > > > > > have
> > > > > > > > > > >>> made
> > > > > > > > > > >>> > > some
> > > > > > > > > > >>> > > > > > > changes
> > > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > particular,
> > > > > we
> > > > > > > are
> > > > > > > > > > >>> changing
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > format
> > > > > > > > > > >>> > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> TransactionLogValue.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > comments
> > > > > > related
> > > > > > > > to
> > > > > > > > > > >>> that.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100.
> > > Just
> > > > to
> > > > > > be
> > > > > > > > > clear.
> > > > > > > > > > >>> The
> > > > > > > > > > >>> > > > > overflow
> > > > > > > > > > >>> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > generated)
> > > > > is
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > >>> > current
> > > > > > > > > > >>> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> equals
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101.
> > For
> > > > the
> > > > > > > "not
> > > > > > > > > > epoch
> > > > > > > > > > >>> > > > overflow"
> > > > > > > > > > >>> > > > > > > case,
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > write
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> tagged
> > > > field
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> complete
> > > > > > > > > > >>> > > > > phase.
> > > > > > > > > > >>> > > > > > Do
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > need
> > > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > doesn't
> > > > > change
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > >>> case?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102.
> > It
> > > > > seems
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > >>> > meaning
> > > > > > > > > > >>> > > > for
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > ProducerId/ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > TransactionLogValue
> > > > > > > > > > >>> changes
> > > > > > > > > > >>> > > > > > depending
> > > > > > > > > > >>> > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > > TransactionStatus
> > > > > > > > > > is
> > > > > > > > > > >>> > > > ongoing,
> > > > > > > > > > >>> > > > > > they
> > > > > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > current
> > > > > > > > > > >>> ProducerEpoch.
> > > > > > > > > > >>> > > When
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > > > > >>> > > they
> > > > > > > > > > >>> > > > > > > > represent
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > > > > ProducerEpoch.
> > > > > > > > > > >>> When the
> > > > > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > further
> > > > > depend
> > > > > > > on
> > > > > > > > > > >>> whether
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> overflows
> > > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > overflow,
> > > > > they
> > > > > > > > > > represent
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> (max).
> > > > > > > Otherwise,
> > > > > > > > > they
> > > > > > > > > > >>> > > represent
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > newly
> > > > > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > ProducerEpoch
> > > > > > of
> > > > > > > > 0.
> > > > > > > > > Is
> > > > > > > > > > >>> that
> > > > > > > > > > >>> > > > right?
> > > > > > > > > > >>> > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> seems
> > > > > > > > > > >>> > > > > > > > >> > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> > > provide
> > > > > > some
> > > > > > > > > > examples
> > > > > > > > > > >>> > like
> > > > > > > > > > >>> > > > what
> > > > > > > > > > >>> > > > > > > Artem
> > > > > > > > > > >>> > > > > > > > >> has
> > > > > > > > > > >>> > > > > > > > >> > > done
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > considered a
> > > > > > > > simpler
> > > > > > > > > > >>> design
> > > > > > > > > > >>> > > > where
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> ProducerId/ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > represent
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > same
> > > > > value
> > > > > > > > (e.g.
> > > > > > > > > > >>> for the
> > > > > > > > > > >>> > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > TransactionStatus
> > > > > > > > > and
> > > > > > > > > > >>> epoch
> > > > > > > > > > >>> > > > > > overflow?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103.
> > > It's
> > > > > not
> > > > > > > > clear
> > > > > > > > > to
> > > > > > > > > > >>> me
> > > > > > > > > > >>> > why
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > need
> > > > > > > > > > >>> > > > > > > 3
> > > > > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > PrevProducerId,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > NextProducerId.
> > > > > > > > > Could
> > > > > > > > > > we
> > > > > > > > > > >>> > just
> > > > > > > > > > >>> > > > have
> > > > > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104.
> > For
> > > > > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > > > > >>> > > > > if
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> set
> > > the
> > > > > > > > > producerId
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> >
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Calvin,

Thanks for the reply. The changes look good to me.

Jun

On Tue, Feb 11, 2025 at 3:27 PM Calvin Liu <ca...@confluent.io.invalid>
wrote:

> Hi Jun,
> Thanks for the feedback, will update the name as suggested.
> The default values are
> add.partitions.to.txn.retry.backoff.ms=20
> -- This is because the client side AddPartitionsToTxn retry backoff is
> 20ms.
> add.partitions.to.txn.retry.backoff.max.ms=100
> -- We want this value to be small to avoid client side timeout and retry,
> but also long enough for the markers to propagate.
>
>
> On Tue, Feb 11, 2025 at 2:36 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Hi, Calvin,
> >
> > Thanks for the update. The change looks good to me. Some minor comments.
> >
> > For retry timeout, our convention is to use retry.backoff.max.ms and
> > retry.backoff.ms. So, to be consistent, the new configurations should be
> > add.partitions.to.txn.retry.backoff.max.ms and
> > add.partitions.to.txn.retry.backoff.ms.
> >
> > Also, could you specify the defaults for these two values?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> >
> > On Mon, Feb 10, 2025 at 2:45 PM Calvin Liu <ca...@confluent.io.invalid>
> > wrote:
> >
> > > Hey team,
> > > In order to address the performance issue in the
> > > https://issues.apache.org/jira/browse/KAFKA-18654. We are making a
> > change
> > > to retry the AddPartitionToTxn requests on the server side when hitting
> > the
> > > CONCURRENT_TRANSACTION error. This error is common after the
> transaction
> > is
> > > committed but before the markers are fully propagated to all the
> involved
> > > partitions. With the server side retry, the client can avoid the
> frequent
> > > and expensive retry for the produce request. In order to better control
> > the
> > > retry behavior, we proposed to add 2 following configs.
> > >
> > >    - *add.partitions.to.txn.max.timeout.ms
> > >    <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum
> > > retry
> > >    timeout when the server attempts to add the partition to the
> > > transaction.
> > >    - *add.partitions.to.txn.retry.backoff.ms
> > >    <http://add.partitions.to.txn.retry.backoff.ms>* defines how
> > frequently
> > >    the server will retry the AddPartitionToTxn.
> > >
> > > Thanks!
> > >
> > > On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hey folks, just wanted to share another quick update with KIP-890.
> > > >
> > > > Making the change to bump epoch after every transaction means that we
> > no
> > > > longer need to call InitProducerId during the life of the producer to
> > > fence
> > > > requests from previous transactions. With KIP-890 part 2, the new
> > client
> > > > will no longer call InitProducerId except on startup to fence a
> > previous
> > > > instance.
> > > >
> > > > With this change, some other calls to InitProducerId were inspected
> > > > including the call after receiving an InvalidPidMappingException.
> This
> > > > exception was changed to abortable as part of KIP-360: Improve
> > > reliability
> > > > of idempotent/transactional producer
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> > > > >.
> > > > However, this change means that we can violate EOS guarantees. As an
> > > > example:
> > > >
> > > > Consider an application that is copying data from one partition to
> > > another
> > > >
> > > >    - Application instance A processes to offset 4
> > > >    - Application instance B comes up and fences application instance
> A
> > > >    - Application instance B processes to offset 5
> > > >    - Application instances A and B are idle for
> > > > transaction.id.expiration.ms,
> > > >    transaction id expires on server
> > > >    - Application instance A attempts to process offset 5 (since in
> its
> > > >    view, that is next) -- if we recover from invalid pid mapping, we
> > can
> > > >    duplicate this processing
> > > >
> > > > Thus, INVALID_PID_MAPPING should be fatal to the producer and as part
> > of
> > > > KIP-890 the new client in 4.0 will treat the exception as fatal.
> > > >
> > > > This is consistent with KIP-1050: Consistent error handling for
> > > > Transactions
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> > > > >
> > > > where
> > > > errors that are fatal to the producer are in the "application
> > > recoverable"
> > > > category. This is a grouping that indicates to the client that the
> > > producer
> > > > needs to restart and recovery on the application side is necessary.
> > > > KIP-1050 is approved so we are consistent with that decision.
> > > >
> > > > Let me know if there are any questions. I've updated the KIP to
> reflect
> > > > this change.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Sounds good, Justine. It would be useful to document that in the
> KIP.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > > > > <jo...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > The client will send the newest EndTxn request version if and
> only
> > if
> > > > > both
> > > > > > the client and the server support kip-890 part 2.
> > > > > > We set the value in the record based on the EndTxn version.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <jun@confluent.io.invalid
> >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Justine,
> > > > > > >
> > > > > > > Thanks for the reply.
> > > > > > >
> > > > > > > 120. If the broker sends TV Y for the finalized version in
> > > > > > > ApiVersionResponse, but the client doesn't support Y, how does
> > the
> > > > > broker
> > > > > > > know the TV that the client supports?
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Jun,
> > > > > > > >
> > > > > > > > No worries. Work on this KIP has been blocked for a bit
> anyways
> > > --
> > > > > > > catching
> > > > > > > > up and rereading what I wrote :)
> > > > > > > >
> > > > > > > > 120. ClientTransactionProtocolVersion is the transaction
> > version
> > > as
> > > > > > > defined
> > > > > > > > by the highest transaction version (feature version value)
> > > > supported
> > > > > by
> > > > > > > the
> > > > > > > > client and the server. This works by the broker sending an
> > > > > > > > ApiVersionsRequest to the client with the finalized version.
> > > > Assuming
> > > > > > > > kip-890 part 2 is enabled by transaction version Y, if this
> > > request
> > > > > > > > contains finalized version Y and the client has the logic to
> > set
> > > > this
> > > > > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2
> > not
> > > > > > enable)
> > > > > > > > the client will send Y - 1, even though the client has the
> > > ability
> > > > to
> > > > > > > > support kip-890 part 2.
> > > > > > > >
> > > > > > > > 121. You are correct that this is not needed. However,
> > currently
> > > > that
> > > > > > > field
> > > > > > > > is already being set in memory -- just not written to disk. I
> > > think
> > > > > it
> > > > > > is
> > > > > > > > ok to write it to disk though. Let me know if you think
> > > otherwise.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Justine,
> > > > > > > > >
> > > > > > > > > Thanks for the update and sorry for the late reply.
> > > > > > > > >
> > > > > > > > > 120. I am wondering what value is used for
> > > > > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > > > > EndTxnRequest?
> > > > > > > > >
> > > > > > > > > 121. Earlier, you made the change to set lastProducerId in
> > > > PREPARE
> > > > > to
> > > > > > > > > indicate that the marker is written for the new client.
> With
> > > the
> > > > > new
> > > > > > > > > ClientTransactionProtocolVersion field, it seems this is no
> > > > longer
> > > > > > > > > necessary.
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi there -- another update!
> > > > > > > > > >
> > > > > > > > > > When looking into the implementation for the safe epoch
> > > bumps I
> > > > > > > > realized
> > > > > > > > > > that we are already populating previousProducerID in
> memory
> > > as
> > > > > part
> > > > > > > of
> > > > > > > > > > KIP-360.
> > > > > > > > > > If we are to start using flexible fields, it is better to
> > > > always
> > > > > > use
> > > > > > > > this
> > > > > > > > > > information and have an explicit (tagged) field to
> indicate
> > > > > whether
> > > > > > > the
> > > > > > > > > > client supports KIP-890 part 2.
> > > > > > > > > >
> > > > > > > > > > I've included the extra field and how it is set in the
> KIP.
> > > > I've
> > > > > > also
> > > > > > > > > > updated the KIP to explain that we will be setting the
> > tagged
> > > > > > fields
> > > > > > > > when
> > > > > > > > > > they are available for all transitions.
> > > > > > > > > >
> > > > > > > > > > Finally, I added clearer text about the transaction
> > protocol
> > > > > > versions
> > > > > > > > > > included with this KIP. 1 for flexible transaction state
> > > > records
> > > > > > and
> > > > > > > 2
> > > > > > > > > for
> > > > > > > > > > KIP-890 part 2 enablement.
> > > > > > > > > >
> > > > > > > > > > Justine
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > > > > jolshan@confluent.io
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey there -- small update to the KIP,
> > > > > > > > > > >
> > > > > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and
> bumping
> > > > > > > > > TxnOffsetCommit
> > > > > > > > > > > and Produce requests. I've changed the name in the KIP
> to
> > > > > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > > > > AbortableTransactionException to match the pattern we
> had
> > > for
> > > > > > other
> > > > > > > > > > errors.
> > > > > > > > > > > I also mentioned bumping all 6 transactional APIs so we
> > can
> > > > > > future
> > > > > > > > > > > proof/support the error on the client going forward.
> If a
> > > > > future
> > > > > > > > change
> > > > > > > > > > > wants to have an error scenario that requires us to
> abort
> > > the
> > > > > > > > > > transaction,
> > > > > > > > > > > we can rely on the 3.8+ clients to support it. We ran
> > into
> > > > > issues
> > > > > > > > > finding
> > > > > > > > > > > good/generic error codes that older clients could
> support
> > > > while
> > > > > > > > working
> > > > > > > > > > on
> > > > > > > > > > > this KIP, so this should help in the future.
> > > > > > > > > > >
> > > > > > > > > > > The features discussion is still ongoing in KIP-1022.
> > Will
> > > > > update
> > > > > > > > again
> > > > > > > > > > > here when that concludes.
> > > > > > > > > > >
> > > > > > > > > > > Justine
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > > > > jolshan@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> I don't think AddPartitions is a good example since we
> > > > > currenly
> > > > > > > > don't
> > > > > > > > > > >> gate the version on TV or MV. (We only set a different
> > > flag
> > > > > > > > depending
> > > > > > > > > on
> > > > > > > > > > >> the TV)
> > > > > > > > > > >>
> > > > > > > > > > >> Even if we did want to gate it on TV, I think the idea
> > is
> > > to
> > > > > > move
> > > > > > > > away
> > > > > > > > > > >> from MV gating inter broker protocols. Ideally we can
> > get
> > > > to a
> > > > > > > state
> > > > > > > > > > where
> > > > > > > > > > >> MV is just used for metadata changes.
> > > > > > > > > > >>
> > > > > > > > > > >> I think some of this discussion might fit more with
> the
> > > > > feature
> > > > > > > > > version
> > > > > > > > > > >> KIP, so I can try to open that up soon. Until we
> settle
> > > > that,
> > > > > > some
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> work in KIP-890 is blocked.
> > > > > > > > > > >>
> > > > > > > > > > >> Justine
> > > > > > > > > > >>
> > > > > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Hi, Justine,
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Since AddPartitions is an inter broker request, will
> > its
> > > > > > version
> > > > > > > be
> > > > > > > > > > gated
> > > > > > > > > > >>> only by TV or other features like MV too? For
> example,
> > if
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > > > >>> change
> > > > > > > > > > >>> the protocol for AddPartitions for reasons other than
> > txn
> > > > > > > > > verification
> > > > > > > > > > in
> > > > > > > > > > >>> the future, will the new version be gated by a new
> MV?
> > If
> > > > so,
> > > > > > > does
> > > > > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Jun
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > > > > involved,
> > > > > > > only
> > > > > > > > > the
> > > > > > > > > > >>> > transactional records that should only be gated by
> > TV)
> > > > > > > > > > >>> > Another TV gates the ability to turn on kip-890
> part
> > 2.
> > > > > This
> > > > > > > > would
> > > > > > > > > > >>> gate the
> > > > > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > > > > transactions),
> > > > > > > > > and
> > > > > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the
> > > version
> > > > > is
> > > > > > > > > already
> > > > > > > > > > >>> used
> > > > > > > > > > >>> > without TV.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > I think the only concern is the Produce request and
> > we
> > > > > could
> > > > > > > > > consider
> > > > > > > > > > >>> work
> > > > > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Justine
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > Hi, Justine,
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> > > > forward,
> > > > > > will
> > > > > > > > > those
> > > > > > > > > > >>> > > PRC/record protocols only be guarded by TV and
> not
> > by
> > > > > other
> > > > > > > > > > features
> > > > > > > > > > >>> like
> > > > > > > > > > >>> > > MV?
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Jun
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > > Hi Jun,
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Sorry I think I misunderstood your question or
> > > > answered
> > > > > > > > > > >>> incorrectly.
> > > > > > > > > > >>> > The
> > > > > > > > > > >>> > > TV
> > > > > > > > > > >>> > > > version should ideally be fully independent
> from
> > > MV.
> > > > > > > > > > >>> > > > At least for the changes I proposed, TV should
> > not
> > > > > affect
> > > > > > > MV
> > > > > > > > > and
> > > > > > > > > > MV
> > > > > > > > > > >>> > > should
> > > > > > > > > > >>> > > > not affect TV/
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > I think if we downgrade TV, only that feature
> > > should
> > > > > > > > downgrade.
> > > > > > > > > > >>> > Likewise
> > > > > > > > > > >>> > > > the same with MV. The finalizedFeatures should
> > just
> > > > > > reflect
> > > > > > > > the
> > > > > > > > > > >>> feature
> > > > > > > > > > >>> > > > downgrade we made.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > I also plan to write a new KIP for managing the
> > > disk
> > > > > > format
> > > > > > > > and
> > > > > > > > > > >>> upgrade
> > > > > > > > > > >>> > > > tool as we will need new flags to support these
> > > > > features.
> > > > > > > > That
> > > > > > > > > > >>> should
> > > > > > > > > > >>> > > help
> > > > > > > > > > >>> > > > clarify some things.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Justine
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > > > > <jun@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > > Hi, Justine,
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> > > > downgrade
> > > > > > > > another
> > > > > > > > > > >>> feature
> > > > > > > > > > >>> > > > (say
> > > > > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What
> would
> > we
> > > > > > return
> > > > > > > > for
> > > > > > > > > > >>> > > > > FinalizedFeatures for MV in
> ApiVersionsResponse
> > > in
> > > > > that
> > > > > > > > case?
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Thanks,
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Jun
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > wrote:
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > > > > (transaction
> > > > > > > > > > version)
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > will
> > > > > > > > > > >>> > > > > stop
> > > > > > > > > > >>> > > > > > using the add partitions to txn
> optimization
> > > and
> > > > > stop
> > > > > > > > > writing
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > flexible
> > > > > > > > > > >>> > > > > > feature version of the log.
> > > > > > > > > > >>> > > > > > In the compatibility section I included
> some
> > > > > > > explanations
> > > > > > > > > on
> > > > > > > > > > >>> how
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > done.
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > Thanks,
> > > > > > > > > > >>> > > > > > Justine
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > wrote:
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > If we ever downgrade the transaction
> > feature,
> > > > any
> > > > > > > > feature
> > > > > > > > > > >>> > depending
> > > > > > > > > > >>> > > > on
> > > > > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > > > > >>> > > > > > >
> > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > > > > changes
> > > > > > > > > > made
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > > > KIP-890
> > > > > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > Jun
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine
> > > Olshan
> > > > > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > > > > questions
> > > > > > > > about
> > > > > > > > > > >>> MV/IBP.
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > Looking at the options, I think it
> makes
> > > the
> > > > > most
> > > > > > > > sense
> > > > > > > > > > to
> > > > > > > > > > >>> > > create a
> > > > > > > > > > >>> > > > > > > > separate feature for transactions and
> use
> > > > that
> > > > > to
> > > > > > > > > version
> > > > > > > > > > >>> gate
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > features
> > > > > > > > > > >>> > > > > > > > we need to version gate (flexible
> > > > transactional
> > > > > > > state
> > > > > > > > > > >>> records
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > > using
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > new protocol)
> > > > > > > > > > >>> > > > > > > > I've updated the KIP to include this
> > > change.
> > > > > > > > Hopefully
> > > > > > > > > > >>> that's
> > > > > > > > > > >>> > > > > > everything
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> > > > Olshan
> > > > > <
> > > > > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > I will update the KIP with the prev
> > field
> > > > for
> > > > > > > > prepare
> > > > > > > > > > as
> > > > > > > > > > >>> > well.
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> > field): x
> > > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > or z
> > > > if
> > > > > y
> > > > > > > will
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> > field): x
> > > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we
> > > overflowed
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > > > > >>> > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun
> Rao
> > > > > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > > > > >>> > > > > > > > >> (1) My point was that the
> coordinator
> > > > could
> > > > > > fail
> > > > > > > > > right
> > > > > > > > > > >>> after
> > > > > > > > > > >>> > > > > writing
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> > > > coordinator
> > > > > > > > > generates
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > complete
> > > > > > > > > > >>> > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> after the failover, it needs some
> > field
> > > > from
> > > > > > the
> > > > > > > > > > prepare
> > > > > > > > > > >>> > > marker
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > > > >> determine whether it's written by
> the
> > > new
> > > > > > > client.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> (2) The changing of the behavior
> > sounds
> > > > good
> > > > > > to
> > > > > > > > me.
> > > > > > > > > We
> > > > > > > > > > >>> only
> > > > > > > > > > >>> > > want
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> success if the prepare state is
> > written
> > > by
> > > > > the
> > > > > > > new
> > > > > > > > > > >>> client.
> > > > > > > > > > >>> > So,
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we
> > also
> > > > > need
> > > > > > > sth
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > > marker
> > > > > > > > > > >>> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> tell us whether it's written by the
> > new
> > > > > > client.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation.
> That
> > > > sounds
> > > > > > > good
> > > > > > > > to
> > > > > > > > > > me.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> Jun
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM
> > Justine
> > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid>
> wrote:
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually
> > have
> > > > two
> > > > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > > need
> > > > to
> > > > > > > write
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > tagged field in the end maker so
> > that
> > > we
> > > > > > know
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > is
> > > > > > > > > > >>> > > > > > from
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> new
> > > > > > > > > > >>> > > > > > > > >> > client. Since the end maker is
> > derived
> > > > > from
> > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > > > marker,
> > > > > > > > > > >>> > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > write the previous produce Id in
> the
> > > > > prepare
> > > > > > > > > marker
> > > > > > > > > > >>> field
> > > > > > > > > > >>> > > too?
> > > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > > >>> > > > > > > > >> > we will lose this information when
> > > > > deriving
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > >>> > marker.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in
> the
> > > > > normal
> > > > > > > > > producer
> > > > > > > > > > >>> ID
> > > > > > > > > > >>> > > field.
> > > > > > > > > > >>> > > > > So
> > > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > need it in prepare and that was
> > always
> > > > the
> > > > > > > plan.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I
> will
> > > > > > enumerate
> > > > > > > > the
> > > > > > > > > > >>> fields
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > add
> > > > > > > > > > >>> > > > > > > them
> > > > > > > > > > >>> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > > > > >>> > > > > > > > >> > Say we have producer ID x and
> epoch
> > y.
> > > > > When
> > > > > > we
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > epoch
> > > > > > > > > > >>> > > > y
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > get
> > > > > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > > field):
> > > > > > empty
> > > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field):
> empty
> > > or
> > > > z
> > > > > > if y
> > > > > > > > > will
> > > > > > > > > > >>> > overflow
> > > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > > field):
> > > > x
> > > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field):
> empty
> > > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> > > > overflowed
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we
> > retry
> > > > and
> > > > > > see
> > > > > > > > > epoch
> > > > > > > > > > -
> > > > > > > > > > >>> 1 +
> > > > > > > > > > >>> > ID
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > last
> > > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > > >>> > > > > > > > >> > fields and are issuing the same
> > > command
> > > > > (ie
> > > > > > > > commit
> > > > > > > > > > not
> > > > > > > > > > >>> > > abort),
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890
> > > seems
> > > > to
> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> > > > making
> > > > > > this
> > > > > > > > > > change?
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer
> > if
> > > > the
> > > > > > > epoch
> > > > > > > > is
> > > > > > > > > > >>> bumped
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > get a
> > > > > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are
> > > intentionally
> > > > > > > adding
> > > > > > > > > this
> > > > > > > > > > >>> to
> > > > > > > > > > >>> > > > prevent
> > > > > > > > > > >>> > > > > > > > fencing.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > 112. We already merged the code
> that
> > > > adds
> > > > > > the
> > > > > > > > > > >>> VerifyOnly
> > > > > > > > > > >>> > > field
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which
> is
> > an
> > > > > inter
> > > > > > > > > broker
> > > > > > > > > > >>> > request.
> > > > > > > > > > >>> > > > It
> > > > > > > > > > >>> > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for
> that.
> > Do
> > > > you
> > > > > > > know
> > > > > > > > > why?
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> > > > interbroker
> > > > > > > > requests
> > > > > > > > > > as
> > > > > > > > > > >>> > > > > ApiVersions
> > > > > > > > > > >>> > > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> > > > errors
> > > > > > > > > correctly
> > > > > > > > > > >>> if we
> > > > > > > > > > >>> > > > > receive
> > > > > > > > > > >>> > > > > > > them
> > > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > > >>> > > > > > > > >> > edge cases like
> upgrades/downgrades.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM
> Jun
> > > Rao
> > > > > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually
> > > have
> > > > > two
> > > > > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > > need
> > > > to
> > > > > > > write
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so
> > > that
> > > > we
> > > > > > > know
> > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > >>> > > marker
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > new
> > > > > > > > > > >>> > > > > > > > >> > > client. Since the end maker is
> > > derived
> > > > > > from
> > > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> should we
> > > > > > > > > > >>> > > > > > > > >> > > write the previous produce Id in
> > the
> > > > > > prepare
> > > > > > > > > > marker
> > > > > > > > > > >>> > field
> > > > > > > > > > >>> > > > too?
> > > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > > >>> > > > > > > > >> > > we will lose this information
> when
> > > > > > deriving
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > >>> > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we
> > > retry
> > > > > and
> > > > > > > see
> > > > > > > > > > epoch
> > > > > > > > > > >>> - 1
> > > > > > > > > > >>> > +
> > > > > > > > > > >>> > > ID
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > last
> > > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> > > > command
> > > > > > (ie
> > > > > > > > > commit
> > > > > > > > > > >>> not
> > > > > > > > > > >>> > > > abort),
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> return
> > > > > > > > > > >>> > > > > > > > >> > > success. The logic before
> KIP-890
> > > > seems
> > > > > to
> > > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > > in this case. Are we
> intentionally
> > > > > making
> > > > > > > this
> > > > > > > > > > >>> change?
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > 112. We already merged the code
> > that
> > > > > adds
> > > > > > > the
> > > > > > > > > > >>> VerifyOnly
> > > > > > > > > > >>> > > > field
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which
> > is
> > > an
> > > > > > inter
> > > > > > > > > > broker
> > > > > > > > > > >>> > > request.
> > > > > > > > > > >>> > > > > It
> > > > > > > > > > >>> > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for
> > that.
> > > Do
> > > > > you
> > > > > > > > know
> > > > > > > > > > why?
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> > > > Justine
> > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen"
> > to
> > > > > > "current
> > > > > > > > > > >>> producer id
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > > > > epoch"
> > > > > > > > > > >>> > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > > was the part that was
> confusing
> > > > > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am
> > > still
> > > > > > > > > finalizing
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > design.
> > > > > > > > > > >>> > > > > I
> > > > > > > > > > >>> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > update
> > > > > > > > > > >>> > > > > > > > >> > > > the KIP once I get the
> > information
> > > > > > > > finalized.
> > > > > > > > > > >>> Sorry
> > > > > > > > > > >>> > for
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > delays.
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at
> 10:50 AM
> > > Jun
> > > > > Rao
> > > > > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow
> > case,
> > > > the
> > > > > > > > previous
> > > > > > > > > > ID
> > > > > > > > > > >>> is
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > as
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker
> > too,
> > > > but
> > > > > we
> > > > > > > set
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > ID
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you
> mentioned
> > > that
> > > > > > this
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > > that
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > > >>> > > > > > > > >> > > > > written by the new client so
> > > that
> > > > we
> > > > > > > could
> > > > > > > > > > >>> return
> > > > > > > > > > >>> > > > success
> > > > > > > > > > >>> > > > > on
> > > > > > > > > > >>> > > > > > > > >> retried
> > > > > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was
> > trying
> > > > to
> > > > > > > > > understand
> > > > > > > > > > >>> why
> > > > > > > > > > >>> > > this
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > not
> > > > > > > > > > >>> > > > > > > > >> needed
> > > > > > > > > > >>> > > > > > > > >> > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > the prepare marker since
> retry
> > > can
> > > > > > > happen
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > state
> > > > > > > > > > >>> > > > > > > > >> too.
> > > > > > > > > > >>> > > > > > > > >> > Is
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare
> > > state,
> > > > we
> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > > > > >>> > > > > > > > >> > > > > of success on retried
> endMaker
> > > > > > requests?
> > > > > > > > If
> > > > > > > > > > so,
> > > > > > > > > > >>> > should
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > change
> > > > > > > > > > >>> > > > > > > > >> "If
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID
> > in
> > > > last
> > > > > > > seen
> > > > > > > > > > >>> fields and
> > > > > > > > > > >>> > > are
> > > > > > > > > > >>> > > > > > > issuing
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > command (ie commit not
> abort)
> > we
> > > > can
> > > > > > > > return
> > > > > > > > > > >>> (with
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > epoch)"
> > > > > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP,
> a
> > > > > delayed
> > > > > > > > > endMaker
> > > > > > > > > > >>> > request
> > > > > > > > > > >>> > > > > > carries
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced.
> > This
> > > > can
> > > > > > > > > > >>> commit/abort a
> > > > > > > > > > >>> > > > future
> > > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure
> if
> > > we
> > > > > have
> > > > > > > > seen
> > > > > > > > > > >>> this in
> > > > > > > > > > >>> > > > > practice
> > > > > > > > > > >>> > > > > > > > >> though.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would
> be
> > > > useful
> > > > > > to
> > > > > > > > make
> > > > > > > > > > it
> > > > > > > > > > >>> > clear
> > > > > > > > > > >>> > > > that
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> now
> > > > > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field
> > from
> > > > the
> > > > > > log
> > > > > > > > > > >>> reliably.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > > >>> > > > > since
> > > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > called
> > > > > > > > > > >>> > > > > > > > >> > > > > across brokers and we are
> > > changing
> > > > > its
> > > > > > > > > schema.
> > > > > > > > > > >>> Are
> > > > > > > > > > >>> > you
> > > > > > > > > > >>> > > > > > saying
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > don't
> > > > > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that
> we
> > > > > already
> > > > > > > > > > >>> implemented
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > server
> > > > > > > > > > >>> > > > > > > > side
> > > > > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > > >>> > > > > across
> > > > > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at
> > 5:05 PM
> > > > > > Justine
> > > > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the
> > > previous
> > > > ID
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> Prepare
> > > > > > > > > > >>> > > > field
> > > > > > > > > > >>> > > > > > > since
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > > it. It is the same
> producer
> > ID
> > > > as
> > > > > > the
> > > > > > > > main
> > > > > > > > > > >>> > producer
> > > > > > > > > > >>> > > ID
> > > > > > > > > > >>> > > > > > > field.
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> > > > reread
> > > > > > your
> > > > > > > > > > message
> > > > > > > > > > >>> > about
> > > > > > > > > > >>> > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn
> > > marker
> > > > > > after
> > > > > > > > the
> > > > > > > > > > >>> > > transaction
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > > already
> > > > > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > > > > transaction
> > > > > > > > > early
> > > > > > > > > > >>> > without
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > fixes
> > > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > > >>> > > > > > > > >> > > part
> > > > > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this
> terminology
> > > was
> > > > > used
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > >>> previous
> > > > > > > > > > >>> > > KIP
> > > > > > > > > > >>> > > > > and
> > > > > > > > > > >>> > > > > > > > never
> > > > > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log
> --
> > > > only
> > > > > in
> > > > > > > > > memory
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which
> interbroker
> > > > > > protocol
> > > > > > > > are
> > > > > > > > > > you
> > > > > > > > > > >>> > > > referring
> > > > > > > > > > >>> > > > > > to?
> > > > > > > > > > >>> > > > > > > I
> > > > > > > > > > >>> > > > > > > > am
> > > > > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> > > > remove
> > > > > > the
> > > > > > > > > extra
> > > > > > > > > > >>> add
> > > > > > > > > > >>> > > > > partitions
> > > > > > > > > > >>> > > > > > > > call
> > > > > > > > > > >>> > > > > > > > >> > and I
> > > > > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I
> > > have
> > > > > yet
> > > > > > to
> > > > > > > > > > update
> > > > > > > > > > >>> that
> > > > > > > > > > >>> > > > > section
> > > > > > > > > > >>> > > > > > > as
> > > > > > > > > > >>> > > > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > > > > >>> > > > > > > > >> > > > > > the design so please stay
> > > tuned.
> > > > > Was
> > > > > > > > there
> > > > > > > > > > >>> > anything
> > > > > > > > > > >>> > > > else
> > > > > > > > > > >>> > > > > > you
> > > > > > > > > > >>> > > > > > > > >> > thought
> > > > > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at
> > > 3:07 PM
> > > > > Jun
> > > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create
> > any
> > > > > issue.
> > > > > > > It
> > > > > > > > > just
> > > > > > > > > > >>> makes
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > a
> > > > > > > > > > >>> > > > > > bit
> > > > > > > > > > >>> > > > > > > > >> hard to
> > > > > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged
> > produce
> > > > id
> > > > > > > field
> > > > > > > > > > >>> means. We
> > > > > > > > > > >>> > > are
> > > > > > > > > > >>> > > > > > > > >> essentially
> > > > > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> > > > (completing
> > > > > a
> > > > > > > txn
> > > > > > > > > and
> > > > > > > > > > >>> init a
> > > > > > > > > > >>> > > new
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> Id)
> > > > > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this
> > may
> > > > be
> > > > > > fine
> > > > > > > > > too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> > > > comments.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason
> > > that
> > > > we
> > > > > > > only
> > > > > > > > > set
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > previous
> > > > > > > > > > >>> > > > > > > > >> produce id
> > > > > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > field in the complete
> > > marker,
> > > > > but
> > > > > > > not
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> > > > return
> > > > > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > retried
> > > > > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your
> > second
> > > > > point
> > > > > > is
> > > > > > > > > > >>> mentioned
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > KIP. I
> > > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > > >>> > > > > > > > >> > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message
> case
> > > can
> > > > > > also
> > > > > > > > > > violate
> > > > > > > > > > >>> EOS
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > > > > addPartitionsToTxn
> > > > > > > > > > >>> > request
> > > > > > > > > > >>> > > > > comes
> > > > > > > > > > >>> > > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > > > > previous
> > > > > > > > > > (aborted)
> > > > > > > > > > >>> > > > > transaction
> > > > > > > > > > >>> > > > > > > > become
> > > > > > > > > > >>> > > > > > > > >> > part
> > > > > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > The above is the case
> > when a
> > > > > > delayed
> > > > > > > > > > >>> message is
> > > > > > > > > > >>> > > > > appended
> > > > > > > > > > >>> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > data
> > > > > > > > > > >>> > > > > > > > >> > > > > > > partition. What I
> > mentioned
> > > > is a
> > > > > > > > > slightly
> > > > > > > > > > >>> > > different
> > > > > > > > > > >>> > > > > case
> > > > > > > > > > >>> > > > > > > > when
> > > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to
> the
> > > > > > > transaction
> > > > > > > > > log
> > > > > > > > > > >>> > > partition.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once
> we
> > > > move
> > > > > > past
> > > > > > > > the
> > > > > > > > > > >>> Prepare
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > > > > > Complete
> > > > > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry
> about
> > > > > lastSeen
> > > > > > > > > fields
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > > clear
> > > > > > > > > > >>> > > > > > them,
> > > > > > > > > > >>> > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.".
> > Is
> > > > the
> > > > > > > > lastSeen
> > > > > > > > > > >>> field
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > as
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field
> in
> > > > > > > > > > >>> TransactionLogValue?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip
> changes
> > > the
> > > > > > > > > > inter-broker
> > > > > > > > > > >>> > > > protocol,
> > > > > > > > > > >>> > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> > > > feature
> > > > > > only
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > >>> > KRaft
> > > > > > > > > > >>> > > > > mode?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> > > > 11:13 AM
> > > > > > > > Justine
> > > > > > > > > > >>> Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are
> getting
> > to
> > > > > > > > convergence
> > > > > > > > > > on
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > design.
> > > > > > > > > > >>> > > > > > :)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it
> > > seems
> > > > a
> > > > > > > little
> > > > > > > > > > >>> "weird".
> > > > > > > > > > >>> > > I'm
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > > > sure
> > > > > > > > > > >>> > > > > > > > >> what
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra
> > record
> > > > to
> > > > > > the
> > > > > > > > log.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool
> to
> > > > > > describe
> > > > > > > > > > >>> transactions
> > > > > > > > > > >>> > > > won't
> > > > > > > > > > >>> > > > > > > work
> > > > > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to
> > > calculate
> > > > > the
> > > > > > > > time
> > > > > > > > > > >>> since
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason
> like
> > > > this,
> > > > > > it
> > > > > > > is
> > > > > > > > > > >>> enough to
> > > > > > > > > > >>> > > > > > convince
> > > > > > > > > > >>> > > > > > > me
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems
> > > like
> > > > it
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > >>> > > replacing
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > record
> > > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is
> this
> > > > > correct?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024
> at
> > > > > 5:14 PM
> > > > > > > Jun
> > > > > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> > > > explanation.
> > > > > I
> > > > > > > > > > >>> understand the
> > > > > > > > > > >>> > > > > > intention
> > > > > > > > > > >>> > > > > > > > >> now.
> > > > > > > > > > >>> > > > > > > > >> > In
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> > > > non-tagged
> > > > > > > field
> > > > > > > > to
> > > > > > > > > > >>> the old
> > > > > > > > > > >>> > > pid
> > > > > > > > > > >>> > > > > > (and
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> max
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so
> that
> > > we
> > > > > > could
> > > > > > > > > > >>> correctly
> > > > > > > > > > >>> > > write
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> marker to
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the
> > broker
> > > > > > > > downgrades.
> > > > > > > > > > When
> > > > > > > > > > >>> > > writing
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> > > > already
> > > > > > been
> > > > > > > > > > >>> written to
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > data
> > > > > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to
> > the
> > > > new
> > > > > > pid
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to
> > work.
> > > > > It's
> > > > > > > > just a
> > > > > > > > > > bit
> > > > > > > > > > >>> > > > > > inconsistent
> > > > > > > > > > >>> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete
> marker
> > to
> > > > use
> > > > > > > > > different
> > > > > > > > > > >>> pids
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > > this
> > > > > > > > > > >>> > > > > > > > >> special
> > > > > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> > > > complete
> > > > > > > > marker,
> > > > > > > > > it
> > > > > > > > > > >>> seems
> > > > > > > > > > >>> > > > that
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > never
> > > > > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete
> > > marker
> > > > > with
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > >>> pid.
> > > > > > > > > > >>> > > Not
> > > > > > > > > > >>> > > > > sure
> > > > > > > > > > >>> > > > > > > if
> > > > > > > > > > >>> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems
> a
> > > bit
> > > > > > weird.
> > > > > > > > > > >>> Instead of
> > > > > > > > > > >>> > > > > writing
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid,
> > could
> > > we
> > > > > > write
> > > > > > > > two
> > > > > > > > > > >>> > records:
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by
> a
> > > > > > > > > > >>> TransactionLogValue
> > > > > > > > > > >>> > with
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > new
> > > > > > > > > > >>> > > > > > > > pid
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make
> > the
> > > > two
> > > > > > > > records
> > > > > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > same
> > > > > > > > > > >>> > > > > > batch
> > > > > > > > > > >>> > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> > > > atomically.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 5:40 PM
> > > > > > > > > > Justine
> > > > > > > > > > >>> > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > <jolshan@confluent.io.invalid
> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare
> > marker
> > > > is
> > > > > > > > written,
> > > > > > > > > > >>> but the
> > > > > > > > > > >>> > > > > endTxn
> > > > > > > > > > >>> > > > > > > > >> response
> > > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when
> > the
> > > > > > server
> > > > > > > > > > >>> downgrades
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare
> > > marker
> > > > is
> > > > > > > > > written,
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > endTxn
> > > > > > > > > > >>> > > > > > > > response
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the
> > server
> > > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am
> still a
> > > > > little
> > > > > > > > > > confused.
> > > > > > > > > > >>> In
> > > > > > > > > > >>> > > both
> > > > > > > > > > >>> > > > of
> > > > > > > > > > >>> > > > > > > these
> > > > > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log
> has
> > > the
> > > > > old
> > > > > > > > > producer
> > > > > > > > > > >>> ID.
> > > > > > > > > > >>> > We
> > > > > > > > > > >>> > > > > don't
> > > > > > > > > > >>> > > > > > > > write
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare
> > > marker's
> > > > > non
> > > > > > > > tagged
> > > > > > > > > > >>> fields.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server
> > > downgrades
> > > > > > now,
> > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > >>> > read
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > records
> > > > > > > > > > >>> > > > > > > > >> not
> > > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the
> > > complete
> > > > > > marker
> > > > > > > > > will
> > > > > > > > > > >>> also
> > > > > > > > > > >>> > > have
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > old
> > > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used
> the
> > > new
> > > > > > > producer
> > > > > > > > > ID,
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > would
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since
> > the
> > > > > > producer
> > > > > > > > id
> > > > > > > > > > >>> doesn't
> > > > > > > > > > >>> > > > match
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> > > > correct
> > > > > > on
> > > > > > > > the
> > > > > > > > > > data
> > > > > > > > > > >>> > > > > partition.)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow
> > case,
> > > > I'd
> > > > > > > expect
> > > > > > > > > the
> > > > > > > > > > >>> > > following
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > > >> happen on
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we
> retry
> > > > EndTxn
> > > > > > --
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > same
> > > > > > > > > > >>> > > > > > > producer
> > > > > > > > > > >>> > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the
> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't
> > > retry
> > > > > > > EndTxn
> > > > > > > > > and
> > > > > > > > > > >>> use
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> producer id
> > > > > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > > > > InvalidPidMappingException
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> > > > special
> > > > > > > > handling
> > > > > > > > > > for
> > > > > > > > > > >>> > when
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we
> could
> > > get
> > > > an
> > > > > > API
> > > > > > > > > > version
> > > > > > > > > > >>> > > request
> > > > > > > > > > >>> > > > > > > showing
> > > > > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In
> > that
> > > > > case,
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > >>> call
> > > > > > > > > > >>> > > > > > > > initProducerId
> > > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In
> the
> > > > > > overflow
> > > > > > > > > case,
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > > > > correctly
> > > > > > > > > > >>> > > > > > > > gives
> > > > > > > > > > >>> > > > > > > > >> > us a
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> > > > corresponding
> > > > > > case
> > > > > > > > > would
> > > > > > > > > > >>> be
> > > > > > > > > > >>> > > where
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> *complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the
> > endTxn
> > > > is
> > > > > > not
> > > > > > > > > > >>> received by
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > client
> > > > > > > > > > >>> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This
> > would
> > > > > > result
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old
> > > one.
> > > > > If
> > > > > > > the
> > > > > > > > > > client
> > > > > > > > > > >>> > > > retries,
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > InvalidPidMappingException.
> > > > > > > The
> > > > > > > > > > >>> > > InitProducerId
> > > > > > > > > > >>> > > > > > > > scenario
> > > > > > > > > > >>> > > > > > > > >> > above
> > > > > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > > > > compatibility
> > > > > > > > > story
> > > > > > > > > > is
> > > > > > > > > > >>> > meant
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > > > support
> > > > > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping
> the
> > > > > > > > transactional
> > > > > > > > > > >>> > > correctness.
> > > > > > > > > > >>> > > > > > > Keeping
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is
> > not
> > > > the
> > > > > > > > > priority.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I
> > can
> > > > > also
> > > > > > > add
> > > > > > > > > text
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > KIP
> > > > > > > > > > >>> > > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes
> > some
> > > > edge
> > > > > > > > cases.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 4:10 PM
> > > > > > > > > Jun
> > > > > > > > > > >>> Rao
> > > > > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the
> > > reply.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we
> > > don't
> > > > > need
> > > > > > > to
> > > > > > > > > > >>> optimize
> > > > > > > > > > >>> > for
> > > > > > > > > > >>> > > > > > fencing
> > > > > > > > > > >>> > > > > > > > >> during
> > > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding
> > > consistency,
> > > > > > there
> > > > > > > > are
> > > > > > > > > > two
> > > > > > > > > > >>> > > > possible
> > > > > > > > > > >>> > > > > > > cases:
> > > > > > > > > > >>> > > > > > > > >> (1)
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but
> > the
> > > > > endTxn
> > > > > > > > > > response
> > > > > > > > > > >>> is
> > > > > > > > > > >>> > not
> > > > > > > > > > >>> > > > > > > received
> > > > > > > > > > >>> > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > server
> downgrades;
> > > (2)
> > > > > > the
> > > > > > > > > > prepare
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > > written,
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by
> the
> > > > > client
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > >>> > server
> > > > > > > > > > >>> > > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> In
> > > > > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the
> old
> > > > > produce
> > > > > > Id
> > > > > > > > and
> > > > > > > > > > in
> > > > > > > > > > >>> (2),
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > > >>> > > > > > > > >> > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If
> we
> > > > > > downgrade
> > > > > > > > > right
> > > > > > > > > > >>> after
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to
> both
> > > (1)
> > > > > and
> > > > > > > (2)
> > > > > > > > > > >>> since we
> > > > > > > > > > >>> > > can
> > > > > > > > > > >>> > > > > only
> > > > > > > > > > >>> > > > > > > put
> > > > > > > > > > >>> > > > > > > > >> one
> > > > > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce
> > Id
> > > > > field.
> > > > > > > > It's
> > > > > > > > > > >>> also not
> > > > > > > > > > >>> > > > clear
> > > > > > > > > > >>> > > > > > > which
> > > > > > > > > > >>> > > > > > > > >> case
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could
> > probably
> > > > be
> > > > > > > > > consistent
> > > > > > > > > > >>> with
> > > > > > > > > > >>> > > > either
> > > > > > > > > > >>> > > > > > > case.
> > > > > > > > > > >>> > > > > > > > >> By
> > > > > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in
> the
> > > > > prepare
> > > > > > > > > marker,
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > are
> > > > > > > > > > >>> > > > > > > consistent
> > > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the
> > slight
> > > > > > benefit
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > >>> > > produce
> > > > > > > > > > >>> > > > > > field
> > > > > > > > > > >>> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker
> > are
> > > > > > > consistent
> > > > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > overflow
> > > > > > > > > > >>> > > > > > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > > 3:11 PM
> > > > > > > > > > >>> Justine
> > > > > > > > > > >>> > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case
> you
> > > > > > describe,
> > > > > > > we
> > > > > > > > > > would
> > > > > > > > > > >>> > need
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > have
> > > > > > > > > > >>> > > > > > > a
> > > > > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful
> > EndTxn,
> > > > > and a
> > > > > > > > > > >>> successful
> > > > > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> > > > request
> > > > > > go
> > > > > > > > > > through
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > a
> > > > > > > > > > >>> > > > > given
> > > > > > > > > > >>> > > > > > > > >> producer.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to
> > > figure
> > > > > out
> > > > > > > if
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > >>> > > possible
> > > > > > > > > > >>> > > > > for
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > client
> > > > > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous
> > request
> > > > is
> > > > > > > > delayed
> > > > > > > > > > >>> > somewhere.
> > > > > > > > > > >>> > > > But
> > > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > > >>> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence
> the
> > > > > client.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the
> > > overflow
> > > > > > case.
> > > > > > > > In
> > > > > > > > > > the
> > > > > > > > > > >>> > > overflow
> > > > > > > > > > >>> > > > > > case,
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are
> > > different
> > > > on
> > > > > > the
> > > > > > > > > > marker
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > on
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to
> > use
> > > > the
> > > > > > max
> > > > > > > > > epoch
> > > > > > > > > > >>> but
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> > > > epoch 0
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> > > > transactional
> > > > > > > > > > >>> > > > > > > state.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > > > > downgrade
> > > > > > > > case,
> > > > > > > > > we
> > > > > > > > > > >>> want
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > > > see
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the
> client
> > > will
> > > > > > have.
> > > > > > > > If
> > > > > > > > > we
> > > > > > > > > > >>> > > complete
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> commit,
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is
> > reloaded,
> > > > we
> > > > > > need
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > >>> > > > producer
> > > > > > > > > > >>> > > > > ID
> > > > > > > > > > >>> > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid
> > > producer
> > > > ID
> > > > > > > > > mapping.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server
> > > downgrade
> > > > > > cases
> > > > > > > > are
> > > > > > > > > > >>> > > considering
> > > > > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing
> > > from
> > > > > > > previous
> > > > > > > > > > >>> behavior
> > > > > > > > > > >>> > --
> > > > > > > > > > >>> > > > and
> > > > > > > > > > >>> > > > > > are
> > > > > > > > > > >>> > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> > > > safety
> > > > > > from
> > > > > > > > > > fencing
> > > > > > > > > > >>> > > retries
> > > > > > > > > > >>> > > > > (as
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> have
> > > > > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > > > > support).
> > > > > > > > > Perhaps
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > is
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > trade
> > > > > > > > > > >>> > > > > > > > off,
> > > > > > > > > > >>> > > > > > > > >> > but
> > > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > > > > downgrades,
> > > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > > > > > > restarted
> > > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > > >>> > > > > > > > >> > it
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new
> > > producer
> > > > ID
> > > > > > > too).
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan
> 12,
> > > 2024
> > > > > at
> > > > > > > > > 11:42 AM
> > > > > > > > > > >>> Jun
> > > > > > > > > > >>> > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for
> the
> > > > > reply.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If
> the
> > > > marker
> > > > > > is
> > > > > > > > > > written
> > > > > > > > > > >>> by
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> client, we
> > > > > > > > > > >>> > > > > > > > >> > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last
> email
> > > > > > guarantee
> > > > > > > > > that
> > > > > > > > > > >>> any
> > > > > > > > > > >>> > > EndTxn
> > > > > > > > > > >>> > > > > > > > requests
> > > > > > > > > > >>> > > > > > > > >> > with
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the
> same
> > > > > producer
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >>> same
> > > > > > > > > > >>> > > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> Then
> > > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a
> > fenced
> > > > > error
> > > > > > > but
> > > > > > > > > can
> > > > > > > > > > >>> handle
> > > > > > > > > > >>> > > > > > > gracefully
> > > > > > > > > > >>> > > > > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a
> delayed
> > > > > EndTnx
> > > > > > > > > request
> > > > > > > > > > is
> > > > > > > > > > >>> > > > > processed,
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> txn
> > > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next
> > > txn.
> > > > I
> > > > > > > guess
> > > > > > > > in
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > case
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > still
> > > > > > > > > > >>> > > > > > > > >> > return
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> > > > request?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry,
> my
> > > > > > question
> > > > > > > > was
> > > > > > > > > > >>> > > inaccurate.
> > > > > > > > > > >>> > > > > What
> > > > > > > > > > >>> > > > > > > you
> > > > > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The
> downgrade
> > > > > > > > > compatibility I
> > > > > > > > > > >>> > mention
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > that
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > keep
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in
> > the
> > > > > main
> > > > > > > > > > >>> (non-tagged)
> > > > > > > > > > >>> > > > fields
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> did
> > > > > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server
> side."
> > If
> > > > we
> > > > > > want
> > > > > > > > to
> > > > > > > > > do
> > > > > > > > > > >>> this,
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> that we
> > > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current
> > produce
> > > Id
> > > > > and
> > > > > > > max
> > > > > > > > > > >>> epoch in
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > existing
> > > > > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> producerEpoch
> > > > fields
> > > > > > for
> > > > > > > > > both
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > prepare
> > > > > > > > > > >>> > > > > > and
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The
> downgrade
> > > can
> > > > > > happen
> > > > > > > > > after
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > complete
> > > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you
> described,
> > > the
> > > > > > > > > downgraded
> > > > > > > > > > >>> > > > coordinator
> > > > > > > > > > >>> > > > > > will
> > > > > > > > > > >>> > > > > > > > see
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of
> the
> > > old
> > > > > > one.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan
> > 12,
> > > > 2024
> > > > > > at
> > > > > > > > > > 10:44 AM
> > > > > > > > > > >>> > > Justine
> > > > > > > > > > >>> > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can
> update
> > > the
> > > > > > > > > > description.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe
> > your
> > > > > > second
> > > > > > > > > point
> > > > > > > > > > is
> > > > > > > > > > >>> > > > mentioned
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it
> > is
> > > > > > helpful.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> > delayed
> > > > > > message
> > > > > > > > case
> > > > > > > > > > can
> > > > > > > > > > >>> > also
> > > > > > > > > > >>> > > > > > violate
> > > > > > > > > > >>> > > > > > > > EOS
> > > > > > > > > > >>> > > > > > > > >> if
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in
> > after
> > > > the
> > > > > > > next
> > > > > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > > > > >>> > > > > > > > >> request
> > > > > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> > > > message
> > > > > > > from a
> > > > > > > > > > >>> previous
> > > > > > > > > > >>> > > > > > (aborted)
> > > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the
> > marker
> > > is
> > > > > > > written
> > > > > > > > > by
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > new
> > > > > > > > > > >>> > > > > > client,
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > > > > guarantee
> > > > > > > > that
> > > > > > > > > > any
> > > > > > > > > > >>> > EndTxn
> > > > > > > > > > >>> > > > > > > requests
> > > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> > > > producer
> > > > > > and
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > >>> > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> Then we
> > > > > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced
> error
> > > but
> > > > > can
> > > > > > > > > handle
> > > > > > > > > > >>> > > gracefully
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > > >> described
> > > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't
> > think
> > > a
> > > > > > > boolean
> > > > > > > > is
> > > > > > > > > > >>> useful
> > > > > > > > > > >>> > > > since
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence
> or
> > > > lack
> > > > > of
> > > > > > > the
> > > > > > > > > > >>> tagged
> > > > > > > > > > >>> > > field
> > > > > > > > > > >>> > > > > > being
> > > > > > > > > > >>> > > > > > > > >> > written.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > prepare
> > > > > > marker
> > > > > > > we
> > > > > > > > > > will
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In
> > the
> > > > > > Complete
> > > > > > > > > state
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > may
> > > > > > > > > > >>> > > > not.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not
> sure
> > > why
> > > > > the
> > > > > > > > > ongoing
> > > > > > > > > > >>> state
> > > > > > > > > > >>> > > > > matters
> > > > > > > > > > >>> > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> this
> > > > > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not
> sure
> > > > what
> > > > > > you
> > > > > > > > are
> > > > > > > > > > >>> > referring
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > writing
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> > > > prepare
> > > > > > > > marker.
> > > > > > > > > > >>> This is
> > > > > > > > > > >>> > > not
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > > overflow
> > > > > > case,
> > > > > > > we
> > > > > > > > > > >>> write the
> > > > > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > > > > >>> > > > > > > > >> > in
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so
> > we
> > > > know
> > > > > > > what
> > > > > > > > we
> > > > > > > > > > >>> > assigned
> > > > > > > > > > >>> > > > when
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> reload
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> > > > complete,
> > > > > we
> > > > > > > > > > >>> transition
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > > ID
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> main
> > > > > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> > > > previous
> > > > > > > > producer
> > > > > > > > > > ID
> > > > > > > > > > >>> > field
> > > > > > > > > > >>> > > > > filled
> > > > > > > > > > >>> > > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> This
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry
> > > case
> > > > > the
> > > > > > > > > > operation
> > > > > > > > > > >>> > > > completed
> > > > > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer.
> > The
> > > > > > > downgrade
> > > > > > > > > > >>> > > compatibility
> > > > > > > > > > >>> > > > I
> > > > > > > > > > >>> > > > > > > > mention
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> ID
> > > and
> > > > > > epoch
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> main
> > > > > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > > > > >>> > > > > > > > >> > fields
> > > > > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on
> the
> > > > server
> > > > > > > side.
> > > > > > > > > If
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > server
> > > > > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > addresses
> > > > > both
> > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri,
> Jan
> > > 12,
> > > > > 2024
> > > > > > > at
> > > > > > > > > > >>> 10:21 AM
> > > > > > > > > > >>> > Jun
> > > > > > > > > > >>> > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi,
> > Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks
> for
> > > the
> > > > > > > reply.
> > > > > > > > > > Sorry
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > delay. I
> > > > > > > > > > >>> > > > > > > > >> > have a
> > > > > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I
> > think
> > > > the
> > > > > > > > > > motivation
> > > > > > > > > > >>> > > section
> > > > > > > > > > >>> > > > > > could
> > > > > > > > > > >>> > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > motivations
> > > > > listed
> > > > > > > by
> > > > > > > > > the
> > > > > > > > > > >>> KIP is
> > > > > > > > > > >>> > > > "This
> > > > > > > > > > >>> > > > > > can
> > > > > > > > > > >>> > > > > > > > >> happen
> > > > > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or
> delayed
> > > due
> > > > > to
> > > > > > > > > > networking
> > > > > > > > > > >>> > > issues
> > > > > > > > > > >>> > > > > or a
> > > > > > > > > > >>> > > > > > > > >> network
> > > > > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts,
> > and
> > > > then
> > > > > > the
> > > > > > > > > > delayed
> > > > > > > > > > >>> > > message
> > > > > > > > > > >>> > > > > > > finally
> > > > > > > > > > >>> > > > > > > > >> > comes
> > > > > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very
> > > accurate.
> > > > > > > Without
> > > > > > > > > > >>> KIP-890,
> > > > > > > > > > >>> > > > > > currently,
> > > > > > > > > > >>> > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts
> an
> > > > > ongoing
> > > > > > > > > > >>> transaction,
> > > > > > > > > > >>> > it
> > > > > > > > > > >>> > > > > > already
> > > > > > > > > > >>> > > > > > > > >> bumps
> > > > > > > > > > >>> > > > > > > > >> > up
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which
> > > prevents
> > > > > the
> > > > > > > > > delayed
> > > > > > > > > > >>> > produce
> > > > > > > > > > >>> > > > > > message
> > > > > > > > > > >>> > > > > > > > >> from
> > > > > > > > > > >>> > > > > > > > >> > > being
> > > > > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> partition.
> > > > What
> > > > > > can
> > > > > > > > > cause
> > > > > > > > > > a
> > > > > > > > > > >>> > > hanging
> > > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> completes
> > > > > (either
> > > > > > > > aborts
> > > > > > > > > > or
> > > > > > > > > > >>> > > > commits) a
> > > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> successful
> > > ack
> > > > > on
> > > > > > > > > messages
> > > > > > > > > > >>> > > published
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> same
> > > > > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible
> > for
> > > > the
> > > > > > > > delayed
> > > > > > > > > > >>> message
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > be
> > > > > > > > > > >>> > > > > > > > >> appended
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> > > > causing
> > > > > a
> > > > > > > > > > >>> transaction to
> > > > > > > > > > >>> > > > hang.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A
> similar
> > > > issue
> > > > > > (not
> > > > > > > > > > >>> mentioned
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker
> in
> > > the
> > > > > > > > > > coordinator's
> > > > > > > > > > >>> log.
> > > > > > > > > > >>> > > For
> > > > > > > > > > >>> > > > > > > > example,
> > > > > > > > > > >>> > > > > > > > >> > it's
> > > > > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > EndTxnRequest
> > > > to
> > > > > > be
> > > > > > > > > > delayed
> > > > > > > > > > >>> on
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > > > > >>> > > > > > > > >> > By
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > EndTxnRequest
> > > > is
> > > > > > > > > > processed,
> > > > > > > > > > >>> it's
> > > > > > > > > > >>> > > > > > possible
> > > > > > > > > > >>> > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> completed
> > > and
> > > > a
> > > > > > new
> > > > > > > > txn
> > > > > > > > > > has
> > > > > > > > > > >>> > > started.
> > > > > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > > > > >>> > > > > > > > >> > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped
> on
> > > > every
> > > > > > txn,
> > > > > > > > the
> > > > > > > > > > >>> delayed
> > > > > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > > > > >>> > > > > > > > >> > will
> > > > > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare
> > > marker
> > > > > > (and
> > > > > > > > > > >>> eventually a
> > > > > > > > > > >>> > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> marker)
> > > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't
> > cause
> > > > the
> > > > > > > > > > transaction
> > > > > > > > > > >>> to
> > > > > > > > > > >>> > > hang,
> > > > > > > > > > >>> > > > > but
> > > > > > > > > > >>> > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> > proposal
> > > > in
> > > > > > this
> > > > > > > > KIP
> > > > > > > > > > >>> will
> > > > > > > > > > >>> > > > address
> > > > > > > > > > >>> > > > > > this
> > > > > > > > > > >>> > > > > > > > >> issue
> > > > > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> > > > "However, I
> > > > > > was
> > > > > > > > > > >>> writing it
> > > > > > > > > > >>> > so
> > > > > > > > > > >>> > > > > that
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old
> > clients
> > > > > where
> > > > > > we
> > > > > > > > > don't
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > ability
> > > > > > > > > > >>> > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients
> > that
> > > > > can.
> > > > > > > (Old
> > > > > > > > > > >>> clients
> > > > > > > > > > >>> > > don't
> > > > > > > > > > >>> > > > > > bump
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for
> > sure
> > > > the
> > > > > > > write
> > > > > > > > > > >>> belongs
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > given
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I
> am
> > > > > > wondering
> > > > > > > > why
> > > > > > > > > > we
> > > > > > > > > > >>> need
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written
> by
> > > the
> > > > > old
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >>> new
> > > > > > > > > > >>> > > > client.
> > > > > > > > > > >>> > > > > > > Could
> > > > > > > > > > >>> > > > > > > > >> you
> > > > > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > differently
> > > if
> > > > > we
> > > > > > > know
> > > > > > > > > the
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > is
> > > > > > > > > > >>> > > > > > > written
> > > > > > > > > > >>> > > > > > > > >> by
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If
> > we
> > > do
> > > > > > need
> > > > > > > a
> > > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > >>> > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> whether
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old
> > and
> > > > the
> > > > > > new
> > > > > > > > > > client.
> > > > > > > > > > >>> > Would
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > > be
> > > > > > > > > > >>> > > > > > > > >> simpler
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field
> > > instead
> > > > of
> > > > > > > > > > indirectly
> > > > > > > > > > >>> > > through
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> previous
> > > > > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3
> It's
> > > not
> > > > > > clear
> > > > > > > > to
> > > > > > > > > me
> > > > > > > > > > >>> why
> > > > > > > > > > >>> > we
> > > > > > > > > > >>> > > > only
> > > > > > > > > > >>> > > > > > add
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > complete
> > > > > > marker,
> > > > > > > > but
> > > > > > > > > > >>> not in
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether
> a
> > > > marker
> > > > > > is
> > > > > > > > > > written
> > > > > > > > > > >>> by
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> or
> > > > > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do
> this
> > > > > > > > consistently
> > > > > > > > > > for
> > > > > > > > > > >>> all
> > > > > > > > > > >>> > > > > markers.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4
> What
> > > > about
> > > > > > the
> > > > > > > > > > >>> > > > > TransactionLogValue
> > > > > > > > > > >>> > > > > > > > >> record
> > > > > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state?
> > > Should
> > > > we
> > > > > > > also
> > > > > > > > > > >>> > distinguish
> > > > > > > > > > >>> > > > > > whether
> > > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new
> > client?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In
> > the
> > > > > > overflow
> > > > > > > > > case,
> > > > > > > > > > >>> it's
> > > > > > > > > > >>> > > > still
> > > > > > > > > > >>> > > > > > not
> > > > > > > > > > >>> > > > > > > > >> clear
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > me
> > > > > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> > > > produce
> > > > > > Id
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> > prepare
> > > > > > > > > > >>> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> while
> > > > > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> > > > complete
> > > > > > > > marker.
> > > > > > > > > > You
> > > > > > > > > > >>> > > > mentioned
> > > > > > > > > > >>> > > > > > that
> > > > > > > > > > >>> > > > > > > > >> it's
> > > > > > > > > > >>> > > > > > > > >> > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > > > > downgrade
> > > > > > > > with
> > > > > > > > > > >>> either
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> marker
> > > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In
> either
> > > > case,
> > > > > > the
> > > > > > > > > > >>> downgraded
> > > > > > > > > > >>> > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> should
> > > > > > > > > > >>> > > > > > > > >> > > see
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> (probably
> > > the
> > > > > > > previous
> > > > > > > > > > >>> produce
> > > > > > > > > > >>> > > Id),
> > > > > > > > > > >>> > > > > > right?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed,
> > Dec
> > > > 20,
> > > > > > 2023
> > > > > > > > at
> > > > > > > > > > >>> 6:00 PM
> > > > > > > > > > >>> > > > > Justine
> > > > > > > > > > >>> > > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey
> Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks
> > for
> > > > > > taking
> > > > > > > a
> > > > > > > > > look
> > > > > > > > > > >>> at
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > KIP
> > > > > > > > > > >>> > > > > > > again.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100.
> For
> > > the
> > > > > > epoch
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > case,
> > > > > > > > > > >>> > > > > only
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps
> > the
> > > > > > behavior
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >>> rest
> > > > > > > > > > >>> > > of
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> markers
> > > > > > > > > > >>> > > > > > > > >> > > where
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch
> of
> > > the
> > > > > > > > > transaction
> > > > > > > > > > >>> > > records +
> > > > > > > > > > >>> > > > > 1.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101.
> You
> > > are
> > > > > > > correct
> > > > > > > > > > that
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > don't
> > > > > > > > > > >>> > > > > > need
> > > > > > > > > > >>> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > write
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the
> > > same.
> > > > > > > > However,
> > > > > > > > > I
> > > > > > > > > > >>> was
> > > > > > > > > > >>> > > > writing
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old
> > > clients
> > > > > > where
> > > > > > > we
> > > > > > > > > > don't
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > ability
> > > > > > > > > > >>> > > > > > > > >> do
> > > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> clients
> > > that
> > > > > > can.
> > > > > > > > (Old
> > > > > > > > > > >>> clients
> > > > > > > > > > >>> > > > don't
> > > > > > > > > > >>> > > > > > > bump
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say
> for
> > > sure
> > > > > the
> > > > > > > > write
> > > > > > > > > > >>> belongs
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > given
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> > > > request
> > > > > > > from
> > > > > > > > a
> > > > > > > > > > new
> > > > > > > > > > >>> > > client,
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> fill
> > > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> guarantee
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that
> any
> > > > > EndTxn
> > > > > > > > > requests
> > > > > > > > > > >>> with
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > are
> > > > > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the
> same
> > > > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102.
> In
> > > > > prepare
> > > > > > > > phase,
> > > > > > > > > > we
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had.
> It
> > is
> > > > the
> > > > > > > > > producer
> > > > > > > > > > >>> ID and
> > > > > > > > > > >>> > > > epoch
> > > > > > > > > > >>> > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> are
> > > > > > > > > > >>> > > > > > > > >> > on
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase,
> > we
> > > > stay
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > >>> unless
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set
> the
> > > > > producer
> > > > > > > ID
> > > > > > > > to
> > > > > > > > > > >>> the new
> > > > > > > > > > >>> > > one
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > >> generated
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > complete.
> > > > This
> > > > > > is
> > > > > > > > for
> > > > > > > > > > >>> > downgrade
> > > > > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > > > > >>> > > > > > > > >> > > The
> > > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> > > > guards
> > > > > > for
> > > > > > > > > > retries
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > > > > > failovers.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> > prepare
> > > > > phase
> > > > > > > for
> > > > > > > > > > epoch
> > > > > > > > > > >>> > > > overflow
> > > > > > > > > > >>> > > > > > case
> > > > > > > > > > >>> > > > > > > > >> only
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID.
> This
> > > is
> > > > > for
> > > > > > > the
> > > > > > > > > case
> > > > > > > > > > >>> where
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > reload
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> prepare
> > > > state.
> > > > > > > Once
> > > > > > > > > the
> > > > > > > > > > >>> > > > transaction
> > > > > > > > > > >>> > > > > is
> > > > > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> > > > client
> > > > > > > > already
> > > > > > > > > is
> > > > > > > > > > >>> > using.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> > commit
> > > > > phase,
> > > > > > > we
> > > > > > > > > > store
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > previous
> > > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I
> think
> > it
> > > > is
> > > > > > > easier
> > > > > > > > > to
> > > > > > > > > > >>> think
> > > > > > > > > > >>> > of
> > > > > > > > > > >>> > > > it
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> how
> > > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > epoch
> > > > > > before,
> > > > > > > > with
> > > > > > > > > > >>> some
> > > > > > > > > > >>> > > extra
> > > > > > > > > > >>> > > > > > > > bookeeping
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> > > > fields.
> > > > > > We
> > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > >>> do
> > > > > > > > > > >>> > it
> > > > > > > > > > >>> > > > this
> > > > > > > > > > >>> > > > > > way
> > > > > > > > > > >>> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103.
> > Next
> > > > > > producer
> > > > > > > > ID
> > > > > > > > > is
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > > > prepare
> > > > > > > > > > >>> > > > > > > > status
> > > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > > > > complete.
> > > > > > > The
> > > > > > > > > > >>> reason why
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > need
> > > > > > > > > > >>> > > > > > > two
> > > > > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > backwards
> > > > > > > > > compatibility.
> > > > > > > > > > >>> We
> > > > > > > > > > >>> > need
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > keep
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > non-tagged
> > > > > field
> > > > > > > in
> > > > > > > > > case
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > > > downgrade.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104.
> We
> > > set
> > > > > the
> > > > > > > > fields
> > > > > > > > > > as
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > do
> > > > > > > > > > >>> > > in
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do
> this
> > > for
> > > > > > > > > > compatibility
> > > > > > > > > > >>> --
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > non-tagged
> > > > > > fields)
> > > > > > > > It
> > > > > > > > > > >>> will be
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > old
> > > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > > >>> > > > > > > > >> > ID
> > > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope
> > this
> > > > > helps.
> > > > > > > Let
> > > > > > > > > me
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > > you
> > > > > > > > > > >>> > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > further
> > > > > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On
> Wed,
> > > Dec
> > > > > 20,
> > > > > > > 2023
> > > > > > > > > at
> > > > > > > > > > >>> > 3:33 PM
> > > > > > > > > > >>> > > > Jun
> > > > > > > > > > >>> > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> > > > Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It
> > seems
> > > > > that
> > > > > > > you
> > > > > > > > > have
> > > > > > > > > > >>> made
> > > > > > > > > > >>> > > some
> > > > > > > > > > >>> > > > > > > changes
> > > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > particular,
> > > > > we
> > > > > > > are
> > > > > > > > > > >>> changing
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > format
> > > > > > > > > > >>> > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> TransactionLogValue.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > comments
> > > > > > related
> > > > > > > > to
> > > > > > > > > > >>> that.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100.
> > > Just
> > > > to
> > > > > > be
> > > > > > > > > clear.
> > > > > > > > > > >>> The
> > > > > > > > > > >>> > > > > overflow
> > > > > > > > > > >>> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > generated)
> > > > > is
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > >>> > current
> > > > > > > > > > >>> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> equals
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101.
> > For
> > > > the
> > > > > > > "not
> > > > > > > > > > epoch
> > > > > > > > > > >>> > > > overflow"
> > > > > > > > > > >>> > > > > > > case,
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > write
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> tagged
> > > > field
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> complete
> > > > > > > > > > >>> > > > > phase.
> > > > > > > > > > >>> > > > > > Do
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > need
> > > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > doesn't
> > > > > change
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > >>> case?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102.
> > It
> > > > > seems
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > >>> > meaning
> > > > > > > > > > >>> > > > for
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > ProducerId/ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > TransactionLogValue
> > > > > > > > > > >>> changes
> > > > > > > > > > >>> > > > > > depending
> > > > > > > > > > >>> > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > > TransactionStatus
> > > > > > > > > > is
> > > > > > > > > > >>> > > > ongoing,
> > > > > > > > > > >>> > > > > > they
> > > > > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > current
> > > > > > > > > > >>> ProducerEpoch.
> > > > > > > > > > >>> > > When
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > > > > >>> > > they
> > > > > > > > > > >>> > > > > > > > represent
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > > > > ProducerEpoch.
> > > > > > > > > > >>> When the
> > > > > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > further
> > > > > depend
> > > > > > > on
> > > > > > > > > > >>> whether
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> overflows
> > > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > overflow,
> > > > > they
> > > > > > > > > > represent
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> (max).
> > > > > > > Otherwise,
> > > > > > > > > they
> > > > > > > > > > >>> > > represent
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > newly
> > > > > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > ProducerEpoch
> > > > > > of
> > > > > > > > 0.
> > > > > > > > > Is
> > > > > > > > > > >>> that
> > > > > > > > > > >>> > > > right?
> > > > > > > > > > >>> > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> seems
> > > > > > > > > > >>> > > > > > > > >> > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> > > provide
> > > > > > some
> > > > > > > > > > examples
> > > > > > > > > > >>> > like
> > > > > > > > > > >>> > > > what
> > > > > > > > > > >>> > > > > > > Artem
> > > > > > > > > > >>> > > > > > > > >> has
> > > > > > > > > > >>> > > > > > > > >> > > done
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > considered a
> > > > > > > > simpler
> > > > > > > > > > >>> design
> > > > > > > > > > >>> > > > where
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> ProducerId/ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > represent
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > same
> > > > > value
> > > > > > > > (e.g.
> > > > > > > > > > >>> for the
> > > > > > > > > > >>> > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > TransactionStatus
> > > > > > > > > and
> > > > > > > > > > >>> epoch
> > > > > > > > > > >>> > > > > > overflow?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103.
> > > It's
> > > > > not
> > > > > > > > clear
> > > > > > > > > to
> > > > > > > > > > >>> me
> > > > > > > > > > >>> > why
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > need
> > > > > > > > > > >>> > > > > > > 3
> > > > > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > PrevProducerId,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > NextProducerId.
> > > > > > > > > Could
> > > > > > > > > > we
> > > > > > > > > > >>> > just
> > > > > > > > > > >>> > > > have
> > > > > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104.
> > For
> > > > > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > > > > >>> > > > > if
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> set
> > > the
> > > > > > > > > producerId
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> >
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Calvin Liu <ca...@confluent.io.INVALID>]

Hi Jun,
Thanks for the feedback, will update the name as suggested.
The default values are
add.partitions.to.txn.retry.backoff.ms=20
-- This is because the client side AddPartitionsToTxn retry backoff is 20ms.
add.partitions.to.txn.retry.backoff.max.ms=100
-- We want this value to be small to avoid client side timeout and retry,
but also long enough for the markers to propagate.


On Tue, Feb 11, 2025 at 2:36 PM Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Calvin,
>
> Thanks for the update. The change looks good to me. Some minor comments.
>
> For retry timeout, our convention is to use retry.backoff.max.ms and
> retry.backoff.ms. So, to be consistent, the new configurations should be
> add.partitions.to.txn.retry.backoff.max.ms and
> add.partitions.to.txn.retry.backoff.ms.
>
> Also, could you specify the defaults for these two values?
>
> Thanks,
>
> Jun
>
>
>
>
> On Mon, Feb 10, 2025 at 2:45 PM Calvin Liu <ca...@confluent.io.invalid>
> wrote:
>
> > Hey team,
> > In order to address the performance issue in the
> > https://issues.apache.org/jira/browse/KAFKA-18654. We are making a
> change
> > to retry the AddPartitionToTxn requests on the server side when hitting
> the
> > CONCURRENT_TRANSACTION error. This error is common after the transaction
> is
> > committed but before the markers are fully propagated to all the involved
> > partitions. With the server side retry, the client can avoid the frequent
> > and expensive retry for the produce request. In order to better control
> the
> > retry behavior, we proposed to add 2 following configs.
> >
> >    - *add.partitions.to.txn.max.timeout.ms
> >    <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum
> > retry
> >    timeout when the server attempts to add the partition to the
> > transaction.
> >    - *add.partitions.to.txn.retry.backoff.ms
> >    <http://add.partitions.to.txn.retry.backoff.ms>* defines how
> frequently
> >    the server will retry the AddPartitionToTxn.
> >
> > Thanks!
> >
> > On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan
> > <jo...@confluent.io.invalid>
> > wrote:
> >
> > > Hey folks, just wanted to share another quick update with KIP-890.
> > >
> > > Making the change to bump epoch after every transaction means that we
> no
> > > longer need to call InitProducerId during the life of the producer to
> > fence
> > > requests from previous transactions. With KIP-890 part 2, the new
> client
> > > will no longer call InitProducerId except on startup to fence a
> previous
> > > instance.
> > >
> > > With this change, some other calls to InitProducerId were inspected
> > > including the call after receiving an InvalidPidMappingException. This
> > > exception was changed to abortable as part of KIP-360: Improve
> > reliability
> > > of idempotent/transactional producer
> > > <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> > > >.
> > > However, this change means that we can violate EOS guarantees. As an
> > > example:
> > >
> > > Consider an application that is copying data from one partition to
> > another
> > >
> > >    - Application instance A processes to offset 4
> > >    - Application instance B comes up and fences application instance A
> > >    - Application instance B processes to offset 5
> > >    - Application instances A and B are idle for
> > > transaction.id.expiration.ms,
> > >    transaction id expires on server
> > >    - Application instance A attempts to process offset 5 (since in its
> > >    view, that is next) -- if we recover from invalid pid mapping, we
> can
> > >    duplicate this processing
> > >
> > > Thus, INVALID_PID_MAPPING should be fatal to the producer and as part
> of
> > > KIP-890 the new client in 4.0 will treat the exception as fatal.
> > >
> > > This is consistent with KIP-1050: Consistent error handling for
> > > Transactions
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> > > >
> > > where
> > > errors that are fatal to the producer are in the "application
> > recoverable"
> > > category. This is a grouping that indicates to the client that the
> > producer
> > > needs to restart and recovery on the application side is necessary.
> > > KIP-1050 is approved so we are consistent with that decision.
> > >
> > > Let me know if there are any questions. I've updated the KIP to reflect
> > > this change.
> > >
> > > Justine
> > >
> > > On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >
> > > > Sounds good, Justine. It would be useful to document that in the KIP.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > > > <jo...@confluent.io.invalid>
> > > > wrote:
> > > >
> > > > > The client will send the newest EndTxn request version if and only
> if
> > > > both
> > > > > the client and the server support kip-890 part 2.
> > > > > We set the value in the record based on the EndTxn version.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi, Justine,
> > > > > >
> > > > > > Thanks for the reply.
> > > > > >
> > > > > > 120. If the broker sends TV Y for the finalized version in
> > > > > > ApiVersionResponse, but the client doesn't support Y, how does
> the
> > > > broker
> > > > > > know the TV that the client supports?
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > > > <jo...@confluent.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Jun,
> > > > > > >
> > > > > > > No worries. Work on this KIP has been blocked for a bit anyways
> > --
> > > > > > catching
> > > > > > > up and rereading what I wrote :)
> > > > > > >
> > > > > > > 120. ClientTransactionProtocolVersion is the transaction
> version
> > as
> > > > > > defined
> > > > > > > by the highest transaction version (feature version value)
> > > supported
> > > > by
> > > > > > the
> > > > > > > client and the server. This works by the broker sending an
> > > > > > > ApiVersionsRequest to the client with the finalized version.
> > > Assuming
> > > > > > > kip-890 part 2 is enabled by transaction version Y, if this
> > request
> > > > > > > contains finalized version Y and the client has the logic to
> set
> > > this
> > > > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2
> not
> > > > > enable)
> > > > > > > the client will send Y - 1, even though the client has the
> > ability
> > > to
> > > > > > > support kip-890 part 2.
> > > > > > >
> > > > > > > 121. You are correct that this is not needed. However,
> currently
> > > that
> > > > > > field
> > > > > > > is already being set in memory -- just not written to disk. I
> > think
> > > > it
> > > > > is
> > > > > > > ok to write it to disk though. Let me know if you think
> > otherwise.
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao
> <jun@confluent.io.invalid
> > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Justine,
> > > > > > > >
> > > > > > > > Thanks for the update and sorry for the late reply.
> > > > > > > >
> > > > > > > > 120. I am wondering what value is used for
> > > > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > > > EndTxnRequest?
> > > > > > > >
> > > > > > > > 121. Earlier, you made the change to set lastProducerId in
> > > PREPARE
> > > > to
> > > > > > > > indicate that the marker is written for the new client. With
> > the
> > > > new
> > > > > > > > ClientTransactionProtocolVersion field, it seems this is no
> > > longer
> > > > > > > > necessary.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi there -- another update!
> > > > > > > > >
> > > > > > > > > When looking into the implementation for the safe epoch
> > bumps I
> > > > > > > realized
> > > > > > > > > that we are already populating previousProducerID in memory
> > as
> > > > part
> > > > > > of
> > > > > > > > > KIP-360.
> > > > > > > > > If we are to start using flexible fields, it is better to
> > > always
> > > > > use
> > > > > > > this
> > > > > > > > > information and have an explicit (tagged) field to indicate
> > > > whether
> > > > > > the
> > > > > > > > > client supports KIP-890 part 2.
> > > > > > > > >
> > > > > > > > > I've included the extra field and how it is set in the KIP.
> > > I've
> > > > > also
> > > > > > > > > updated the KIP to explain that we will be setting the
> tagged
> > > > > fields
> > > > > > > when
> > > > > > > > > they are available for all transitions.
> > > > > > > > >
> > > > > > > > > Finally, I added clearer text about the transaction
> protocol
> > > > > versions
> > > > > > > > > included with this KIP. 1 for flexible transaction state
> > > records
> > > > > and
> > > > > > 2
> > > > > > > > for
> > > > > > > > > KIP-890 part 2 enablement.
> > > > > > > > >
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > > > jolshan@confluent.io
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey there -- small update to the KIP,
> > > > > > > > > >
> > > > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > > > > > TxnOffsetCommit
> > > > > > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > > > AbortableTransactionException to match the pattern we had
> > for
> > > > > other
> > > > > > > > > errors.
> > > > > > > > > > I also mentioned bumping all 6 transactional APIs so we
> can
> > > > > future
> > > > > > > > > > proof/support the error on the client going forward. If a
> > > > future
> > > > > > > change
> > > > > > > > > > wants to have an error scenario that requires us to abort
> > the
> > > > > > > > > transaction,
> > > > > > > > > > we can rely on the 3.8+ clients to support it. We ran
> into
> > > > issues
> > > > > > > > finding
> > > > > > > > > > good/generic error codes that older clients could support
> > > while
> > > > > > > working
> > > > > > > > > on
> > > > > > > > > > this KIP, so this should help in the future.
> > > > > > > > > >
> > > > > > > > > > The features discussion is still ongoing in KIP-1022.
> Will
> > > > update
> > > > > > > again
> > > > > > > > > > here when that concludes.
> > > > > > > > > >
> > > > > > > > > > Justine
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > > > jolshan@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> I don't think AddPartitions is a good example since we
> > > > currenly
> > > > > > > don't
> > > > > > > > > >> gate the version on TV or MV. (We only set a different
> > flag
> > > > > > > depending
> > > > > > > > on
> > > > > > > > > >> the TV)
> > > > > > > > > >>
> > > > > > > > > >> Even if we did want to gate it on TV, I think the idea
> is
> > to
> > > > > move
> > > > > > > away
> > > > > > > > > >> from MV gating inter broker protocols. Ideally we can
> get
> > > to a
> > > > > > state
> > > > > > > > > where
> > > > > > > > > >> MV is just used for metadata changes.
> > > > > > > > > >>
> > > > > > > > > >> I think some of this discussion might fit more with the
> > > > feature
> > > > > > > > version
> > > > > > > > > >> KIP, so I can try to open that up soon. Until we settle
> > > that,
> > > > > some
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >> work in KIP-890 is blocked.
> > > > > > > > > >>
> > > > > > > > > >> Justine
> > > > > > > > > >>
> > > > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > > > <jun@confluent.io.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi, Justine,
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > >>>
> > > > > > > > > >>> Since AddPartitions is an inter broker request, will
> its
> > > > > version
> > > > > > be
> > > > > > > > > gated
> > > > > > > > > >>> only by TV or other features like MV too? For example,
> if
> > > we
> > > > > need
> > > > > > > to
> > > > > > > > > >>> change
> > > > > > > > > >>> the protocol for AddPartitions for reasons other than
> txn
> > > > > > > > verification
> > > > > > > > > in
> > > > > > > > > >>> the future, will the new version be gated by a new MV?
> If
> > > so,
> > > > > > does
> > > > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > > > >>>
> > > > > > > > > >>> Jun
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > > > involved,
> > > > > > only
> > > > > > > > the
> > > > > > > > > >>> > transactional records that should only be gated by
> TV)
> > > > > > > > > >>> > Another TV gates the ability to turn on kip-890 part
> 2.
> > > > This
> > > > > > > would
> > > > > > > > > >>> gate the
> > > > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > > > transactions),
> > > > > > > > and
> > > > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the
> > version
> > > > is
> > > > > > > > already
> > > > > > > > > >>> used
> > > > > > > > > >>> > without TV.
> > > > > > > > > >>> >
> > > > > > > > > >>> > I think the only concern is the Produce request and
> we
> > > > could
> > > > > > > > consider
> > > > > > > > > >>> work
> > > > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > > > >>> >
> > > > > > > > > >>> > Justine
> > > > > > > > > >>> >
> > > > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > > > <jun@confluent.io.invalid
> > > > > > > >
> > > > > > > > > >>> wrote:
> > > > > > > > > >>> >
> > > > > > > > > >>> > > Hi, Justine,
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> > > forward,
> > > > > will
> > > > > > > > those
> > > > > > > > > >>> > > PRC/record protocols only be guarded by TV and not
> by
> > > > other
> > > > > > > > > features
> > > > > > > > > >>> like
> > > > > > > > > >>> > > MV?
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > Thanks,
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > Jun
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > wrote:
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > > Hi Jun,
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > Sorry I think I misunderstood your question or
> > > answered
> > > > > > > > > >>> incorrectly.
> > > > > > > > > >>> > The
> > > > > > > > > >>> > > TV
> > > > > > > > > >>> > > > version should ideally be fully independent from
> > MV.
> > > > > > > > > >>> > > > At least for the changes I proposed, TV should
> not
> > > > affect
> > > > > > MV
> > > > > > > > and
> > > > > > > > > MV
> > > > > > > > > >>> > > should
> > > > > > > > > >>> > > > not affect TV/
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > I think if we downgrade TV, only that feature
> > should
> > > > > > > downgrade.
> > > > > > > > > >>> > Likewise
> > > > > > > > > >>> > > > the same with MV. The finalizedFeatures should
> just
> > > > > reflect
> > > > > > > the
> > > > > > > > > >>> feature
> > > > > > > > > >>> > > > downgrade we made.
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > I also plan to write a new KIP for managing the
> > disk
> > > > > format
> > > > > > > and
> > > > > > > > > >>> upgrade
> > > > > > > > > >>> > > > tool as we will need new flags to support these
> > > > features.
> > > > > > > That
> > > > > > > > > >>> should
> > > > > > > > > >>> > > help
> > > > > > > > > >>> > > > clarify some things.
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > Justine
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > > > <jun@confluent.io.invalid
> > > > > > > > > >
> > > > > > > > > >>> > > wrote:
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > > Hi, Justine,
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> > > downgrade
> > > > > > > another
> > > > > > > > > >>> feature
> > > > > > > > > >>> > > > (say
> > > > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would
> we
> > > > > return
> > > > > > > for
> > > > > > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse
> > in
> > > > that
> > > > > > > case?
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > Thanks,
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > Jun
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > wrote:
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > > Hey Jun,
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > > > (transaction
> > > > > > > > > version)
> > > > > > > > > >>> we
> > > > > > > > > >>> > > will
> > > > > > > > > >>> > > > > stop
> > > > > > > > > >>> > > > > > using the add partitions to txn optimization
> > and
> > > > stop
> > > > > > > > writing
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > flexible
> > > > > > > > > >>> > > > > > feature version of the log.
> > > > > > > > > >>> > > > > > In the compatibility section I included some
> > > > > > explanations
> > > > > > > > on
> > > > > > > > > >>> how
> > > > > > > > > >>> > this
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > done.
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > Thanks,
> > > > > > > > > >>> > > > > > Justine
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > > > >>> > > > > wrote:
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > If we ever downgrade the transaction
> feature,
> > > any
> > > > > > > feature
> > > > > > > > > >>> > depending
> > > > > > > > > >>> > > > on
> > > > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > > > >>> > > > > > >
> > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > > > changes
> > > > > > > > > made
> > > > > > > > > >>> in
> > > > > > > > > >>> > > > KIP-890
> > > > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > Jun
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine
> > Olshan
> > > > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > >>> > > > > > > wrote:
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > > > questions
> > > > > > > about
> > > > > > > > > >>> MV/IBP.
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > Looking at the options, I think it makes
> > the
> > > > most
> > > > > > > sense
> > > > > > > > > to
> > > > > > > > > >>> > > create a
> > > > > > > > > >>> > > > > > > > separate feature for transactions and use
> > > that
> > > > to
> > > > > > > > version
> > > > > > > > > >>> gate
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > features
> > > > > > > > > >>> > > > > > > > we need to version gate (flexible
> > > transactional
> > > > > > state
> > > > > > > > > >>> records
> > > > > > > > > >>> > and
> > > > > > > > > >>> > > > > using
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > new protocol)
> > > > > > > > > >>> > > > > > > > I've updated the KIP to include this
> > change.
> > > > > > > Hopefully
> > > > > > > > > >>> that's
> > > > > > > > > >>> > > > > > everything
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > Justine
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> > > Olshan
> > > > <
> > > > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > I will update the KIP with the prev
> field
> > > for
> > > > > > > prepare
> > > > > > > > > as
> > > > > > > > > >>> > well.
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> field): x
> > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> or z
> > > if
> > > > y
> > > > > > will
> > > > > > > > > >>> overflow
> > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> field): x
> > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we
> > overflowed
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > > > >>> > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > > > >>> > > > > > > > >> (1) My point was that the coordinator
> > > could
> > > > > fail
> > > > > > > > right
> > > > > > > > > >>> after
> > > > > > > > > >>> > > > > writing
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> > > coordinator
> > > > > > > > generates
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > complete
> > > > > > > > > >>> > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> after the failover, it needs some
> field
> > > from
> > > > > the
> > > > > > > > > prepare
> > > > > > > > > >>> > > marker
> > > > > > > > > >>> > > > to
> > > > > > > > > >>> > > > > > > > >> determine whether it's written by the
> > new
> > > > > > client.
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> (2) The changing of the behavior
> sounds
> > > good
> > > > > to
> > > > > > > me.
> > > > > > > > We
> > > > > > > > > >>> only
> > > > > > > > > >>> > > want
> > > > > > > > > >>> > > > > to
> > > > > > > > > >>> > > > > > > > return
> > > > > > > > > >>> > > > > > > > >> success if the prepare state is
> written
> > by
> > > > the
> > > > > > new
> > > > > > > > > >>> client.
> > > > > > > > > >>> > So,
> > > > > > > > > >>> > > > in
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we
> also
> > > > need
> > > > > > sth
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>> > > prepare
> > > > > > > > > >>> > > > > > > marker
> > > > > > > > > >>> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> tell us whether it's written by the
> new
> > > > > client.
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That
> > > sounds
> > > > > > good
> > > > > > > to
> > > > > > > > > me.
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> Jun
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM
> Justine
> > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually
> have
> > > two
> > > > > > > > questions.
> > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > need
> > > to
> > > > > > write
> > > > > > > > the
> > > > > > > > > >>> > > previous
> > > > > > > > > >>> > > > > > > produce
> > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > >>> > > > > > > > >> > tagged field in the end maker so
> that
> > we
> > > > > know
> > > > > > if
> > > > > > > > the
> > > > > > > > > >>> > marker
> > > > > > > > > >>> > > is
> > > > > > > > > >>> > > > > > from
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> new
> > > > > > > > > >>> > > > > > > > >> > client. Since the end maker is
> derived
> > > > from
> > > > > > the
> > > > > > > > > >>> prepare
> > > > > > > > > >>> > > > marker,
> > > > > > > > > >>> > > > > > > should
> > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > >>> > > > > > > > >> > write the previous produce Id in the
> > > > prepare
> > > > > > > > marker
> > > > > > > > > >>> field
> > > > > > > > > >>> > > too?
> > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > >>> > > > > > > > >> > we will lose this information when
> > > > deriving
> > > > > > the
> > > > > > > > end
> > > > > > > > > >>> > marker.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in the
> > > > normal
> > > > > > > > producer
> > > > > > > > > >>> ID
> > > > > > > > > >>> > > field.
> > > > > > > > > >>> > > > > So
> > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > >>> > > > > > > > >> > need it in prepare and that was
> always
> > > the
> > > > > > plan.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> > > > > enumerate
> > > > > > > the
> > > > > > > > > >>> fields
> > > > > > > > > >>> > and
> > > > > > > > > >>> > > > add
> > > > > > > > > >>> > > > > > > them
> > > > > > > > > >>> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch
> y.
> > > > When
> > > > > we
> > > > > > > > > >>> overflow
> > > > > > > > > >>> > > epoch
> > > > > > > > > >>> > > > y
> > > > > > > > > >>> > > > > we
> > > > > > > > > >>> > > > > > > get
> > > > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > field):
> > > > > empty
> > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > or
> > > z
> > > > > if y
> > > > > > > > will
> > > > > > > > > >>> > overflow
> > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > field):
> > > x
> > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> > > overflowed
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we
> retry
> > > and
> > > > > see
> > > > > > > > epoch
> > > > > > > > > -
> > > > > > > > > >>> 1 +
> > > > > > > > > >>> > ID
> > > > > > > > > >>> > > > in
> > > > > > > > > >>> > > > > > last
> > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > >>> > > > > > > > >> > fields and are issuing the same
> > command
> > > > (ie
> > > > > > > commit
> > > > > > > > > not
> > > > > > > > > >>> > > abort),
> > > > > > > > > >>> > > > > we
> > > > > > > > > >>> > > > > > > > return
> > > > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890
> > seems
> > > to
> > > > > > > return
> > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> > > making
> > > > > this
> > > > > > > > > change?
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer
> if
> > > the
> > > > > > epoch
> > > > > > > is
> > > > > > > > > >>> bumped
> > > > > > > > > >>> > > and
> > > > > > > > > >>> > > > we
> > > > > > > > > >>> > > > > > > get a
> > > > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are
> > intentionally
> > > > > > adding
> > > > > > > > this
> > > > > > > > > >>> to
> > > > > > > > > >>> > > > prevent
> > > > > > > > > >>> > > > > > > > fencing.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > 112. We already merged the code that
> > > adds
> > > > > the
> > > > > > > > > >>> VerifyOnly
> > > > > > > > > >>> > > field
> > > > > > > > > >>> > > > > in
> > > > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is
> an
> > > > inter
> > > > > > > > broker
> > > > > > > > > >>> > request.
> > > > > > > > > >>> > > > It
> > > > > > > > > >>> > > > > > > seems
> > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that.
> Do
> > > you
> > > > > > know
> > > > > > > > why?
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> > > interbroker
> > > > > > > requests
> > > > > > > > > as
> > > > > > > > > >>> > > > > ApiVersions
> > > > > > > > > >>> > > > > > > > should
> > > > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> > > errors
> > > > > > > > correctly
> > > > > > > > > >>> if we
> > > > > > > > > >>> > > > > receive
> > > > > > > > > >>> > > > > > > them
> > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun
> > Rao
> > > > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually
> > have
> > > > two
> > > > > > > > > questions.
> > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > need
> > > to
> > > > > > write
> > > > > > > > the
> > > > > > > > > >>> > > previous
> > > > > > > > > >>> > > > > > > produce
> > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so
> > that
> > > we
> > > > > > know
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > >>> > > marker
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > > from
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > new
> > > > > > > > > >>> > > > > > > > >> > > client. Since the end maker is
> > derived
> > > > > from
> > > > > > > the
> > > > > > > > > >>> prepare
> > > > > > > > > >>> > > > > marker,
> > > > > > > > > >>> > > > > > > > >> should we
> > > > > > > > > >>> > > > > > > > >> > > write the previous produce Id in
> the
> > > > > prepare
> > > > > > > > > marker
> > > > > > > > > >>> > field
> > > > > > > > > >>> > > > too?
> > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > >>> > > > > > > > >> > > we will lose this information when
> > > > > deriving
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > >>> > > marker.
> > > > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we
> > retry
> > > > and
> > > > > > see
> > > > > > > > > epoch
> > > > > > > > > >>> - 1
> > > > > > > > > >>> > +
> > > > > > > > > >>> > > ID
> > > > > > > > > >>> > > > > in
> > > > > > > > > >>> > > > > > > last
> > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> > > command
> > > > > (ie
> > > > > > > > commit
> > > > > > > > > >>> not
> > > > > > > > > >>> > > > abort),
> > > > > > > > > >>> > > > > > we
> > > > > > > > > >>> > > > > > > > >> return
> > > > > > > > > >>> > > > > > > > >> > > success. The logic before KIP-890
> > > seems
> > > > to
> > > > > > > > return
> > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > >>> > > > > > > > >> > > in this case. Are we intentionally
> > > > making
> > > > > > this
> > > > > > > > > >>> change?
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > 112. We already merged the code
> that
> > > > adds
> > > > > > the
> > > > > > > > > >>> VerifyOnly
> > > > > > > > > >>> > > > field
> > > > > > > > > >>> > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which
> is
> > an
> > > > > inter
> > > > > > > > > broker
> > > > > > > > > >>> > > request.
> > > > > > > > > >>> > > > > It
> > > > > > > > > >>> > > > > > > > seems
> > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for
> that.
> > Do
> > > > you
> > > > > > > know
> > > > > > > > > why?
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> > > Justine
> > > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen"
> to
> > > > > "current
> > > > > > > > > >>> producer id
> > > > > > > > > >>> > > and
> > > > > > > > > >>> > > > > > > epoch"
> > > > > > > > > >>> > > > > > > > if
> > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am
> > still
> > > > > > > > finalizing
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > design.
> > > > > > > > > >>> > > > > I
> > > > > > > > > >>> > > > > > > will
> > > > > > > > > >>> > > > > > > > >> > update
> > > > > > > > > >>> > > > > > > > >> > > > the KIP once I get the
> information
> > > > > > > finalized.
> > > > > > > > > >>> Sorry
> > > > > > > > > >>> > for
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > delays.
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM
> > Jun
> > > > Rao
> > > > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow
> case,
> > > the
> > > > > > > previous
> > > > > > > > > ID
> > > > > > > > > >>> is
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > same
> > > > > > > > > >>> > > > > > as
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker
> too,
> > > but
> > > > we
> > > > > > set
> > > > > > > > the
> > > > > > > > > >>> > > previous
> > > > > > > > > >>> > > > ID
> > > > > > > > > >>> > > > > > in
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned
> > that
> > > > > this
> > > > > > is
> > > > > > > > to
> > > > > > > > > >>> know
> > > > > > > > > >>> > > that
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > >>> > > > > > > > >> > > > > written by the new client so
> > that
> > > we
> > > > > > could
> > > > > > > > > >>> return
> > > > > > > > > >>> > > > success
> > > > > > > > > >>> > > > > on
> > > > > > > > > >>> > > > > > > > >> retried
> > > > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was
> trying
> > > to
> > > > > > > > understand
> > > > > > > > > >>> why
> > > > > > > > > >>> > > this
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > not
> > > > > > > > > >>> > > > > > > > >> needed
> > > > > > > > > >>> > > > > > > > >> > > for
> > > > > > > > > >>> > > > > > > > >> > > > > the prepare marker since retry
> > can
> > > > > > happen
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>> > > prepare
> > > > > > > > > >>> > > > > > state
> > > > > > > > > >>> > > > > > > > >> too.
> > > > > > > > > >>> > > > > > > > >> > Is
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare
> > state,
> > > we
> > > > > > > return
> > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> > > > > requests?
> > > > > > > If
> > > > > > > > > so,
> > > > > > > > > >>> > should
> > > > > > > > > >>> > > > we
> > > > > > > > > >>> > > > > > > change
> > > > > > > > > >>> > > > > > > > >> "If
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID
> in
> > > last
> > > > > > seen
> > > > > > > > > >>> fields and
> > > > > > > > > >>> > > are
> > > > > > > > > >>> > > > > > > issuing
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > same
> > > > > > > > > >>> > > > > > > > >> > > > > command (ie commit not abort)
> we
> > > can
> > > > > > > return
> > > > > > > > > >>> (with
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > epoch)"
> > > > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a
> > > > delayed
> > > > > > > > endMaker
> > > > > > > > > >>> > request
> > > > > > > > > >>> > > > > > carries
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced.
> This
> > > can
> > > > > > > > > >>> commit/abort a
> > > > > > > > > >>> > > > future
> > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if
> > we
> > > > have
> > > > > > > seen
> > > > > > > > > >>> this in
> > > > > > > > > >>> > > > > practice
> > > > > > > > > >>> > > > > > > > >> though.
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be
> > > useful
> > > > > to
> > > > > > > make
> > > > > > > > > it
> > > > > > > > > >>> > clear
> > > > > > > > > >>> > > > that
> > > > > > > > > >>> > > > > > we
> > > > > > > > > >>> > > > > > > > can
> > > > > > > > > >>> > > > > > > > >> now
> > > > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field
> from
> > > the
> > > > > log
> > > > > > > > > >>> reliably.
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > >>> > > > > since
> > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > >>> > > > > > > > >> > > called
> > > > > > > > > >>> > > > > > > > >> > > > > across brokers and we are
> > changing
> > > > its
> > > > > > > > schema.
> > > > > > > > > >>> Are
> > > > > > > > > >>> > you
> > > > > > > > > >>> > > > > > saying
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > don't
> > > > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that we
> > > > already
> > > > > > > > > >>> implemented
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > server
> > > > > > > > > >>> > > > > > > > side
> > > > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > >>> > > > > across
> > > > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at
> 5:05 PM
> > > > > Justine
> > > > > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > <jolshan@confluent.io.invalid
> >
> > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the
> > previous
> > > ID
> > > > > in
> > > > > > > the
> > > > > > > > > >>> Prepare
> > > > > > > > > >>> > > > field
> > > > > > > > > >>> > > > > > > since
> > > > > > > > > >>> > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > > > >>> > > > > > > > >> > > > > > it. It is the same producer
> ID
> > > as
> > > > > the
> > > > > > > main
> > > > > > > > > >>> > producer
> > > > > > > > > >>> > > ID
> > > > > > > > > >>> > > > > > > field.
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> > > reread
> > > > > your
> > > > > > > > > message
> > > > > > > > > >>> > about
> > > > > > > > > >>> > > > > > delayed
> > > > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn
> > marker
> > > > > after
> > > > > > > the
> > > > > > > > > >>> > > transaction
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > > > already
> > > > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > > > transaction
> > > > > > > > early
> > > > > > > > > >>> > without
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > fixes
> > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > >>> > > > > > > > >> > > part
> > > > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology
> > was
> > > > used
> > > > > > in
> > > > > > > a
> > > > > > > > > >>> previous
> > > > > > > > > >>> > > KIP
> > > > > > > > > >>> > > > > and
> > > > > > > > > >>> > > > > > > > never
> > > > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log --
> > > only
> > > > in
> > > > > > > > memory
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> > > > > protocol
> > > > > > > are
> > > > > > > > > you
> > > > > > > > > >>> > > > referring
> > > > > > > > > >>> > > > > > to?
> > > > > > > > > >>> > > > > > > I
> > > > > > > > > >>> > > > > > > > am
> > > > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> > > remove
> > > > > the
> > > > > > > > extra
> > > > > > > > > >>> add
> > > > > > > > > >>> > > > > partitions
> > > > > > > > > >>> > > > > > > > call
> > > > > > > > > >>> > > > > > > > >> > and I
> > > > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I
> > have
> > > > yet
> > > > > to
> > > > > > > > > update
> > > > > > > > > >>> that
> > > > > > > > > >>> > > > > section
> > > > > > > > > >>> > > > > > > as
> > > > > > > > > >>> > > > > > > > I
> > > > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > > > >>> > > > > > > > >> > > > > > the design so please stay
> > tuned.
> > > > Was
> > > > > > > there
> > > > > > > > > >>> > anything
> > > > > > > > > >>> > > > else
> > > > > > > > > >>> > > > > > you
> > > > > > > > > >>> > > > > > > > >> > thought
> > > > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at
> > 3:07 PM
> > > > Jun
> > > > > > Rao
> > > > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create
> any
> > > > issue.
> > > > > > It
> > > > > > > > just
> > > > > > > > > >>> makes
> > > > > > > > > >>> > > it
> > > > > > > > > >>> > > > a
> > > > > > > > > >>> > > > > > bit
> > > > > > > > > >>> > > > > > > > >> hard to
> > > > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged
> produce
> > > id
> > > > > > field
> > > > > > > > > >>> means. We
> > > > > > > > > >>> > > are
> > > > > > > > > >>> > > > > > > > >> essentially
> > > > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> > > (completing
> > > > a
> > > > > > txn
> > > > > > > > and
> > > > > > > > > >>> init a
> > > > > > > > > >>> > > new
> > > > > > > > > >>> > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> Id)
> > > > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this
> may
> > > be
> > > > > fine
> > > > > > > > too.
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> > > comments.
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason
> > that
> > > we
> > > > > > only
> > > > > > > > set
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > previous
> > > > > > > > > >>> > > > > > > > >> produce id
> > > > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > field in the complete
> > marker,
> > > > but
> > > > > > not
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>> > > prepare
> > > > > > > > > >>> > > > > > > marker,
> > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> > > return
> > > > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > > > >>> > > > > > on
> > > > > > > > > >>> > > > > > > > >> > retried
> > > > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your
> second
> > > > point
> > > > > is
> > > > > > > > > >>> mentioned
> > > > > > > > > >>> > in
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > KIP. I
> > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > >>> > > > > > > > >> > > add
> > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message case
> > can
> > > > > also
> > > > > > > > > violate
> > > > > > > > > >>> EOS
> > > > > > > > > >>> > if
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > delayed
> > > > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > > > addPartitionsToTxn
> > > > > > > > > >>> > request
> > > > > > > > > >>> > > > > comes
> > > > > > > > > >>> > > > > > > in.
> > > > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > > > previous
> > > > > > > > > (aborted)
> > > > > > > > > >>> > > > > transaction
> > > > > > > > > >>> > > > > > > > become
> > > > > > > > > >>> > > > > > > > >> > part
> > > > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > The above is the case
> when a
> > > > > delayed
> > > > > > > > > >>> message is
> > > > > > > > > >>> > > > > appended
> > > > > > > > > >>> > > > > > > to
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > data
> > > > > > > > > >>> > > > > > > > >> > > > > > > partition. What I
> mentioned
> > > is a
> > > > > > > > slightly
> > > > > > > > > >>> > > different
> > > > > > > > > >>> > > > > case
> > > > > > > > > >>> > > > > > > > when
> > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > > > > > transaction
> > > > > > > > log
> > > > > > > > > >>> > > partition.
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we
> > > move
> > > > > past
> > > > > > > the
> > > > > > > > > >>> Prepare
> > > > > > > > > >>> > > and
> > > > > > > > > >>> > > > > > > > Complete
> > > > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about
> > > > lastSeen
> > > > > > > > fields
> > > > > > > > > >>> and
> > > > > > > > > >>> > > clear
> > > > > > > > > >>> > > > > > them,
> > > > > > > > > >>> > > > > > > > just
> > > > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.".
> Is
> > > the
> > > > > > > lastSeen
> > > > > > > > > >>> field
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > same
> > > > > > > > > >>> > > > > > as
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > > > > > >>> TransactionLogValue?
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes
> > the
> > > > > > > > > inter-broker
> > > > > > > > > >>> > > > protocol,
> > > > > > > > > >>> > > > > > > should
> > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> > > feature
> > > > > only
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > >>> > KRaft
> > > > > > > > > >>> > > > > mode?
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> > > 11:13 AM
> > > > > > > Justine
> > > > > > > > > >>> Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > <jolshan@confluent.io.invalid
> > > >
> > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting
> to
> > > > > > > convergence
> > > > > > > > > on
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > design.
> > > > > > > > > >>> > > > > > :)
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it
> > seems
> > > a
> > > > > > little
> > > > > > > > > >>> "weird".
> > > > > > > > > >>> > > I'm
> > > > > > > > > >>> > > > > not
> > > > > > > > > >>> > > > > > > sure
> > > > > > > > > >>> > > > > > > > >> what
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra
> record
> > > to
> > > > > the
> > > > > > > log.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> > > > > describe
> > > > > > > > > >>> transactions
> > > > > > > > > >>> > > > won't
> > > > > > > > > >>> > > > > > > work
> > > > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to
> > calculate
> > > > the
> > > > > > > time
> > > > > > > > > >>> since
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like
> > > this,
> > > > > it
> > > > > > is
> > > > > > > > > >>> enough to
> > > > > > > > > >>> > > > > > convince
> > > > > > > > > >>> > > > > > > me
> > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > >>> > > > > > > > >> > > need
> > > > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems
> > like
> > > it
> > > > > > would
> > > > > > > > be
> > > > > > > > > >>> > > replacing
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > record
> > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this
> > > > correct?
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at
> > > > 5:14 PM
> > > > > > Jun
> > > > > > > > Rao
> > > > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> > > explanation.
> > > > I
> > > > > > > > > >>> understand the
> > > > > > > > > >>> > > > > > intention
> > > > > > > > > >>> > > > > > > > >> now.
> > > > > > > > > >>> > > > > > > > >> > In
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> > > non-tagged
> > > > > > field
> > > > > > > to
> > > > > > > > > >>> the old
> > > > > > > > > >>> > > pid
> > > > > > > > > >>> > > > > > (and
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> max
> > > > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that
> > we
> > > > > could
> > > > > > > > > >>> correctly
> > > > > > > > > >>> > > write
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> marker to
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the
> broker
> > > > > > > downgrades.
> > > > > > > > > When
> > > > > > > > > >>> > > writing
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> complete
> > > > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> > > already
> > > > > been
> > > > > > > > > >>> written to
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > data
> > > > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to
> the
> > > new
> > > > > pid
> > > > > > to
> > > > > > > > > avoid
> > > > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to
> work.
> > > > It's
> > > > > > > just a
> > > > > > > > > bit
> > > > > > > > > >>> > > > > > inconsistent
> > > > > > > > > >>> > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker
> to
> > > use
> > > > > > > > different
> > > > > > > > > >>> pids
> > > > > > > > > >>> > in
> > > > > > > > > >>> > > > > this
> > > > > > > > > >>> > > > > > > > >> special
> > > > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> > > complete
> > > > > > > marker,
> > > > > > > > it
> > > > > > > > > >>> seems
> > > > > > > > > >>> > > > that
> > > > > > > > > >>> > > > > we
> > > > > > > > > >>> > > > > > > > will
> > > > > > > > > >>> > > > > > > > >> > never
> > > > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete
> > marker
> > > > with
> > > > > > the
> > > > > > > > old
> > > > > > > > > >>> pid.
> > > > > > > > > >>> > > Not
> > > > > > > > > >>> > > > > sure
> > > > > > > > > >>> > > > > > > if
> > > > > > > > > >>> > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a
> > bit
> > > > > weird.
> > > > > > > > > >>> Instead of
> > > > > > > > > >>> > > > > writing
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid,
> could
> > we
> > > > > write
> > > > > > > two
> > > > > > > > > >>> > records:
> > > > > > > > > >>> > > a
> > > > > > > > > >>> > > > > > > complete
> > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > > > > > >>> TransactionLogValue
> > > > > > > > > >>> > with
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > new
> > > > > > > > > >>> > > > > > > > pid
> > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > >>> > > > > > > > >> > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make
> the
> > > two
> > > > > > > records
> > > > > > > > > in
> > > > > > > > > >>> the
> > > > > > > > > >>> > > same
> > > > > > > > > >>> > > > > > batch
> > > > > > > > > >>> > > > > > > > so
> > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> > > atomically.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 5:40 PM
> > > > > > > > > Justine
> > > > > > > > > >>> > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > <jolshan@confluent.io.invalid
> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare
> marker
> > > is
> > > > > > > written,
> > > > > > > > > >>> but the
> > > > > > > > > >>> > > > > endTxn
> > > > > > > > > >>> > > > > > > > >> response
> > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when
> the
> > > > > server
> > > > > > > > > >>> downgrades
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare
> > marker
> > > is
> > > > > > > > written,
> > > > > > > > > >>> the
> > > > > > > > > >>> > > endTxn
> > > > > > > > > >>> > > > > > > > response
> > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the
> server
> > > > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a
> > > > little
> > > > > > > > > confused.
> > > > > > > > > >>> In
> > > > > > > > > >>> > > both
> > > > > > > > > >>> > > > of
> > > > > > > > > >>> > > > > > > these
> > > > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has
> > the
> > > > old
> > > > > > > > producer
> > > > > > > > > >>> ID.
> > > > > > > > > >>> > We
> > > > > > > > > >>> > > > > don't
> > > > > > > > > >>> > > > > > > > write
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare
> > marker's
> > > > non
> > > > > > > tagged
> > > > > > > > > >>> fields.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server
> > downgrades
> > > > > now,
> > > > > > it
> > > > > > > > > would
> > > > > > > > > >>> > read
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > records
> > > > > > > > > >>> > > > > > > > >> not
> > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the
> > complete
> > > > > marker
> > > > > > > > will
> > > > > > > > > >>> also
> > > > > > > > > >>> > > have
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > old
> > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the
> > new
> > > > > > producer
> > > > > > > > ID,
> > > > > > > > > >>> we
> > > > > > > > > >>> > > would
> > > > > > > > > >>> > > > > not
> > > > > > > > > >>> > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since
> the
> > > > > producer
> > > > > > > id
> > > > > > > > > >>> doesn't
> > > > > > > > > >>> > > > match
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> > > correct
> > > > > on
> > > > > > > the
> > > > > > > > > data
> > > > > > > > > >>> > > > > partition.)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow
> case,
> > > I'd
> > > > > > expect
> > > > > > > > the
> > > > > > > > > >>> > > following
> > > > > > > > > >>> > > > > to
> > > > > > > > > >>> > > > > > > > >> happen on
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry
> > > EndTxn
> > > > > --
> > > > > > it
> > > > > > > > is
> > > > > > > > > >>> the
> > > > > > > > > >>> > same
> > > > > > > > > >>> > > > > > > producer
> > > > > > > > > >>> > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the
> > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't
> > retry
> > > > > > EndTxn
> > > > > > > > and
> > > > > > > > > >>> use
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > >> producer id
> > > > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > > > InvalidPidMappingException
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> > > special
> > > > > > > handling
> > > > > > > > > for
> > > > > > > > > >>> > when
> > > > > > > > > >>> > > a
> > > > > > > > > >>> > > > > > server
> > > > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could
> > get
> > > an
> > > > > API
> > > > > > > > > version
> > > > > > > > > >>> > > request
> > > > > > > > > >>> > > > > > > showing
> > > > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In
> that
> > > > case,
> > > > > > we
> > > > > > > > can
> > > > > > > > > >>> call
> > > > > > > > > >>> > > > > > > > initProducerId
> > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> > > > > overflow
> > > > > > > > case,
> > > > > > > > > >>> this
> > > > > > > > > >>> > > > > correctly
> > > > > > > > > >>> > > > > > > > gives
> > > > > > > > > >>> > > > > > > > >> > us a
> > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> > > corresponding
> > > > > case
> > > > > > > > would
> > > > > > > > > >>> be
> > > > > > > > > >>> > > where
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> *complete
> > > > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the
> endTxn
> > > is
> > > > > not
> > > > > > > > > >>> received by
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > client
> > > > > > > > > >>> > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This
> would
> > > > > result
> > > > > > in
> > > > > > > > the
> > > > > > > > > >>> > > > transaction
> > > > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old
> > one.
> > > > If
> > > > > > the
> > > > > > > > > client
> > > > > > > > > >>> > > > retries,
> > > > > > > > > >>> > > > > it
> > > > > > > > > >>> > > > > > > > will
> > > > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > InvalidPidMappingException.
> > > > > > The
> > > > > > > > > >>> > > InitProducerId
> > > > > > > > > >>> > > > > > > > scenario
> > > > > > > > > >>> > > > > > > > >> > above
> > > > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > > > compatibility
> > > > > > > > story
> > > > > > > > > is
> > > > > > > > > >>> > meant
> > > > > > > > > >>> > > > to
> > > > > > > > > >>> > > > > > > > support
> > > > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > > > > > transactional
> > > > > > > > > >>> > > correctness.
> > > > > > > > > >>> > > > > > > Keeping
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is
> not
> > > the
> > > > > > > > priority.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I
> can
> > > > also
> > > > > > add
> > > > > > > > text
> > > > > > > > > >>> in
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > KIP
> > > > > > > > > >>> > > > > > > about
> > > > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes
> some
> > > edge
> > > > > > > cases.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 4:10 PM
> > > > > > > > Jun
> > > > > > > > > >>> Rao
> > > > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the
> > reply.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we
> > don't
> > > > need
> > > > > > to
> > > > > > > > > >>> optimize
> > > > > > > > > >>> > for
> > > > > > > > > >>> > > > > > fencing
> > > > > > > > > >>> > > > > > > > >> during
> > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding
> > consistency,
> > > > > there
> > > > > > > are
> > > > > > > > > two
> > > > > > > > > >>> > > > possible
> > > > > > > > > >>> > > > > > > cases:
> > > > > > > > > >>> > > > > > > > >> (1)
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but
> the
> > > > endTxn
> > > > > > > > > response
> > > > > > > > > >>> is
> > > > > > > > > >>> > not
> > > > > > > > > >>> > > > > > > received
> > > > > > > > > >>> > > > > > > > by
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades;
> > (2)
> > > > > the
> > > > > > > > > prepare
> > > > > > > > > >>> > marker
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > > > written,
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the
> > > > client
> > > > > > when
> > > > > > > > the
> > > > > > > > > >>> > server
> > > > > > > > > >>> > > > > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> In
> > > > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old
> > > > produce
> > > > > Id
> > > > > > > and
> > > > > > > > > in
> > > > > > > > > >>> (2),
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > client
> > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > >>> > > > > > > > >> > > have
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> > > > > downgrade
> > > > > > > > right
> > > > > > > > > >>> after
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both
> > (1)
> > > > and
> > > > > > (2)
> > > > > > > > > >>> since we
> > > > > > > > > >>> > > can
> > > > > > > > > >>> > > > > only
> > > > > > > > > >>> > > > > > > put
> > > > > > > > > >>> > > > > > > > >> one
> > > > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce
> Id
> > > > field.
> > > > > > > It's
> > > > > > > > > >>> also not
> > > > > > > > > >>> > > > clear
> > > > > > > > > >>> > > > > > > which
> > > > > > > > > >>> > > > > > > > >> case
> > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could
> probably
> > > be
> > > > > > > > consistent
> > > > > > > > > >>> with
> > > > > > > > > >>> > > > either
> > > > > > > > > >>> > > > > > > case.
> > > > > > > > > >>> > > > > > > > >> By
> > > > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the
> > > > prepare
> > > > > > > > marker,
> > > > > > > > > >>> we
> > > > > > > > > >>> > are
> > > > > > > > > >>> > > > > > > consistent
> > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the
> slight
> > > > > benefit
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > >>> > > produce
> > > > > > > > > >>> > > > > > field
> > > > > > > > > >>> > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker
> are
> > > > > > consistent
> > > > > > > > in
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > overflow
> > > > > > > > > >>> > > > > > > case.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 3:11 PM
> > > > > > > > > >>> Justine
> > > > > > > > > >>> > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > <jolshan@confluent.io.invalid
> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> > > > > describe,
> > > > > > we
> > > > > > > > > would
> > > > > > > > > >>> > need
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > > > have
> > > > > > > > > >>> > > > > > > a
> > > > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful
> EndTxn,
> > > > and a
> > > > > > > > > >>> successful
> > > > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> > > request
> > > > > go
> > > > > > > > > through
> > > > > > > > > >>> for
> > > > > > > > > >>> > a
> > > > > > > > > >>> > > > > given
> > > > > > > > > >>> > > > > > > > >> producer.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to
> > figure
> > > > out
> > > > > > if
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > >>> > > possible
> > > > > > > > > >>> > > > > for
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > client
> > > > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous
> request
> > > is
> > > > > > > delayed
> > > > > > > > > >>> > somewhere.
> > > > > > > > > >>> > > > But
> > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > >>> > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the
> > > > client.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the
> > overflow
> > > > > case.
> > > > > > > In
> > > > > > > > > the
> > > > > > > > > >>> > > overflow
> > > > > > > > > >>> > > > > > case,
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are
> > different
> > > on
> > > > > the
> > > > > > > > > marker
> > > > > > > > > >>> and
> > > > > > > > > >>> > on
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to
> use
> > > the
> > > > > max
> > > > > > > > epoch
> > > > > > > > > >>> but
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> > > epoch 0
> > > > > in
> > > > > > > the
> > > > > > > > > >>> > > > transactional
> > > > > > > > > >>> > > > > > > state.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > > > downgrade
> > > > > > > case,
> > > > > > > > we
> > > > > > > > > >>> want
> > > > > > > > > >>> > to
> > > > > > > > > >>> > > > see
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client
> > will
> > > > > have.
> > > > > > > If
> > > > > > > > we
> > > > > > > > > >>> > > complete
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> commit,
> > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is
> reloaded,
> > > we
> > > > > need
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > >>> > > > producer
> > > > > > > > > >>> > > > > ID
> > > > > > > > > >>> > > > > > > in
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid
> > producer
> > > ID
> > > > > > > > mapping.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server
> > downgrade
> > > > > cases
> > > > > > > are
> > > > > > > > > >>> > > considering
> > > > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing
> > from
> > > > > > previous
> > > > > > > > > >>> behavior
> > > > > > > > > >>> > --
> > > > > > > > > >>> > > > and
> > > > > > > > > >>> > > > > > are
> > > > > > > > > >>> > > > > > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> > > safety
> > > > > from
> > > > > > > > > fencing
> > > > > > > > > >>> > > retries
> > > > > > > > > >>> > > > > (as
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> have
> > > > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > > > support).
> > > > > > > > Perhaps
> > > > > > > > > >>> this
> > > > > > > > > >>> > is
> > > > > > > > > >>> > > a
> > > > > > > > > >>> > > > > > trade
> > > > > > > > > >>> > > > > > > > off,
> > > > > > > > > >>> > > > > > > > >> > but
> > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > > > downgrades,
> > > > > > > it
> > > > > > > > > will
> > > > > > > > > >>> > have
> > > > > > > > > >>> > > > > > > restarted
> > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > >>> > > > > > > > >> > it
> > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new
> > producer
> > > ID
> > > > > > too).
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > > 11:42 AM
> > > > > > > > > >>> Jun
> > > > > > > > > >>> > Rao
> > > > > > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the
> > > > reply.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the
> > > marker
> > > > > is
> > > > > > > > > written
> > > > > > > > > >>> by
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > >> client, we
> > > > > > > > > >>> > > > > > > > >> > > can
> > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> > > > > guarantee
> > > > > > > > that
> > > > > > > > > >>> any
> > > > > > > > > >>> > > EndTxn
> > > > > > > > > >>> > > > > > > > requests
> > > > > > > > > >>> > > > > > > > >> > with
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same
> > > > producer
> > > > > > and
> > > > > > > > the
> > > > > > > > > >>> same
> > > > > > > > > >>> > > > > > > transaction.
> > > > > > > > > >>> > > > > > > > >> Then
> > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a
> fenced
> > > > error
> > > > > > but
> > > > > > > > can
> > > > > > > > > >>> handle
> > > > > > > > > >>> > > > > > > gracefully
> > > > > > > > > >>> > > > > > > > as
> > > > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed
> > > > EndTnx
> > > > > > > > request
> > > > > > > > > is
> > > > > > > > > >>> > > > > processed,
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> txn
> > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next
> > txn.
> > > I
> > > > > > guess
> > > > > > > in
> > > > > > > > > >>> this
> > > > > > > > > >>> > case
> > > > > > > > > >>> > > > we
> > > > > > > > > >>> > > > > > > still
> > > > > > > > > >>> > > > > > > > >> > return
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> > > request?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> > > > > question
> > > > > > > was
> > > > > > > > > >>> > > inaccurate.
> > > > > > > > > >>> > > > > What
> > > > > > > > > >>> > > > > > > you
> > > > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > > > > > compatibility I
> > > > > > > > > >>> > mention
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > that
> > > > > > > > > >>> > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > keep
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in
> the
> > > > main
> > > > > > > > > >>> (non-tagged)
> > > > > > > > > >>> > > > fields
> > > > > > > > > >>> > > > > as
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> did
> > > > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side."
> If
> > > we
> > > > > want
> > > > > > > to
> > > > > > > > do
> > > > > > > > > >>> this,
> > > > > > > > > >>> > > it
> > > > > > > > > >>> > > > > > seems
> > > > > > > > > >>> > > > > > > > >> that we
> > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current
> produce
> > Id
> > > > and
> > > > > > max
> > > > > > > > > >>> epoch in
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > existing
> > > > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch
> > > fields
> > > > > for
> > > > > > > > both
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > prepare
> > > > > > > > > >>> > > > > > and
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade
> > can
> > > > > happen
> > > > > > > > after
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > complete
> > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described,
> > the
> > > > > > > > downgraded
> > > > > > > > > >>> > > > coordinator
> > > > > > > > > >>> > > > > > will
> > > > > > > > > >>> > > > > > > > see
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the
> > old
> > > > > one.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan
> 12,
> > > 2024
> > > > > at
> > > > > > > > > 10:44 AM
> > > > > > > > > >>> > > Justine
> > > > > > > > > >>> > > > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > > >
> > > > > > > > > >>> > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update
> > the
> > > > > > > > > description.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe
> your
> > > > > second
> > > > > > > > point
> > > > > > > > > is
> > > > > > > > > >>> > > > mentioned
> > > > > > > > > >>> > > > > > in
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it
> is
> > > > > helpful.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> delayed
> > > > > message
> > > > > > > case
> > > > > > > > > can
> > > > > > > > > >>> > also
> > > > > > > > > >>> > > > > > violate
> > > > > > > > > >>> > > > > > > > EOS
> > > > > > > > > >>> > > > > > > > >> if
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in
> after
> > > the
> > > > > > next
> > > > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > > > >>> > > > > > > > >> request
> > > > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> > > message
> > > > > > from a
> > > > > > > > > >>> previous
> > > > > > > > > >>> > > > > > (aborted)
> > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> > > transaction.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the
> marker
> > is
> > > > > > written
> > > > > > > > by
> > > > > > > > > >>> the
> > > > > > > > > >>> > new
> > > > > > > > > >>> > > > > > client,
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > > > guarantee
> > > > > > > that
> > > > > > > > > any
> > > > > > > > > >>> > EndTxn
> > > > > > > > > >>> > > > > > > requests
> > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> > > producer
> > > > > and
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > >>> > > > > > transaction.
> > > > > > > > > >>> > > > > > > > >> Then we
> > > > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error
> > but
> > > > can
> > > > > > > > handle
> > > > > > > > > >>> > > gracefully
> > > > > > > > > >>> > > > > as
> > > > > > > > > >>> > > > > > > > >> described
> > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't
> think
> > a
> > > > > > boolean
> > > > > > > is
> > > > > > > > > >>> useful
> > > > > > > > > >>> > > > since
> > > > > > > > > >>> > > > > it
> > > > > > > > > >>> > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or
> > > lack
> > > > of
> > > > > > the
> > > > > > > > > >>> tagged
> > > > > > > > > >>> > > field
> > > > > > > > > >>> > > > > > being
> > > > > > > > > >>> > > > > > > > >> > written.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> prepare
> > > > > marker
> > > > > > we
> > > > > > > > > will
> > > > > > > > > >>> have
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > same
> > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In
> the
> > > > > Complete
> > > > > > > > state
> > > > > > > > > >>> we
> > > > > > > > > >>> > may
> > > > > > > > > >>> > > > not.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> > why
> > > > the
> > > > > > > > ongoing
> > > > > > > > > >>> state
> > > > > > > > > >>> > > > > matters
> > > > > > > > > >>> > > > > > > for
> > > > > > > > > >>> > > > > > > > >> this
> > > > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> > > what
> > > > > you
> > > > > > > are
> > > > > > > > > >>> > referring
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > > > about
> > > > > > > > > >>> > > > > > > > >> > writing
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> > > prepare
> > > > > > > marker.
> > > > > > > > > >>> This is
> > > > > > > > > >>> > > not
> > > > > > > > > >>> > > > in
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > overflow
> > > > > case,
> > > > > > we
> > > > > > > > > >>> write the
> > > > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > > > >>> > > > > > > > >> > in
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so
> we
> > > know
> > > > > > what
> > > > > > > we
> > > > > > > > > >>> > assigned
> > > > > > > > > >>> > > > when
> > > > > > > > > >>> > > > > > we
> > > > > > > > > >>> > > > > > > > >> reload
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> > > complete,
> > > > we
> > > > > > > > > >>> transition
> > > > > > > > > >>> > this
> > > > > > > > > >>> > > > ID
> > > > > > > > > >>> > > > > to
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> main
> > > > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> > > previous
> > > > > > > producer
> > > > > > > > > ID
> > > > > > > > > >>> > field
> > > > > > > > > >>> > > > > filled
> > > > > > > > > >>> > > > > > > in.
> > > > > > > > > >>> > > > > > > > >> This
> > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry
> > case
> > > > the
> > > > > > > > > operation
> > > > > > > > > >>> > > > completed
> > > > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer.
> The
> > > > > > downgrade
> > > > > > > > > >>> > > compatibility
> > > > > > > > > >>> > > > I
> > > > > > > > > >>> > > > > > > > mention
> > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID
> > and
> > > > > epoch
> > > > > > in
> > > > > > > > the
> > > > > > > > > >>> main
> > > > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > > > >>> > > > > > > > >> > fields
> > > > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the
> > > server
> > > > > > side.
> > > > > > > > If
> > > > > > > > > >>> the
> > > > > > > > > >>> > > server
> > > > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> addresses
> > > > both
> > > > > > the
> > > > > > > > > >>> prepare
> > > > > > > > > >>> > and
> > > > > > > > > >>> > > > > > complete
> > > > > > > > > >>> > > > > > > > >> state
> > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan
> > 12,
> > > > 2024
> > > > > > at
> > > > > > > > > >>> 10:21 AM
> > > > > > > > > >>> > Jun
> > > > > > > > > >>> > > > Rao
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi,
> Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for
> > the
> > > > > > reply.
> > > > > > > > > Sorry
> > > > > > > > > >>> for
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > delay. I
> > > > > > > > > >>> > > > > > > > >> > have a
> > > > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I
> think
> > > the
> > > > > > > > > motivation
> > > > > > > > > >>> > > section
> > > > > > > > > >>> > > > > > could
> > > > > > > > > >>> > > > > > > be
> > > > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> motivations
> > > > listed
> > > > > > by
> > > > > > > > the
> > > > > > > > > >>> KIP is
> > > > > > > > > >>> > > > "This
> > > > > > > > > >>> > > > > > can
> > > > > > > > > >>> > > > > > > > >> happen
> > > > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed
> > due
> > > > to
> > > > > > > > > networking
> > > > > > > > > >>> > > issues
> > > > > > > > > >>> > > > > or a
> > > > > > > > > >>> > > > > > > > >> network
> > > > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts,
> and
> > > then
> > > > > the
> > > > > > > > > delayed
> > > > > > > > > >>> > > message
> > > > > > > > > >>> > > > > > > finally
> > > > > > > > > >>> > > > > > > > >> > comes
> > > > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very
> > accurate.
> > > > > > Without
> > > > > > > > > >>> KIP-890,
> > > > > > > > > >>> > > > > > currently,
> > > > > > > > > >>> > > > > > > > if
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an
> > > > ongoing
> > > > > > > > > >>> transaction,
> > > > > > > > > >>> > it
> > > > > > > > > >>> > > > > > already
> > > > > > > > > >>> > > > > > > > >> bumps
> > > > > > > > > >>> > > > > > > > >> > up
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which
> > prevents
> > > > the
> > > > > > > > delayed
> > > > > > > > > >>> > produce
> > > > > > > > > >>> > > > > > message
> > > > > > > > > >>> > > > > > > > >> from
> > > > > > > > > >>> > > > > > > > >> > > being
> > > > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition.
> > > What
> > > > > can
> > > > > > > > cause
> > > > > > > > > a
> > > > > > > > > >>> > > hanging
> > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes
> > > > (either
> > > > > > > aborts
> > > > > > > > > or
> > > > > > > > > >>> > > > commits) a
> > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful
> > ack
> > > > on
> > > > > > > > messages
> > > > > > > > > >>> > > published
> > > > > > > > > >>> > > > > in
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> same
> > > > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible
> for
> > > the
> > > > > > > delayed
> > > > > > > > > >>> message
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > be
> > > > > > > > > >>> > > > > > > > >> appended
> > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> > > causing
> > > > a
> > > > > > > > > >>> transaction to
> > > > > > > > > >>> > > > hang.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar
> > > issue
> > > > > (not
> > > > > > > > > >>> mentioned
> > > > > > > > > >>> > in
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in
> > the
> > > > > > > > > coordinator's
> > > > > > > > > >>> log.
> > > > > > > > > >>> > > For
> > > > > > > > > >>> > > > > > > > example,
> > > > > > > > > >>> > > > > > > > >> > it's
> > > > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > EndTxnRequest
> > > to
> > > > > be
> > > > > > > > > delayed
> > > > > > > > > >>> on
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > > > >>> > > > > > > > >> > By
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > EndTxnRequest
> > > is
> > > > > > > > > processed,
> > > > > > > > > >>> it's
> > > > > > > > > >>> > > > > > possible
> > > > > > > > > >>> > > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed
> > and
> > > a
> > > > > new
> > > > > > > txn
> > > > > > > > > has
> > > > > > > > > >>> > > started.
> > > > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > > > >>> > > > > > > > >> > > since
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on
> > > every
> > > > > txn,
> > > > > > > the
> > > > > > > > > >>> delayed
> > > > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > > > >>> > > > > > > > >> > will
> > > > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare
> > marker
> > > > > (and
> > > > > > > > > >>> eventually a
> > > > > > > > > >>> > > > > > complete
> > > > > > > > > >>> > > > > > > > >> marker)
> > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't
> cause
> > > the
> > > > > > > > > transaction
> > > > > > > > > >>> to
> > > > > > > > > >>> > > hang,
> > > > > > > > > >>> > > > > but
> > > > > > > > > >>> > > > > > > it
> > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> proposal
> > > in
> > > > > this
> > > > > > > KIP
> > > > > > > > > >>> will
> > > > > > > > > >>> > > > address
> > > > > > > > > >>> > > > > > this
> > > > > > > > > >>> > > > > > > > >> issue
> > > > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> > > "However, I
> > > > > was
> > > > > > > > > >>> writing it
> > > > > > > > > >>> > so
> > > > > > > > > >>> > > > > that
> > > > > > > > > >>> > > > > > we
> > > > > > > > > >>> > > > > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old
> clients
> > > > where
> > > > > we
> > > > > > > > don't
> > > > > > > > > >>> have
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > ability
> > > > > > > > > >>> > > > > > > > do
> > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients
> that
> > > > can.
> > > > > > (Old
> > > > > > > > > >>> clients
> > > > > > > > > >>> > > don't
> > > > > > > > > >>> > > > > > bump
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for
> sure
> > > the
> > > > > > write
> > > > > > > > > >>> belongs
> > > > > > > > > >>> > to
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > given
> > > > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> > > > > wondering
> > > > > > > why
> > > > > > > > > we
> > > > > > > > > >>> need
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by
> > the
> > > > old
> > > > > > and
> > > > > > > > the
> > > > > > > > > >>> new
> > > > > > > > > >>> > > > client.
> > > > > > > > > >>> > > > > > > Could
> > > > > > > > > >>> > > > > > > > >> you
> > > > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> differently
> > if
> > > > we
> > > > > > know
> > > > > > > > the
> > > > > > > > > >>> > marker
> > > > > > > > > >>> > > is
> > > > > > > > > >>> > > > > > > written
> > > > > > > > > >>> > > > > > > > >> by
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If
> we
> > do
> > > > > need
> > > > > > a
> > > > > > > > way
> > > > > > > > > to
> > > > > > > > > >>> > > > > distinguish
> > > > > > > > > >>> > > > > > > > >> whether
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old
> and
> > > the
> > > > > new
> > > > > > > > > client.
> > > > > > > > > >>> > Would
> > > > > > > > > >>> > > it
> > > > > > > > > >>> > > > > be
> > > > > > > > > >>> > > > > > > > >> simpler
> > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field
> > instead
> > > of
> > > > > > > > > indirectly
> > > > > > > > > >>> > > through
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> previous
> > > > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's
> > not
> > > > > clear
> > > > > > > to
> > > > > > > > me
> > > > > > > > > >>> why
> > > > > > > > > >>> > we
> > > > > > > > > >>> > > > only
> > > > > > > > > >>> > > > > > add
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> complete
> > > > > marker,
> > > > > > > but
> > > > > > > > > >>> not in
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a
> > > marker
> > > > > is
> > > > > > > > > written
> > > > > > > > > >>> by
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > client
> > > > > > > > > >>> > > > > > > > >> or
> > > > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > > > > > consistently
> > > > > > > > > for
> > > > > > > > > >>> all
> > > > > > > > > >>> > > > > markers.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What
> > > about
> > > > > the
> > > > > > > > > >>> > > > > TransactionLogValue
> > > > > > > > > >>> > > > > > > > >> record
> > > > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state?
> > Should
> > > we
> > > > > > also
> > > > > > > > > >>> > distinguish
> > > > > > > > > >>> > > > > > whether
> > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new
> client?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In
> the
> > > > > overflow
> > > > > > > > case,
> > > > > > > > > >>> it's
> > > > > > > > > >>> > > > still
> > > > > > > > > >>> > > > > > not
> > > > > > > > > >>> > > > > > > > >> clear
> > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > >>> > > > > > > > >> > > me
> > > > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> > > produce
> > > > > Id
> > > > > > in
> > > > > > > > the
> > > > > > > > > >>> > prepare
> > > > > > > > > >>> > > > > > marker
> > > > > > > > > >>> > > > > > > > >> while
> > > > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> > > complete
> > > > > > > marker.
> > > > > > > > > You
> > > > > > > > > >>> > > > mentioned
> > > > > > > > > >>> > > > > > that
> > > > > > > > > >>> > > > > > > > >> it's
> > > > > > > > > >>> > > > > > > > >> > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > > > downgrade
> > > > > > > with
> > > > > > > > > >>> either
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> marker
> > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either
> > > case,
> > > > > the
> > > > > > > > > >>> downgraded
> > > > > > > > > >>> > > > > > coordinator
> > > > > > > > > >>> > > > > > > > >> should
> > > > > > > > > >>> > > > > > > > >> > > see
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably
> > the
> > > > > > previous
> > > > > > > > > >>> produce
> > > > > > > > > >>> > > Id),
> > > > > > > > > >>> > > > > > right?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed,
> Dec
> > > 20,
> > > > > 2023
> > > > > > > at
> > > > > > > > > >>> 6:00 PM
> > > > > > > > > >>> > > > > Justine
> > > > > > > > > >>> > > > > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks
> for
> > > > > taking
> > > > > > a
> > > > > > > > look
> > > > > > > > > >>> at
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > KIP
> > > > > > > > > >>> > > > > > > again.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For
> > the
> > > > > epoch
> > > > > > > > > >>> overflow
> > > > > > > > > >>> > > case,
> > > > > > > > > >>> > > > > only
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps
> the
> > > > > behavior
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >>> rest
> > > > > > > > > >>> > > of
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> markers
> > > > > > > > > >>> > > > > > > > >> > > where
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of
> > the
> > > > > > > > transaction
> > > > > > > > > >>> > > records +
> > > > > > > > > >>> > > > > 1.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You
> > are
> > > > > > correct
> > > > > > > > > that
> > > > > > > > > >>> we
> > > > > > > > > >>> > > don't
> > > > > > > > > >>> > > > > > need
> > > > > > > > > >>> > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > write
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the
> > same.
> > > > > > > However,
> > > > > > > > I
> > > > > > > > > >>> was
> > > > > > > > > >>> > > > writing
> > > > > > > > > >>> > > > > it
> > > > > > > > > >>> > > > > > > so
> > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old
> > clients
> > > > > where
> > > > > > we
> > > > > > > > > don't
> > > > > > > > > >>> > have
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > ability
> > > > > > > > > >>> > > > > > > > >> do
> > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients
> > that
> > > > > can.
> > > > > > > (Old
> > > > > > > > > >>> clients
> > > > > > > > > >>> > > > don't
> > > > > > > > > >>> > > > > > > bump
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for
> > sure
> > > > the
> > > > > > > write
> > > > > > > > > >>> belongs
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > given
> > > > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> > > request
> > > > > > from
> > > > > > > a
> > > > > > > > > new
> > > > > > > > > >>> > > client,
> > > > > > > > > >>> > > > we
> > > > > > > > > >>> > > > > > > will
> > > > > > > > > >>> > > > > > > > >> fill
> > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any
> > > > EndTxn
> > > > > > > > requests
> > > > > > > > > >>> with
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > same
> > > > > > > > > >>> > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > are
> > > > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > > > > > transaction.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In
> > > > prepare
> > > > > > > phase,
> > > > > > > > > we
> > > > > > > > > >>> have
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > same
> > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It
> is
> > > the
> > > > > > > > producer
> > > > > > > > > >>> ID and
> > > > > > > > > >>> > > > epoch
> > > > > > > > > >>> > > > > > > that
> > > > > > > > > >>> > > > > > > > >> are
> > > > > > > > > >>> > > > > > > > >> > on
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase,
> we
> > > stay
> > > > > the
> > > > > > > > same
> > > > > > > > > >>> unless
> > > > > > > > > >>> > > it
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the
> > > > producer
> > > > > > ID
> > > > > > > to
> > > > > > > > > >>> the new
> > > > > > > > > >>> > > one
> > > > > > > > > >>> > > > > we
> > > > > > > > > >>> > > > > > > > >> generated
> > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> complete.
> > > This
> > > > > is
> > > > > > > for
> > > > > > > > > >>> > downgrade
> > > > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > > > >>> > > > > > > > >> > > The
> > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> > > guards
> > > > > for
> > > > > > > > > retries
> > > > > > > > > >>> and
> > > > > > > > > >>> > > > > > failovers.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> prepare
> > > > phase
> > > > > > for
> > > > > > > > > epoch
> > > > > > > > > >>> > > > overflow
> > > > > > > > > >>> > > > > > case
> > > > > > > > > >>> > > > > > > > >> only
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This
> > is
> > > > for
> > > > > > the
> > > > > > > > case
> > > > > > > > > >>> where
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > > > > reload
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare
> > > state.
> > > > > > Once
> > > > > > > > the
> > > > > > > > > >>> > > > transaction
> > > > > > > > > >>> > > > > is
> > > > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> > > client
> > > > > > > already
> > > > > > > > is
> > > > > > > > > >>> > using.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> commit
> > > > phase,
> > > > > > we
> > > > > > > > > store
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > previous
> > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think
> it
> > > is
> > > > > > easier
> > > > > > > > to
> > > > > > > > > >>> think
> > > > > > > > > >>> > of
> > > > > > > > > >>> > > > it
> > > > > > > > > >>> > > > > as
> > > > > > > > > >>> > > > > > > > just
> > > > > > > > > >>> > > > > > > > >> how
> > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> epoch
> > > > > before,
> > > > > > > with
> > > > > > > > > >>> some
> > > > > > > > > >>> > > extra
> > > > > > > > > >>> > > > > > > > bookeeping
> > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> > > fields.
> > > > > We
> > > > > > > have
> > > > > > > > > to
> > > > > > > > > >>> do
> > > > > > > > > >>> > it
> > > > > > > > > >>> > > > this
> > > > > > > > > >>> > > > > > way
> > > > > > > > > >>> > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103.
> Next
> > > > > producer
> > > > > > > ID
> > > > > > > > is
> > > > > > > > > >>> for
> > > > > > > > > >>> > > > prepare
> > > > > > > > > >>> > > > > > > > status
> > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > > > complete.
> > > > > > The
> > > > > > > > > >>> reason why
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > > > need
> > > > > > > > > >>> > > > > > > two
> > > > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> backwards
> > > > > > > > compatibility.
> > > > > > > > > >>> We
> > > > > > > > > >>> > need
> > > > > > > > > >>> > > > to
> > > > > > > > > >>> > > > > > keep
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> non-tagged
> > > > field
> > > > > > in
> > > > > > > > case
> > > > > > > > > >>> we
> > > > > > > > > >>> > > > > downgrade.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We
> > set
> > > > the
> > > > > > > fields
> > > > > > > > > as
> > > > > > > > > >>> we
> > > > > > > > > >>> > do
> > > > > > > > > >>> > > in
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this
> > for
> > > > > > > > > compatibility
> > > > > > > > > >>> --
> > > > > > > > > >>> > if
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> non-tagged
> > > > > fields)
> > > > > > > It
> > > > > > > > > >>> will be
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > old
> > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > >>> > > > > > > > >> > ID
> > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope
> this
> > > > helps.
> > > > > > Let
> > > > > > > > me
> > > > > > > > > >>> know
> > > > > > > > > >>> > if
> > > > > > > > > >>> > > > you
> > > > > > > > > >>> > > > > > have
> > > > > > > > > >>> > > > > > > > >> > further
> > > > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed,
> > Dec
> > > > 20,
> > > > > > 2023
> > > > > > > > at
> > > > > > > > > >>> > 3:33 PM
> > > > > > > > > >>> > > > Jun
> > > > > > > > > >>> > > > > > Rao
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> > > Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It
> seems
> > > > that
> > > > > > you
> > > > > > > > have
> > > > > > > > > >>> made
> > > > > > > > > >>> > > some
> > > > > > > > > >>> > > > > > > changes
> > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > particular,
> > > > we
> > > > > > are
> > > > > > > > > >>> changing
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > format
> > > > > > > > > >>> > > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> comments
> > > > > related
> > > > > > > to
> > > > > > > > > >>> that.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100.
> > Just
> > > to
> > > > > be
> > > > > > > > clear.
> > > > > > > > > >>> The
> > > > > > > > > >>> > > > > overflow
> > > > > > > > > >>> > > > > > > case
> > > > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > generated)
> > > > is
> > > > > > when
> > > > > > > > the
> > > > > > > > > >>> > current
> > > > > > > > > >>> > > > > epoch
> > > > > > > > > >>> > > > > > > > >> equals
> > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101.
> For
> > > the
> > > > > > "not
> > > > > > > > > epoch
> > > > > > > > > >>> > > > overflow"
> > > > > > > > > >>> > > > > > > case,
> > > > > > > > > >>> > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > write
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged
> > > field
> > > > > in
> > > > > > > the
> > > > > > > > > >>> complete
> > > > > > > > > >>> > > > > phase.
> > > > > > > > > >>> > > > > > Do
> > > > > > > > > >>> > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > need
> > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> doesn't
> > > > change
> > > > > > in
> > > > > > > > this
> > > > > > > > > >>> case?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102.
> It
> > > > seems
> > > > > > that
> > > > > > > > the
> > > > > > > > > >>> > meaning
> > > > > > > > > >>> > > > for
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > ProducerId/ProducerEpoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > TransactionLogValue
> > > > > > > > > >>> changes
> > > > > > > > > >>> > > > > > depending
> > > > > > > > > >>> > > > > > > on
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > TransactionStatus
> > > > > > > > > is
> > > > > > > > > >>> > > > ongoing,
> > > > > > > > > >>> > > > > > they
> > > > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > current
> > > > > > > > > >>> ProducerEpoch.
> > > > > > > > > >>> > > When
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > > > >>> > > they
> > > > > > > > > >>> > > > > > > > represent
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > > > ProducerEpoch.
> > > > > > > > > >>> When the
> > > > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> further
> > > > depend
> > > > > > on
> > > > > > > > > >>> whether
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > epoch
> > > > > > > > > >>> > > > > > > > >> overflows
> > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > overflow,
> > > > they
> > > > > > > > > represent
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > current
> > > > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > ProducerEpoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > > > > > Otherwise,
> > > > > > > > they
> > > > > > > > > >>> > > represent
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > newly
> > > > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > ProducerEpoch
> > > > > of
> > > > > > > 0.
> > > > > > > > Is
> > > > > > > > > >>> that
> > > > > > > > > >>> > > > right?
> > > > > > > > > >>> > > > > > > This
> > > > > > > > > >>> > > > > > > > >> seems
> > > > > > > > > >>> > > > > > > > >> > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> > provide
> > > > > some
> > > > > > > > > examples
> > > > > > > > > >>> > like
> > > > > > > > > >>> > > > what
> > > > > > > > > >>> > > > > > > Artem
> > > > > > > > > >>> > > > > > > > >> has
> > > > > > > > > >>> > > > > > > > >> > > done
> > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > considered a
> > > > > > > simpler
> > > > > > > > > >>> design
> > > > > > > > > >>> > > > where
> > > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> represent
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> same
> > > > value
> > > > > > > (e.g.
> > > > > > > > > >>> for the
> > > > > > > > > >>> > > > > current
> > > > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > TransactionStatus
> > > > > > > > and
> > > > > > > > > >>> epoch
> > > > > > > > > >>> > > > > > overflow?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103.
> > It's
> > > > not
> > > > > > > clear
> > > > > > > > to
> > > > > > > > > >>> me
> > > > > > > > > >>> > why
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > > > > need
> > > > > > > > > >>> > > > > > > 3
> > > > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > PrevProducerId,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > NextProducerId.
> > > > > > > > Could
> > > > > > > > > we
> > > > > > > > > >>> > just
> > > > > > > > > >>> > > > have
> > > > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104.
> For
> > > > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > > > >>> > > > > if
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set
> > the
> > > > > > > > producerId
> > > > > > > > > >>> and
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> Thanks,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> >
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Calvin,

Thanks for the update. The change looks good to me. Some minor comments.

For retry timeout, our convention is to use retry.backoff.max.ms and
retry.backoff.ms. So, to be consistent, the new configurations should be
add.partitions.to.txn.retry.backoff.max.ms and
add.partitions.to.txn.retry.backoff.ms.

Also, could you specify the defaults for these two values?

Thanks,

Jun




On Mon, Feb 10, 2025 at 2:45 PM Calvin Liu <ca...@confluent.io.invalid>
wrote:

> Hey team,
> In order to address the performance issue in the
> https://issues.apache.org/jira/browse/KAFKA-18654. We are making a change
> to retry the AddPartitionToTxn requests on the server side when hitting the
> CONCURRENT_TRANSACTION error. This error is common after the transaction is
> committed but before the markers are fully propagated to all the involved
> partitions. With the server side retry, the client can avoid the frequent
> and expensive retry for the produce request. In order to better control the
> retry behavior, we proposed to add 2 following configs.
>
>    - *add.partitions.to.txn.max.timeout.ms
>    <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum
> retry
>    timeout when the server attempts to add the partition to the
> transaction.
>    - *add.partitions.to.txn.retry.backoff.ms
>    <http://add.partitions.to.txn.retry.backoff.ms>* defines how frequently
>    the server will retry the AddPartitionToTxn.
>
> Thanks!
>
> On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan
> <jo...@confluent.io.invalid>
> wrote:
>
> > Hey folks, just wanted to share another quick update with KIP-890.
> >
> > Making the change to bump epoch after every transaction means that we no
> > longer need to call InitProducerId during the life of the producer to
> fence
> > requests from previous transactions. With KIP-890 part 2, the new client
> > will no longer call InitProducerId except on startup to fence a previous
> > instance.
> >
> > With this change, some other calls to InitProducerId were inspected
> > including the call after receiving an InvalidPidMappingException. This
> > exception was changed to abortable as part of KIP-360: Improve
> reliability
> > of idempotent/transactional producer
> > <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> > >.
> > However, this change means that we can violate EOS guarantees. As an
> > example:
> >
> > Consider an application that is copying data from one partition to
> another
> >
> >    - Application instance A processes to offset 4
> >    - Application instance B comes up and fences application instance A
> >    - Application instance B processes to offset 5
> >    - Application instances A and B are idle for
> > transaction.id.expiration.ms,
> >    transaction id expires on server
> >    - Application instance A attempts to process offset 5 (since in its
> >    view, that is next) -- if we recover from invalid pid mapping, we can
> >    duplicate this processing
> >
> > Thus, INVALID_PID_MAPPING should be fatal to the producer and as part of
> > KIP-890 the new client in 4.0 will treat the exception as fatal.
> >
> > This is consistent with KIP-1050: Consistent error handling for
> > Transactions
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> > >
> > where
> > errors that are fatal to the producer are in the "application
> recoverable"
> > category. This is a grouping that indicates to the client that the
> producer
> > needs to restart and recovery on the application side is necessary.
> > KIP-1050 is approved so we are consistent with that decision.
> >
> > Let me know if there are any questions. I've updated the KIP to reflect
> > this change.
> >
> > Justine
> >
> > On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Sounds good, Justine. It would be useful to document that in the KIP.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > The client will send the newest EndTxn request version if and only if
> > > both
> > > > the client and the server support kip-890 part 2.
> > > > We set the value in the record based on the EndTxn version.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Hi, Justine,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 120. If the broker sends TV Y for the finalized version in
> > > > > ApiVersionResponse, but the client doesn't support Y, how does the
> > > broker
> > > > > know the TV that the client supports?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > > <jo...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hey Jun,
> > > > > >
> > > > > > No worries. Work on this KIP has been blocked for a bit anyways
> --
> > > > > catching
> > > > > > up and rereading what I wrote :)
> > > > > >
> > > > > > 120. ClientTransactionProtocolVersion is the transaction version
> as
> > > > > defined
> > > > > > by the highest transaction version (feature version value)
> > supported
> > > by
> > > > > the
> > > > > > client and the server. This works by the broker sending an
> > > > > > ApiVersionsRequest to the client with the finalized version.
> > Assuming
> > > > > > kip-890 part 2 is enabled by transaction version Y, if this
> request
> > > > > > contains finalized version Y and the client has the logic to set
> > this
> > > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2 not
> > > > enable)
> > > > > > the client will send Y - 1, even though the client has the
> ability
> > to
> > > > > > support kip-890 part 2.
> > > > > >
> > > > > > 121. You are correct that this is not needed. However, currently
> > that
> > > > > field
> > > > > > is already being set in memory -- just not written to disk. I
> think
> > > it
> > > > is
> > > > > > ok to write it to disk though. Let me know if you think
> otherwise.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <jun@confluent.io.invalid
> >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Justine,
> > > > > > >
> > > > > > > Thanks for the update and sorry for the late reply.
> > > > > > >
> > > > > > > 120. I am wondering what value is used for
> > > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > > EndTxnRequest?
> > > > > > >
> > > > > > > 121. Earlier, you made the change to set lastProducerId in
> > PREPARE
> > > to
> > > > > > > indicate that the marker is written for the new client. With
> the
> > > new
> > > > > > > ClientTransactionProtocolVersion field, it seems this is no
> > longer
> > > > > > > necessary.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi there -- another update!
> > > > > > > >
> > > > > > > > When looking into the implementation for the safe epoch
> bumps I
> > > > > > realized
> > > > > > > > that we are already populating previousProducerID in memory
> as
> > > part
> > > > > of
> > > > > > > > KIP-360.
> > > > > > > > If we are to start using flexible fields, it is better to
> > always
> > > > use
> > > > > > this
> > > > > > > > information and have an explicit (tagged) field to indicate
> > > whether
> > > > > the
> > > > > > > > client supports KIP-890 part 2.
> > > > > > > >
> > > > > > > > I've included the extra field and how it is set in the KIP.
> > I've
> > > > also
> > > > > > > > updated the KIP to explain that we will be setting the tagged
> > > > fields
> > > > > > when
> > > > > > > > they are available for all transitions.
> > > > > > > >
> > > > > > > > Finally, I added clearer text about the transaction protocol
> > > > versions
> > > > > > > > included with this KIP. 1 for flexible transaction state
> > records
> > > > and
> > > > > 2
> > > > > > > for
> > > > > > > > KIP-890 part 2 enablement.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > > jolshan@confluent.io
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey there -- small update to the KIP,
> > > > > > > > >
> > > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > > > > TxnOffsetCommit
> > > > > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > > AbortableTransactionException to match the pattern we had
> for
> > > > other
> > > > > > > > errors.
> > > > > > > > > I also mentioned bumping all 6 transactional APIs so we can
> > > > future
> > > > > > > > > proof/support the error on the client going forward. If a
> > > future
> > > > > > change
> > > > > > > > > wants to have an error scenario that requires us to abort
> the
> > > > > > > > transaction,
> > > > > > > > > we can rely on the 3.8+ clients to support it. We ran into
> > > issues
> > > > > > > finding
> > > > > > > > > good/generic error codes that older clients could support
> > while
> > > > > > working
> > > > > > > > on
> > > > > > > > > this KIP, so this should help in the future.
> > > > > > > > >
> > > > > > > > > The features discussion is still ongoing in KIP-1022. Will
> > > update
> > > > > > again
> > > > > > > > > here when that concludes.
> > > > > > > > >
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > > jolshan@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> I don't think AddPartitions is a good example since we
> > > currenly
> > > > > > don't
> > > > > > > > >> gate the version on TV or MV. (We only set a different
> flag
> > > > > > depending
> > > > > > > on
> > > > > > > > >> the TV)
> > > > > > > > >>
> > > > > > > > >> Even if we did want to gate it on TV, I think the idea is
> to
> > > > move
> > > > > > away
> > > > > > > > >> from MV gating inter broker protocols. Ideally we can get
> > to a
> > > > > state
> > > > > > > > where
> > > > > > > > >> MV is just used for metadata changes.
> > > > > > > > >>
> > > > > > > > >> I think some of this discussion might fit more with the
> > > feature
> > > > > > > version
> > > > > > > > >> KIP, so I can try to open that up soon. Until we settle
> > that,
> > > > some
> > > > > > of
> > > > > > > > the
> > > > > > > > >> work in KIP-890 is blocked.
> > > > > > > > >>
> > > > > > > > >> Justine
> > > > > > > > >>
> > > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > > <jun@confluent.io.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi, Justine,
> > > > > > > > >>>
> > > > > > > > >>> Thanks for the reply.
> > > > > > > > >>>
> > > > > > > > >>> Since AddPartitions is an inter broker request, will its
> > > > version
> > > > > be
> > > > > > > > gated
> > > > > > > > >>> only by TV or other features like MV too? For example, if
> > we
> > > > need
> > > > > > to
> > > > > > > > >>> change
> > > > > > > > >>> the protocol for AddPartitions for reasons other than txn
> > > > > > > verification
> > > > > > > > in
> > > > > > > > >>> the future, will the new version be gated by a new MV? If
> > so,
> > > > > does
> > > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > > >>>
> > > > > > > > >>> Jun
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > > involved,
> > > > > only
> > > > > > > the
> > > > > > > > >>> > transactional records that should only be gated by TV)
> > > > > > > > >>> > Another TV gates the ability to turn on kip-890 part 2.
> > > This
> > > > > > would
> > > > > > > > >>> gate the
> > > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > > transactions),
> > > > > > > and
> > > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the
> version
> > > is
> > > > > > > already
> > > > > > > > >>> used
> > > > > > > > >>> > without TV.
> > > > > > > > >>> >
> > > > > > > > >>> > I think the only concern is the Produce request and we
> > > could
> > > > > > > consider
> > > > > > > > >>> work
> > > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > > >>> >
> > > > > > > > >>> > Justine
> > > > > > > > >>> >
> > > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > > > >>> wrote:
> > > > > > > > >>> >
> > > > > > > > >>> > > Hi, Justine,
> > > > > > > > >>> > >
> > > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> > forward,
> > > > will
> > > > > > > those
> > > > > > > > >>> > > PRC/record protocols only be guarded by TV and not by
> > > other
> > > > > > > > features
> > > > > > > > >>> like
> > > > > > > > >>> > > MV?
> > > > > > > > >>> > >
> > > > > > > > >>> > > Thanks,
> > > > > > > > >>> > >
> > > > > > > > >>> > > Jun
> > > > > > > > >>> > >
> > > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > > >>> > > >
> > > > > > > > >>> > > wrote:
> > > > > > > > >>> > >
> > > > > > > > >>> > > > Hi Jun,
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > Sorry I think I misunderstood your question or
> > answered
> > > > > > > > >>> incorrectly.
> > > > > > > > >>> > The
> > > > > > > > >>> > > TV
> > > > > > > > >>> > > > version should ideally be fully independent from
> MV.
> > > > > > > > >>> > > > At least for the changes I proposed, TV should not
> > > affect
> > > > > MV
> > > > > > > and
> > > > > > > > MV
> > > > > > > > >>> > > should
> > > > > > > > >>> > > > not affect TV/
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > I think if we downgrade TV, only that feature
> should
> > > > > > downgrade.
> > > > > > > > >>> > Likewise
> > > > > > > > >>> > > > the same with MV. The finalizedFeatures should just
> > > > reflect
> > > > > > the
> > > > > > > > >>> feature
> > > > > > > > >>> > > > downgrade we made.
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > I also plan to write a new KIP for managing the
> disk
> > > > format
> > > > > > and
> > > > > > > > >>> upgrade
> > > > > > > > >>> > > > tool as we will need new flags to support these
> > > features.
> > > > > > That
> > > > > > > > >>> should
> > > > > > > > >>> > > help
> > > > > > > > >>> > > > clarify some things.
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > Justine
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >
> > > > > > > > >>> > > wrote:
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > > Hi, Justine,
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> > downgrade
> > > > > > another
> > > > > > > > >>> feature
> > > > > > > > >>> > > > (say
> > > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would we
> > > > return
> > > > > > for
> > > > > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse
> in
> > > that
> > > > > > case?
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > Thanks,
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > Jun
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > wrote:
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > > Hey Jun,
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > > (transaction
> > > > > > > > version)
> > > > > > > > >>> we
> > > > > > > > >>> > > will
> > > > > > > > >>> > > > > stop
> > > > > > > > >>> > > > > > using the add partitions to txn optimization
> and
> > > stop
> > > > > > > writing
> > > > > > > > >>> the
> > > > > > > > >>> > > > > flexible
> > > > > > > > >>> > > > > > feature version of the log.
> > > > > > > > >>> > > > > > In the compatibility section I included some
> > > > > explanations
> > > > > > > on
> > > > > > > > >>> how
> > > > > > > > >>> > this
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > done.
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > Thanks,
> > > > > > > > >>> > > > > > Justine
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > > >>> > > > > wrote:
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > If we ever downgrade the transaction feature,
> > any
> > > > > > feature
> > > > > > > > >>> > depending
> > > > > > > > >>> > > > on
> > > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > > >>> > > > > > >
> (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > > changes
> > > > > > > > made
> > > > > > > > >>> in
> > > > > > > > >>> > > > KIP-890
> > > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > Jun
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine
> Olshan
> > > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > wrote:
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > > questions
> > > > > > about
> > > > > > > > >>> MV/IBP.
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > Looking at the options, I think it makes
> the
> > > most
> > > > > > sense
> > > > > > > > to
> > > > > > > > >>> > > create a
> > > > > > > > >>> > > > > > > > separate feature for transactions and use
> > that
> > > to
> > > > > > > version
> > > > > > > > >>> gate
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > features
> > > > > > > > >>> > > > > > > > we need to version gate (flexible
> > transactional
> > > > > state
> > > > > > > > >>> records
> > > > > > > > >>> > and
> > > > > > > > >>> > > > > using
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > new protocol)
> > > > > > > > >>> > > > > > > > I've updated the KIP to include this
> change.
> > > > > > Hopefully
> > > > > > > > >>> that's
> > > > > > > > >>> > > > > > everything
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > Justine
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> > Olshan
> > > <
> > > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > I will update the KIP with the prev field
> > for
> > > > > > prepare
> > > > > > > > as
> > > > > > > > >>> > well.
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty or z
> > if
> > > y
> > > > > will
> > > > > > > > >>> overflow
> > > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we
> overflowed
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > > >>> > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > > >>> > > > > > > > >> (1) My point was that the coordinator
> > could
> > > > fail
> > > > > > > right
> > > > > > > > >>> after
> > > > > > > > >>> > > > > writing
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> > coordinator
> > > > > > > generates
> > > > > > > > >>> the
> > > > > > > > >>> > > > > complete
> > > > > > > > >>> > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> after the failover, it needs some field
> > from
> > > > the
> > > > > > > > prepare
> > > > > > > > >>> > > marker
> > > > > > > > >>> > > > to
> > > > > > > > >>> > > > > > > > >> determine whether it's written by the
> new
> > > > > client.
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> (2) The changing of the behavior sounds
> > good
> > > > to
> > > > > > me.
> > > > > > > We
> > > > > > > > >>> only
> > > > > > > > >>> > > want
> > > > > > > > >>> > > > > to
> > > > > > > > >>> > > > > > > > return
> > > > > > > > >>> > > > > > > > >> success if the prepare state is written
> by
> > > the
> > > > > new
> > > > > > > > >>> client.
> > > > > > > > >>> > So,
> > > > > > > > >>> > > > in
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we also
> > > need
> > > > > sth
> > > > > > in
> > > > > > > > the
> > > > > > > > >>> > > prepare
> > > > > > > > >>> > > > > > > marker
> > > > > > > > >>> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> tell us whether it's written by the new
> > > > client.
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That
> > sounds
> > > > > good
> > > > > > to
> > > > > > > > me.
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> Jun
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine
> > > > Olshan
> > > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually have
> > two
> > > > > > > questions.
> > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> need
> > to
> > > > > write
> > > > > > > the
> > > > > > > > >>> > > previous
> > > > > > > > >>> > > > > > > produce
> > > > > > > > >>> > > > > > > > Id
> > > > > > > > >>> > > > > > > > >> > tagged field in the end maker so that
> we
> > > > know
> > > > > if
> > > > > > > the
> > > > > > > > >>> > marker
> > > > > > > > >>> > > is
> > > > > > > > >>> > > > > > from
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> new
> > > > > > > > >>> > > > > > > > >> > client. Since the end maker is derived
> > > from
> > > > > the
> > > > > > > > >>> prepare
> > > > > > > > >>> > > > marker,
> > > > > > > > >>> > > > > > > should
> > > > > > > > >>> > > > > > > > >> we
> > > > > > > > >>> > > > > > > > >> > write the previous produce Id in the
> > > prepare
> > > > > > > marker
> > > > > > > > >>> field
> > > > > > > > >>> > > too?
> > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > >>> > > > > > > > >> > we will lose this information when
> > > deriving
> > > > > the
> > > > > > > end
> > > > > > > > >>> > marker.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in the
> > > normal
> > > > > > > producer
> > > > > > > > >>> ID
> > > > > > > > >>> > > field.
> > > > > > > > >>> > > > > So
> > > > > > > > >>> > > > > > > yes,
> > > > > > > > >>> > > > > > > > >> we
> > > > > > > > >>> > > > > > > > >> > need it in prepare and that was always
> > the
> > > > > plan.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> > > > enumerate
> > > > > > the
> > > > > > > > >>> fields
> > > > > > > > >>> > and
> > > > > > > > >>> > > > add
> > > > > > > > >>> > > > > > > them
> > > > > > > > >>> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch y.
> > > When
> > > > we
> > > > > > > > >>> overflow
> > > > > > > > >>> > > epoch
> > > > > > > > >>> > > > y
> > > > > > > > >>> > > > > we
> > > > > > > > >>> > > > > > > get
> > > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> field):
> > > > empty
> > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> or
> > z
> > > > if y
> > > > > > > will
> > > > > > > > >>> > overflow
> > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> field):
> > x
> > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> > overflowed
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry
> > and
> > > > see
> > > > > > > epoch
> > > > > > > > -
> > > > > > > > >>> 1 +
> > > > > > > > >>> > ID
> > > > > > > > >>> > > > in
> > > > > > > > >>> > > > > > last
> > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > >>> > > > > > > > >> > fields and are issuing the same
> command
> > > (ie
> > > > > > commit
> > > > > > > > not
> > > > > > > > >>> > > abort),
> > > > > > > > >>> > > > > we
> > > > > > > > >>> > > > > > > > return
> > > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890
> seems
> > to
> > > > > > return
> > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> > making
> > > > this
> > > > > > > > change?
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer if
> > the
> > > > > epoch
> > > > > > is
> > > > > > > > >>> bumped
> > > > > > > > >>> > > and
> > > > > > > > >>> > > > we
> > > > > > > > >>> > > > > > > get a
> > > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are
> intentionally
> > > > > adding
> > > > > > > this
> > > > > > > > >>> to
> > > > > > > > >>> > > > prevent
> > > > > > > > >>> > > > > > > > fencing.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > 112. We already merged the code that
> > adds
> > > > the
> > > > > > > > >>> VerifyOnly
> > > > > > > > >>> > > field
> > > > > > > > >>> > > > > in
> > > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an
> > > inter
> > > > > > > broker
> > > > > > > > >>> > request.
> > > > > > > > >>> > > > It
> > > > > > > > >>> > > > > > > seems
> > > > > > > > >>> > > > > > > > >> that
> > > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do
> > you
> > > > > know
> > > > > > > why?
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> > interbroker
> > > > > > requests
> > > > > > > > as
> > > > > > > > >>> > > > > ApiVersions
> > > > > > > > >>> > > > > > > > should
> > > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> > errors
> > > > > > > correctly
> > > > > > > > >>> if we
> > > > > > > > >>> > > > > receive
> > > > > > > > >>> > > > > > > them
> > > > > > > > >>> > > > > > > > >> in
> > > > > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun
> Rao
> > > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually
> have
> > > two
> > > > > > > > questions.
> > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> need
> > to
> > > > > write
> > > > > > > the
> > > > > > > > >>> > > previous
> > > > > > > > >>> > > > > > > produce
> > > > > > > > >>> > > > > > > > Id
> > > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so
> that
> > we
> > > > > know
> > > > > > if
> > > > > > > > the
> > > > > > > > >>> > > marker
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > > from
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > new
> > > > > > > > >>> > > > > > > > >> > > client. Since the end maker is
> derived
> > > > from
> > > > > > the
> > > > > > > > >>> prepare
> > > > > > > > >>> > > > > marker,
> > > > > > > > >>> > > > > > > > >> should we
> > > > > > > > >>> > > > > > > > >> > > write the previous produce Id in the
> > > > prepare
> > > > > > > > marker
> > > > > > > > >>> > field
> > > > > > > > >>> > > > too?
> > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > >>> > > > > > > > >> > > we will lose this information when
> > > > deriving
> > > > > > the
> > > > > > > > end
> > > > > > > > >>> > > marker.
> > > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we
> retry
> > > and
> > > > > see
> > > > > > > > epoch
> > > > > > > > >>> - 1
> > > > > > > > >>> > +
> > > > > > > > >>> > > ID
> > > > > > > > >>> > > > > in
> > > > > > > > >>> > > > > > > last
> > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> > command
> > > > (ie
> > > > > > > commit
> > > > > > > > >>> not
> > > > > > > > >>> > > > abort),
> > > > > > > > >>> > > > > > we
> > > > > > > > >>> > > > > > > > >> return
> > > > > > > > >>> > > > > > > > >> > > success. The logic before KIP-890
> > seems
> > > to
> > > > > > > return
> > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > >>> > > > > > > > >> > > in this case. Are we intentionally
> > > making
> > > > > this
> > > > > > > > >>> change?
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > 112. We already merged the code that
> > > adds
> > > > > the
> > > > > > > > >>> VerifyOnly
> > > > > > > > >>> > > > field
> > > > > > > > >>> > > > > > in
> > > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is
> an
> > > > inter
> > > > > > > > broker
> > > > > > > > >>> > > request.
> > > > > > > > >>> > > > > It
> > > > > > > > >>> > > > > > > > seems
> > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for that.
> Do
> > > you
> > > > > > know
> > > > > > > > why?
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> > Justine
> > > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to
> > > > "current
> > > > > > > > >>> producer id
> > > > > > > > >>> > > and
> > > > > > > > >>> > > > > > > epoch"
> > > > > > > > >>> > > > > > > > if
> > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am
> still
> > > > > > > finalizing
> > > > > > > > >>> the
> > > > > > > > >>> > > > design.
> > > > > > > > >>> > > > > I
> > > > > > > > >>> > > > > > > will
> > > > > > > > >>> > > > > > > > >> > update
> > > > > > > > >>> > > > > > > > >> > > > the KIP once I get the information
> > > > > > finalized.
> > > > > > > > >>> Sorry
> > > > > > > > >>> > for
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > delays.
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM
> Jun
> > > Rao
> > > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case,
> > the
> > > > > > previous
> > > > > > > > ID
> > > > > > > > >>> is
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > same
> > > > > > > > >>> > > > > > as
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker too,
> > but
> > > we
> > > > > set
> > > > > > > the
> > > > > > > > >>> > > previous
> > > > > > > > >>> > > > ID
> > > > > > > > >>> > > > > > in
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned
> that
> > > > this
> > > > > is
> > > > > > > to
> > > > > > > > >>> know
> > > > > > > > >>> > > that
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > >>> > > > > > > > >> > > > > written by the new client so
> that
> > we
> > > > > could
> > > > > > > > >>> return
> > > > > > > > >>> > > > success
> > > > > > > > >>> > > > > on
> > > > > > > > >>> > > > > > > > >> retried
> > > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was trying
> > to
> > > > > > > understand
> > > > > > > > >>> why
> > > > > > > > >>> > > this
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > not
> > > > > > > > >>> > > > > > > > >> needed
> > > > > > > > >>> > > > > > > > >> > > for
> > > > > > > > >>> > > > > > > > >> > > > > the prepare marker since retry
> can
> > > > > happen
> > > > > > in
> > > > > > > > the
> > > > > > > > >>> > > prepare
> > > > > > > > >>> > > > > > state
> > > > > > > > >>> > > > > > > > >> too.
> > > > > > > > >>> > > > > > > > >> > Is
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare
> state,
> > we
> > > > > > return
> > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> > > > requests?
> > > > > > If
> > > > > > > > so,
> > > > > > > > >>> > should
> > > > > > > > >>> > > > we
> > > > > > > > >>> > > > > > > change
> > > > > > > > >>> > > > > > > > >> "If
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in
> > last
> > > > > seen
> > > > > > > > >>> fields and
> > > > > > > > >>> > > are
> > > > > > > > >>> > > > > > > issuing
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > same
> > > > > > > > >>> > > > > > > > >> > > > > command (ie commit not abort) we
> > can
> > > > > > return
> > > > > > > > >>> (with
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > epoch)"
> > > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a
> > > delayed
> > > > > > > endMaker
> > > > > > > > >>> > request
> > > > > > > > >>> > > > > > carries
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This
> > can
> > > > > > > > >>> commit/abort a
> > > > > > > > >>> > > > future
> > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if
> we
> > > have
> > > > > > seen
> > > > > > > > >>> this in
> > > > > > > > >>> > > > > practice
> > > > > > > > >>> > > > > > > > >> though.
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be
> > useful
> > > > to
> > > > > > make
> > > > > > > > it
> > > > > > > > >>> > clear
> > > > > > > > >>> > > > that
> > > > > > > > >>> > > > > > we
> > > > > > > > >>> > > > > > > > can
> > > > > > > > >>> > > > > > > > >> now
> > > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field from
> > the
> > > > log
> > > > > > > > >>> reliably.
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > >>> > > > > since
> > > > > > > > >>> > > > > > > > it's
> > > > > > > > >>> > > > > > > > >> > > called
> > > > > > > > >>> > > > > > > > >> > > > > across brokers and we are
> changing
> > > its
> > > > > > > schema.
> > > > > > > > >>> Are
> > > > > > > > >>> > you
> > > > > > > > >>> > > > > > saying
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > >> > don't
> > > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that we
> > > already
> > > > > > > > >>> implemented
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > server
> > > > > > > > >>> > > > > > > > side
> > > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > >>> > > > > across
> > > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM
> > > > Justine
> > > > > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the
> previous
> > ID
> > > > in
> > > > > > the
> > > > > > > > >>> Prepare
> > > > > > > > >>> > > > field
> > > > > > > > >>> > > > > > > since
> > > > > > > > >>> > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > > >>> > > > > > > > >> > > > > > it. It is the same producer ID
> > as
> > > > the
> > > > > > main
> > > > > > > > >>> > producer
> > > > > > > > >>> > > ID
> > > > > > > > >>> > > > > > > field.
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> > reread
> > > > your
> > > > > > > > message
> > > > > > > > >>> > about
> > > > > > > > >>> > > > > > delayed
> > > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn
> marker
> > > > after
> > > > > > the
> > > > > > > > >>> > > transaction
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > > > already
> > > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > > transaction
> > > > > > > early
> > > > > > > > >>> > without
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > fixes
> > > > > > > > >>> > > > > > > > >> in
> > > > > > > > >>> > > > > > > > >> > > part
> > > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology
> was
> > > used
> > > > > in
> > > > > > a
> > > > > > > > >>> previous
> > > > > > > > >>> > > KIP
> > > > > > > > >>> > > > > and
> > > > > > > > >>> > > > > > > > never
> > > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log --
> > only
> > > in
> > > > > > > memory
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> > > > protocol
> > > > > > are
> > > > > > > > you
> > > > > > > > >>> > > > referring
> > > > > > > > >>> > > > > > to?
> > > > > > > > >>> > > > > > > I
> > > > > > > > >>> > > > > > > > am
> > > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> > remove
> > > > the
> > > > > > > extra
> > > > > > > > >>> add
> > > > > > > > >>> > > > > partitions
> > > > > > > > >>> > > > > > > > call
> > > > > > > > >>> > > > > > > > >> > and I
> > > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I
> have
> > > yet
> > > > to
> > > > > > > > update
> > > > > > > > >>> that
> > > > > > > > >>> > > > > section
> > > > > > > > >>> > > > > > > as
> > > > > > > > >>> > > > > > > > I
> > > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > > >>> > > > > > > > >> > > > > > the design so please stay
> tuned.
> > > Was
> > > > > > there
> > > > > > > > >>> > anything
> > > > > > > > >>> > > > else
> > > > > > > > >>> > > > > > you
> > > > > > > > >>> > > > > > > > >> > thought
> > > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at
> 3:07 PM
> > > Jun
> > > > > Rao
> > > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create any
> > > issue.
> > > > > It
> > > > > > > just
> > > > > > > > >>> makes
> > > > > > > > >>> > > it
> > > > > > > > >>> > > > a
> > > > > > > > >>> > > > > > bit
> > > > > > > > >>> > > > > > > > >> hard to
> > > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged produce
> > id
> > > > > field
> > > > > > > > >>> means. We
> > > > > > > > >>> > > are
> > > > > > > > >>> > > > > > > > >> essentially
> > > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> > (completing
> > > a
> > > > > txn
> > > > > > > and
> > > > > > > > >>> init a
> > > > > > > > >>> > > new
> > > > > > > > >>> > > > > > > produce
> > > > > > > > >>> > > > > > > > >> Id)
> > > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this may
> > be
> > > > fine
> > > > > > > too.
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> > comments.
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason
> that
> > we
> > > > > only
> > > > > > > set
> > > > > > > > >>> the
> > > > > > > > >>> > > > previous
> > > > > > > > >>> > > > > > > > >> produce id
> > > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > field in the complete
> marker,
> > > but
> > > > > not
> > > > > > in
> > > > > > > > the
> > > > > > > > >>> > > prepare
> > > > > > > > >>> > > > > > > marker,
> > > > > > > > >>> > > > > > > > >> is
> > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> > return
> > > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > > >>> > > > > > on
> > > > > > > > >>> > > > > > > > >> > retried
> > > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your second
> > > point
> > > > is
> > > > > > > > >>> mentioned
> > > > > > > > >>> > in
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > KIP. I
> > > > > > > > >>> > > > > > > > >> can
> > > > > > > > >>> > > > > > > > >> > > add
> > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message case
> can
> > > > also
> > > > > > > > violate
> > > > > > > > >>> EOS
> > > > > > > > >>> > if
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > delayed
> > > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > > addPartitionsToTxn
> > > > > > > > >>> > request
> > > > > > > > >>> > > > > comes
> > > > > > > > >>> > > > > > > in.
> > > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > > previous
> > > > > > > > (aborted)
> > > > > > > > >>> > > > > transaction
> > > > > > > > >>> > > > > > > > become
> > > > > > > > >>> > > > > > > > >> > part
> > > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > The above is the case when a
> > > > delayed
> > > > > > > > >>> message is
> > > > > > > > >>> > > > > appended
> > > > > > > > >>> > > > > > > to
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > data
> > > > > > > > >>> > > > > > > > >> > > > > > > partition. What I mentioned
> > is a
> > > > > > > slightly
> > > > > > > > >>> > > different
> > > > > > > > >>> > > > > case
> > > > > > > > >>> > > > > > > > when
> > > > > > > > >>> > > > > > > > >> a
> > > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > > > > transaction
> > > > > > > log
> > > > > > > > >>> > > partition.
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we
> > move
> > > > past
> > > > > > the
> > > > > > > > >>> Prepare
> > > > > > > > >>> > > and
> > > > > > > > >>> > > > > > > > Complete
> > > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about
> > > lastSeen
> > > > > > > fields
> > > > > > > > >>> and
> > > > > > > > >>> > > clear
> > > > > > > > >>> > > > > > them,
> > > > > > > > >>> > > > > > > > just
> > > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is
> > the
> > > > > > lastSeen
> > > > > > > > >>> field
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > same
> > > > > > > > >>> > > > > > as
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > > > > >>> TransactionLogValue?
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes
> the
> > > > > > > > inter-broker
> > > > > > > > >>> > > > protocol,
> > > > > > > > >>> > > > > > > should
> > > > > > > > >>> > > > > > > > >> we
> > > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> > feature
> > > > only
> > > > > > for
> > > > > > > > the
> > > > > > > > >>> > KRaft
> > > > > > > > >>> > > > > mode?
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> > 11:13 AM
> > > > > > Justine
> > > > > > > > >>> Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> > > > > > convergence
> > > > > > > > on
> > > > > > > > >>> the
> > > > > > > > >>> > > > > design.
> > > > > > > > >>> > > > > > :)
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it
> seems
> > a
> > > > > little
> > > > > > > > >>> "weird".
> > > > > > > > >>> > > I'm
> > > > > > > > >>> > > > > not
> > > > > > > > >>> > > > > > > sure
> > > > > > > > >>> > > > > > > > >> what
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra record
> > to
> > > > the
> > > > > > log.
> > > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> > > > describe
> > > > > > > > >>> transactions
> > > > > > > > >>> > > > won't
> > > > > > > > >>> > > > > > > work
> > > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to
> calculate
> > > the
> > > > > > time
> > > > > > > > >>> since
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like
> > this,
> > > > it
> > > > > is
> > > > > > > > >>> enough to
> > > > > > > > >>> > > > > > convince
> > > > > > > > >>> > > > > > > me
> > > > > > > > >>> > > > > > > > >> we
> > > > > > > > >>> > > > > > > > >> > > need
> > > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems
> like
> > it
> > > > > would
> > > > > > > be
> > > > > > > > >>> > > replacing
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > record
> > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this
> > > correct?
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at
> > > 5:14 PM
> > > > > Jun
> > > > > > > Rao
> > > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> > explanation.
> > > I
> > > > > > > > >>> understand the
> > > > > > > > >>> > > > > > intention
> > > > > > > > >>> > > > > > > > >> now.
> > > > > > > > >>> > > > > > > > >> > In
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> > non-tagged
> > > > > field
> > > > > > to
> > > > > > > > >>> the old
> > > > > > > > >>> > > pid
> > > > > > > > >>> > > > > > (and
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> max
> > > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that
> we
> > > > could
> > > > > > > > >>> correctly
> > > > > > > > >>> > > write
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> marker to
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> > > > > > downgrades.
> > > > > > > > When
> > > > > > > > >>> > > writing
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> complete
> > > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> > already
> > > > been
> > > > > > > > >>> written to
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > data
> > > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the
> > new
> > > > pid
> > > > > to
> > > > > > > > avoid
> > > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to work.
> > > It's
> > > > > > just a
> > > > > > > > bit
> > > > > > > > >>> > > > > > inconsistent
> > > > > > > > >>> > > > > > > > for
> > > > > > > > >>> > > > > > > > >> a
> > > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker to
> > use
> > > > > > > different
> > > > > > > > >>> pids
> > > > > > > > >>> > in
> > > > > > > > >>> > > > > this
> > > > > > > > >>> > > > > > > > >> special
> > > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> > complete
> > > > > > marker,
> > > > > > > it
> > > > > > > > >>> seems
> > > > > > > > >>> > > > that
> > > > > > > > >>> > > > > we
> > > > > > > > >>> > > > > > > > will
> > > > > > > > >>> > > > > > > > >> > never
> > > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete
> marker
> > > with
> > > > > the
> > > > > > > old
> > > > > > > > >>> pid.
> > > > > > > > >>> > > Not
> > > > > > > > >>> > > > > sure
> > > > > > > > >>> > > > > > > if
> > > > > > > > >>> > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a
> bit
> > > > weird.
> > > > > > > > >>> Instead of
> > > > > > > > >>> > > > > writing
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid, could
> we
> > > > write
> > > > > > two
> > > > > > > > >>> > records:
> > > > > > > > >>> > > a
> > > > > > > > >>> > > > > > > complete
> > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > > > > >>> TransactionLogValue
> > > > > > > > >>> > with
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > new
> > > > > > > > >>> > > > > > > > pid
> > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > >>> > > > > > > > >> > > an
> > > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make the
> > two
> > > > > > records
> > > > > > > > in
> > > > > > > > >>> the
> > > > > > > > >>> > > same
> > > > > > > > >>> > > > > > batch
> > > > > > > > >>> > > > > > > > so
> > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> > atomically.
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at
> > > > 5:40 PM
> > > > > > > > Justine
> > > > > > > > >>> > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker
> > is
> > > > > > written,
> > > > > > > > >>> but the
> > > > > > > > >>> > > > > endTxn
> > > > > > > > >>> > > > > > > > >> response
> > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when the
> > > > server
> > > > > > > > >>> downgrades
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare
> marker
> > is
> > > > > > > written,
> > > > > > > > >>> the
> > > > > > > > >>> > > endTxn
> > > > > > > > >>> > > > > > > > response
> > > > > > > > >>> > > > > > > > >> is
> > > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the server
> > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a
> > > little
> > > > > > > > confused.
> > > > > > > > >>> In
> > > > > > > > >>> > > both
> > > > > > > > >>> > > > of
> > > > > > > > >>> > > > > > > these
> > > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has
> the
> > > old
> > > > > > > producer
> > > > > > > > >>> ID.
> > > > > > > > >>> > We
> > > > > > > > >>> > > > > don't
> > > > > > > > >>> > > > > > > > write
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare
> marker's
> > > non
> > > > > > tagged
> > > > > > > > >>> fields.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server
> downgrades
> > > > now,
> > > > > it
> > > > > > > > would
> > > > > > > > >>> > read
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > records
> > > > > > > > >>> > > > > > > > >> not
> > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the
> complete
> > > > marker
> > > > > > > will
> > > > > > > > >>> also
> > > > > > > > >>> > > have
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > old
> > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the
> new
> > > > > producer
> > > > > > > ID,
> > > > > > > > >>> we
> > > > > > > > >>> > > would
> > > > > > > > >>> > > > > not
> > > > > > > > >>> > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since the
> > > > producer
> > > > > > id
> > > > > > > > >>> doesn't
> > > > > > > > >>> > > > match
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> > correct
> > > > on
> > > > > > the
> > > > > > > > data
> > > > > > > > >>> > > > > partition.)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow case,
> > I'd
> > > > > expect
> > > > > > > the
> > > > > > > > >>> > > following
> > > > > > > > >>> > > > > to
> > > > > > > > >>> > > > > > > > >> happen on
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry
> > EndTxn
> > > > --
> > > > > it
> > > > > > > is
> > > > > > > > >>> the
> > > > > > > > >>> > same
> > > > > > > > >>> > > > > > > producer
> > > > > > > > >>> > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the
> producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't
> retry
> > > > > EndTxn
> > > > > > > and
> > > > > > > > >>> use
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > >> producer id
> > > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > > InvalidPidMappingException
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> > special
> > > > > > handling
> > > > > > > > for
> > > > > > > > >>> > when
> > > > > > > > >>> > > a
> > > > > > > > >>> > > > > > server
> > > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could
> get
> > an
> > > > API
> > > > > > > > version
> > > > > > > > >>> > > request
> > > > > > > > >>> > > > > > > showing
> > > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In that
> > > case,
> > > > > we
> > > > > > > can
> > > > > > > > >>> call
> > > > > > > > >>> > > > > > > > initProducerId
> > > > > > > > >>> > > > > > > > >> to
> > > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> > > > overflow
> > > > > > > case,
> > > > > > > > >>> this
> > > > > > > > >>> > > > > correctly
> > > > > > > > >>> > > > > > > > gives
> > > > > > > > >>> > > > > > > > >> > us a
> > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> > corresponding
> > > > case
> > > > > > > would
> > > > > > > > >>> be
> > > > > > > > >>> > > where
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> *complete
> > > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn
> > is
> > > > not
> > > > > > > > >>> received by
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > client
> > > > > > > > >>> > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would
> > > > result
> > > > > in
> > > > > > > the
> > > > > > > > >>> > > > transaction
> > > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old
> one.
> > > If
> > > > > the
> > > > > > > > client
> > > > > > > > >>> > > > retries,
> > > > > > > > >>> > > > > it
> > > > > > > > >>> > > > > > > > will
> > > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > InvalidPidMappingException.
> > > > > The
> > > > > > > > >>> > > InitProducerId
> > > > > > > > >>> > > > > > > > scenario
> > > > > > > > >>> > > > > > > > >> > above
> > > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > > compatibility
> > > > > > > story
> > > > > > > > is
> > > > > > > > >>> > meant
> > > > > > > > >>> > > > to
> > > > > > > > >>> > > > > > > > support
> > > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > > > > transactional
> > > > > > > > >>> > > correctness.
> > > > > > > > >>> > > > > > > Keeping
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not
> > the
> > > > > > > priority.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can
> > > also
> > > > > add
> > > > > > > text
> > > > > > > > >>> in
> > > > > > > > >>> > the
> > > > > > > > >>> > > > KIP
> > > > > > > > >>> > > > > > > about
> > > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes some
> > edge
> > > > > > cases.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 4:10 PM
> > > > > > > Jun
> > > > > > > > >>> Rao
> > > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the
> reply.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we
> don't
> > > need
> > > > > to
> > > > > > > > >>> optimize
> > > > > > > > >>> > for
> > > > > > > > >>> > > > > > fencing
> > > > > > > > >>> > > > > > > > >> during
> > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding
> consistency,
> > > > there
> > > > > > are
> > > > > > > > two
> > > > > > > > >>> > > > possible
> > > > > > > > >>> > > > > > > cases:
> > > > > > > > >>> > > > > > > > >> (1)
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but the
> > > endTxn
> > > > > > > > response
> > > > > > > > >>> is
> > > > > > > > >>> > not
> > > > > > > > >>> > > > > > > received
> > > > > > > > >>> > > > > > > > by
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades;
> (2)
> > > > the
> > > > > > > > prepare
> > > > > > > > >>> > marker
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > > > written,
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the
> > > client
> > > > > when
> > > > > > > the
> > > > > > > > >>> > server
> > > > > > > > >>> > > > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> In
> > > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old
> > > produce
> > > > Id
> > > > > > and
> > > > > > > > in
> > > > > > > > >>> (2),
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > client
> > > > > > > > >>> > > > > > > > >> will
> > > > > > > > >>> > > > > > > > >> > > have
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> > > > downgrade
> > > > > > > right
> > > > > > > > >>> after
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > prepare
> > > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both
> (1)
> > > and
> > > > > (2)
> > > > > > > > >>> since we
> > > > > > > > >>> > > can
> > > > > > > > >>> > > > > only
> > > > > > > > >>> > > > > > > put
> > > > > > > > >>> > > > > > > > >> one
> > > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id
> > > field.
> > > > > > It's
> > > > > > > > >>> also not
> > > > > > > > >>> > > > clear
> > > > > > > > >>> > > > > > > which
> > > > > > > > >>> > > > > > > > >> case
> > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could probably
> > be
> > > > > > > consistent
> > > > > > > > >>> with
> > > > > > > > >>> > > > either
> > > > > > > > >>> > > > > > > case.
> > > > > > > > >>> > > > > > > > >> By
> > > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the
> > > prepare
> > > > > > > marker,
> > > > > > > > >>> we
> > > > > > > > >>> > are
> > > > > > > > >>> > > > > > > consistent
> > > > > > > > >>> > > > > > > > >> with
> > > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the slight
> > > > benefit
> > > > > > that
> > > > > > > > the
> > > > > > > > >>> > > produce
> > > > > > > > >>> > > > > > field
> > > > > > > > >>> > > > > > > > in
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker are
> > > > > consistent
> > > > > > > in
> > > > > > > > >>> the
> > > > > > > > >>> > > > overflow
> > > > > > > > >>> > > > > > > case.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 3:11 PM
> > > > > > > > >>> Justine
> > > > > > > > >>> > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > <jolshan@confluent.io.invalid
> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> > > > describe,
> > > > > we
> > > > > > > > would
> > > > > > > > >>> > need
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > > > have
> > > > > > > > >>> > > > > > > a
> > > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn,
> > > and a
> > > > > > > > >>> successful
> > > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> > request
> > > > go
> > > > > > > > through
> > > > > > > > >>> for
> > > > > > > > >>> > a
> > > > > > > > >>> > > > > given
> > > > > > > > >>> > > > > > > > >> producer.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to
> figure
> > > out
> > > > > if
> > > > > > it
> > > > > > > > is
> > > > > > > > >>> > > possible
> > > > > > > > >>> > > > > for
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > client
> > > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous request
> > is
> > > > > > delayed
> > > > > > > > >>> > somewhere.
> > > > > > > > >>> > > > But
> > > > > > > > >>> > > > > > > yes,
> > > > > > > > >>> > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the
> > > client.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the
> overflow
> > > > case.
> > > > > > In
> > > > > > > > the
> > > > > > > > >>> > > overflow
> > > > > > > > >>> > > > > > case,
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are
> different
> > on
> > > > the
> > > > > > > > marker
> > > > > > > > >>> and
> > > > > > > > >>> > on
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use
> > the
> > > > max
> > > > > > > epoch
> > > > > > > > >>> but
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> > epoch 0
> > > > in
> > > > > > the
> > > > > > > > >>> > > > transactional
> > > > > > > > >>> > > > > > > state.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > > downgrade
> > > > > > case,
> > > > > > > we
> > > > > > > > >>> want
> > > > > > > > >>> > to
> > > > > > > > >>> > > > see
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client
> will
> > > > have.
> > > > > > If
> > > > > > > we
> > > > > > > > >>> > > complete
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> commit,
> > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded,
> > we
> > > > need
> > > > > > the
> > > > > > > > new
> > > > > > > > >>> > > > producer
> > > > > > > > >>> > > > > ID
> > > > > > > > >>> > > > > > > in
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid
> producer
> > ID
> > > > > > > mapping.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server
> downgrade
> > > > cases
> > > > > > are
> > > > > > > > >>> > > considering
> > > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing
> from
> > > > > previous
> > > > > > > > >>> behavior
> > > > > > > > >>> > --
> > > > > > > > >>> > > > and
> > > > > > > > >>> > > > > > are
> > > > > > > > >>> > > > > > > > not
> > > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> > safety
> > > > from
> > > > > > > > fencing
> > > > > > > > >>> > > retries
> > > > > > > > >>> > > > > (as
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > >> have
> > > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > > support).
> > > > > > > Perhaps
> > > > > > > > >>> this
> > > > > > > > >>> > is
> > > > > > > > >>> > > a
> > > > > > > > >>> > > > > > trade
> > > > > > > > >>> > > > > > > > off,
> > > > > > > > >>> > > > > > > > >> > but
> > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > > downgrades,
> > > > > > it
> > > > > > > > will
> > > > > > > > >>> > have
> > > > > > > > >>> > > > > > > restarted
> > > > > > > > >>> > > > > > > > >> and
> > > > > > > > >>> > > > > > > > >> > it
> > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new
> producer
> > ID
> > > > > too).
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 11:42 AM
> > > > > > > > >>> Jun
> > > > > > > > >>> > Rao
> > > > > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the
> > > reply.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the
> > marker
> > > > is
> > > > > > > > written
> > > > > > > > >>> by
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > >> client, we
> > > > > > > > >>> > > > > > > > >> > > can
> > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> > > > guarantee
> > > > > > > that
> > > > > > > > >>> any
> > > > > > > > >>> > > EndTxn
> > > > > > > > >>> > > > > > > > requests
> > > > > > > > >>> > > > > > > > >> > with
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same
> > > producer
> > > > > and
> > > > > > > the
> > > > > > > > >>> same
> > > > > > > > >>> > > > > > > transaction.
> > > > > > > > >>> > > > > > > > >> Then
> > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced
> > > error
> > > > > but
> > > > > > > can
> > > > > > > > >>> handle
> > > > > > > > >>> > > > > > > gracefully
> > > > > > > > >>> > > > > > > > as
> > > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed
> > > EndTnx
> > > > > > > request
> > > > > > > > is
> > > > > > > > >>> > > > > processed,
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> txn
> > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next
> txn.
> > I
> > > > > guess
> > > > > > in
> > > > > > > > >>> this
> > > > > > > > >>> > case
> > > > > > > > >>> > > > we
> > > > > > > > >>> > > > > > > still
> > > > > > > > >>> > > > > > > > >> > return
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> > request?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> > > > question
> > > > > > was
> > > > > > > > >>> > > inaccurate.
> > > > > > > > >>> > > > > What
> > > > > > > > >>> > > > > > > you
> > > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > > > > compatibility I
> > > > > > > > >>> > mention
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > that
> > > > > > > > >>> > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > keep
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the
> > > main
> > > > > > > > >>> (non-tagged)
> > > > > > > > >>> > > > fields
> > > > > > > > >>> > > > > as
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > >> did
> > > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If
> > we
> > > > want
> > > > > > to
> > > > > > > do
> > > > > > > > >>> this,
> > > > > > > > >>> > > it
> > > > > > > > >>> > > > > > seems
> > > > > > > > >>> > > > > > > > >> that we
> > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current produce
> Id
> > > and
> > > > > max
> > > > > > > > >>> epoch in
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > existing
> > > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch
> > fields
> > > > for
> > > > > > > both
> > > > > > > > >>> the
> > > > > > > > >>> > > > prepare
> > > > > > > > >>> > > > > > and
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade
> can
> > > > happen
> > > > > > > after
> > > > > > > > >>> the
> > > > > > > > >>> > > > > complete
> > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described,
> the
> > > > > > > downgraded
> > > > > > > > >>> > > > coordinator
> > > > > > > > >>> > > > > > will
> > > > > > > > >>> > > > > > > > see
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the
> old
> > > > one.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > > 10:44 AM
> > > > > > > > >>> > > Justine
> > > > > > > > >>> > > > > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > >
> > > > > > > > >>> > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update
> the
> > > > > > > > description.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your
> > > > second
> > > > > > > point
> > > > > > > > is
> > > > > > > > >>> > > > mentioned
> > > > > > > > >>> > > > > > in
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is
> > > > helpful.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed
> > > > message
> > > > > > case
> > > > > > > > can
> > > > > > > > >>> > also
> > > > > > > > >>> > > > > > violate
> > > > > > > > >>> > > > > > > > EOS
> > > > > > > > >>> > > > > > > > >> if
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after
> > the
> > > > > next
> > > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > > >>> > > > > > > > >> request
> > > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> > message
> > > > > from a
> > > > > > > > >>> previous
> > > > > > > > >>> > > > > > (aborted)
> > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> > transaction.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker
> is
> > > > > written
> > > > > > > by
> > > > > > > > >>> the
> > > > > > > > >>> > new
> > > > > > > > >>> > > > > > client,
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > >> can
> > > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > > guarantee
> > > > > > that
> > > > > > > > any
> > > > > > > > >>> > EndTxn
> > > > > > > > >>> > > > > > > requests
> > > > > > > > >>> > > > > > > > >> with
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> > producer
> > > > and
> > > > > > the
> > > > > > > > same
> > > > > > > > >>> > > > > > transaction.
> > > > > > > > >>> > > > > > > > >> Then we
> > > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error
> but
> > > can
> > > > > > > handle
> > > > > > > > >>> > > gracefully
> > > > > > > > >>> > > > > as
> > > > > > > > >>> > > > > > > > >> described
> > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think
> a
> > > > > boolean
> > > > > > is
> > > > > > > > >>> useful
> > > > > > > > >>> > > > since
> > > > > > > > >>> > > > > it
> > > > > > > > >>> > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or
> > lack
> > > of
> > > > > the
> > > > > > > > >>> tagged
> > > > > > > > >>> > > field
> > > > > > > > >>> > > > > > being
> > > > > > > > >>> > > > > > > > >> > written.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare
> > > > marker
> > > > > we
> > > > > > > > will
> > > > > > > > >>> have
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > same
> > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the
> > > > Complete
> > > > > > > state
> > > > > > > > >>> we
> > > > > > > > >>> > may
> > > > > > > > >>> > > > not.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> why
> > > the
> > > > > > > ongoing
> > > > > > > > >>> state
> > > > > > > > >>> > > > > matters
> > > > > > > > >>> > > > > > > for
> > > > > > > > >>> > > > > > > > >> this
> > > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> > what
> > > > you
> > > > > > are
> > > > > > > > >>> > referring
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > > > about
> > > > > > > > >>> > > > > > > > >> > writing
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> > prepare
> > > > > > marker.
> > > > > > > > >>> This is
> > > > > > > > >>> > > not
> > > > > > > > >>> > > > in
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> overflow
> > > > case,
> > > > > we
> > > > > > > > >>> write the
> > > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > > >>> > > > > > > > >> > in
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we
> > know
> > > > > what
> > > > > > we
> > > > > > > > >>> > assigned
> > > > > > > > >>> > > > when
> > > > > > > > >>> > > > > > we
> > > > > > > > >>> > > > > > > > >> reload
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> > complete,
> > > we
> > > > > > > > >>> transition
> > > > > > > > >>> > this
> > > > > > > > >>> > > > ID
> > > > > > > > >>> > > > > to
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> main
> > > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> > previous
> > > > > > producer
> > > > > > > > ID
> > > > > > > > >>> > field
> > > > > > > > >>> > > > > filled
> > > > > > > > >>> > > > > > > in.
> > > > > > > > >>> > > > > > > > >> This
> > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry
> case
> > > the
> > > > > > > > operation
> > > > > > > > >>> > > > completed
> > > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The
> > > > > downgrade
> > > > > > > > >>> > > compatibility
> > > > > > > > >>> > > > I
> > > > > > > > >>> > > > > > > > mention
> > > > > > > > >>> > > > > > > > >> is
> > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID
> and
> > > > epoch
> > > > > in
> > > > > > > the
> > > > > > > > >>> main
> > > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > > >>> > > > > > > > >> > fields
> > > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the
> > server
> > > > > side.
> > > > > > > If
> > > > > > > > >>> the
> > > > > > > > >>> > > server
> > > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses
> > > both
> > > > > the
> > > > > > > > >>> prepare
> > > > > > > > >>> > and
> > > > > > > > >>> > > > > > complete
> > > > > > > > >>> > > > > > > > >> state
> > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan
> 12,
> > > 2024
> > > > > at
> > > > > > > > >>> 10:21 AM
> > > > > > > > >>> > Jun
> > > > > > > > >>> > > > Rao
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for
> the
> > > > > reply.
> > > > > > > > Sorry
> > > > > > > > >>> for
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > delay. I
> > > > > > > > >>> > > > > > > > >> > have a
> > > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think
> > the
> > > > > > > > motivation
> > > > > > > > >>> > > section
> > > > > > > > >>> > > > > > could
> > > > > > > > >>> > > > > > > be
> > > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations
> > > listed
> > > > > by
> > > > > > > the
> > > > > > > > >>> KIP is
> > > > > > > > >>> > > > "This
> > > > > > > > >>> > > > > > can
> > > > > > > > >>> > > > > > > > >> happen
> > > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed
> due
> > > to
> > > > > > > > networking
> > > > > > > > >>> > > issues
> > > > > > > > >>> > > > > or a
> > > > > > > > >>> > > > > > > > >> network
> > > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and
> > then
> > > > the
> > > > > > > > delayed
> > > > > > > > >>> > > message
> > > > > > > > >>> > > > > > > finally
> > > > > > > > >>> > > > > > > > >> > comes
> > > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very
> accurate.
> > > > > Without
> > > > > > > > >>> KIP-890,
> > > > > > > > >>> > > > > > currently,
> > > > > > > > >>> > > > > > > > if
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an
> > > ongoing
> > > > > > > > >>> transaction,
> > > > > > > > >>> > it
> > > > > > > > >>> > > > > > already
> > > > > > > > >>> > > > > > > > >> bumps
> > > > > > > > >>> > > > > > > > >> > up
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which
> prevents
> > > the
> > > > > > > delayed
> > > > > > > > >>> > produce
> > > > > > > > >>> > > > > > message
> > > > > > > > >>> > > > > > > > >> from
> > > > > > > > >>> > > > > > > > >> > > being
> > > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition.
> > What
> > > > can
> > > > > > > cause
> > > > > > > > a
> > > > > > > > >>> > > hanging
> > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes
> > > (either
> > > > > > aborts
> > > > > > > > or
> > > > > > > > >>> > > > commits) a
> > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful
> ack
> > > on
> > > > > > > messages
> > > > > > > > >>> > > published
> > > > > > > > >>> > > > > in
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> same
> > > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for
> > the
> > > > > > delayed
> > > > > > > > >>> message
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > be
> > > > > > > > >>> > > > > > > > >> appended
> > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> > causing
> > > a
> > > > > > > > >>> transaction to
> > > > > > > > >>> > > > hang.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar
> > issue
> > > > (not
> > > > > > > > >>> mentioned
> > > > > > > > >>> > in
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in
> the
> > > > > > > > coordinator's
> > > > > > > > >>> log.
> > > > > > > > >>> > > For
> > > > > > > > >>> > > > > > > > example,
> > > > > > > > >>> > > > > > > > >> > it's
> > > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> EndTxnRequest
> > to
> > > > be
> > > > > > > > delayed
> > > > > > > > >>> on
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > > >>> > > > > > > > >> > By
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> EndTxnRequest
> > is
> > > > > > > > processed,
> > > > > > > > >>> it's
> > > > > > > > >>> > > > > > possible
> > > > > > > > >>> > > > > > > > that
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed
> and
> > a
> > > > new
> > > > > > txn
> > > > > > > > has
> > > > > > > > >>> > > started.
> > > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > > >>> > > > > > > > >> > > since
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on
> > every
> > > > txn,
> > > > > > the
> > > > > > > > >>> delayed
> > > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > > >>> > > > > > > > >> > will
> > > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare
> marker
> > > > (and
> > > > > > > > >>> eventually a
> > > > > > > > >>> > > > > > complete
> > > > > > > > >>> > > > > > > > >> marker)
> > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause
> > the
> > > > > > > > transaction
> > > > > > > > >>> to
> > > > > > > > >>> > > hang,
> > > > > > > > >>> > > > > but
> > > > > > > > >>> > > > > > > it
> > > > > > > > >>> > > > > > > > >> will
> > > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal
> > in
> > > > this
> > > > > > KIP
> > > > > > > > >>> will
> > > > > > > > >>> > > > address
> > > > > > > > >>> > > > > > this
> > > > > > > > >>> > > > > > > > >> issue
> > > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> > "However, I
> > > > was
> > > > > > > > >>> writing it
> > > > > > > > >>> > so
> > > > > > > > >>> > > > > that
> > > > > > > > >>> > > > > > we
> > > > > > > > >>> > > > > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients
> > > where
> > > > we
> > > > > > > don't
> > > > > > > > >>> have
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > ability
> > > > > > > > >>> > > > > > > > do
> > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that
> > > can.
> > > > > (Old
> > > > > > > > >>> clients
> > > > > > > > >>> > > don't
> > > > > > > > >>> > > > > > bump
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure
> > the
> > > > > write
> > > > > > > > >>> belongs
> > > > > > > > >>> > to
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > given
> > > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> > > > wondering
> > > > > > why
> > > > > > > > we
> > > > > > > > >>> need
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by
> the
> > > old
> > > > > and
> > > > > > > the
> > > > > > > > >>> new
> > > > > > > > >>> > > > client.
> > > > > > > > >>> > > > > > > Could
> > > > > > > > >>> > > > > > > > >> you
> > > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently
> if
> > > we
> > > > > know
> > > > > > > the
> > > > > > > > >>> > marker
> > > > > > > > >>> > > is
> > > > > > > > >>> > > > > > > written
> > > > > > > > >>> > > > > > > > >> by
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we
> do
> > > > need
> > > > > a
> > > > > > > way
> > > > > > > > to
> > > > > > > > >>> > > > > distinguish
> > > > > > > > >>> > > > > > > > >> whether
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and
> > the
> > > > new
> > > > > > > > client.
> > > > > > > > >>> > Would
> > > > > > > > >>> > > it
> > > > > > > > >>> > > > > be
> > > > > > > > >>> > > > > > > > >> simpler
> > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field
> instead
> > of
> > > > > > > > indirectly
> > > > > > > > >>> > > through
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> previous
> > > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's
> not
> > > > clear
> > > > > > to
> > > > > > > me
> > > > > > > > >>> why
> > > > > > > > >>> > we
> > > > > > > > >>> > > > only
> > > > > > > > >>> > > > > > add
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete
> > > > marker,
> > > > > > but
> > > > > > > > >>> not in
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > prepare
> > > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a
> > marker
> > > > is
> > > > > > > > written
> > > > > > > > >>> by
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > client
> > > > > > > > >>> > > > > > > > >> or
> > > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > > > > consistently
> > > > > > > > for
> > > > > > > > >>> all
> > > > > > > > >>> > > > > markers.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What
> > about
> > > > the
> > > > > > > > >>> > > > > TransactionLogValue
> > > > > > > > >>> > > > > > > > >> record
> > > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state?
> Should
> > we
> > > > > also
> > > > > > > > >>> > distinguish
> > > > > > > > >>> > > > > > whether
> > > > > > > > >>> > > > > > > > it's
> > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the
> > > > overflow
> > > > > > > case,
> > > > > > > > >>> it's
> > > > > > > > >>> > > > still
> > > > > > > > >>> > > > > > not
> > > > > > > > >>> > > > > > > > >> clear
> > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > >>> > > > > > > > >> > > me
> > > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> > produce
> > > > Id
> > > > > in
> > > > > > > the
> > > > > > > > >>> > prepare
> > > > > > > > >>> > > > > > marker
> > > > > > > > >>> > > > > > > > >> while
> > > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> > complete
> > > > > > marker.
> > > > > > > > You
> > > > > > > > >>> > > > mentioned
> > > > > > > > >>> > > > > > that
> > > > > > > > >>> > > > > > > > >> it's
> > > > > > > > >>> > > > > > > > >> > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > > downgrade
> > > > > > with
> > > > > > > > >>> either
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > prepare
> > > > > > > > >>> > > > > > > > >> marker
> > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either
> > case,
> > > > the
> > > > > > > > >>> downgraded
> > > > > > > > >>> > > > > > coordinator
> > > > > > > > >>> > > > > > > > >> should
> > > > > > > > >>> > > > > > > > >> > > see
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably
> the
> > > > > previous
> > > > > > > > >>> produce
> > > > > > > > >>> > > Id),
> > > > > > > > >>> > > > > > right?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec
> > 20,
> > > > 2023
> > > > > > at
> > > > > > > > >>> 6:00 PM
> > > > > > > > >>> > > > > Justine
> > > > > > > > >>> > > > > > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for
> > > > taking
> > > > > a
> > > > > > > look
> > > > > > > > >>> at
> > > > > > > > >>> > the
> > > > > > > > >>> > > > KIP
> > > > > > > > >>> > > > > > > again.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For
> the
> > > > epoch
> > > > > > > > >>> overflow
> > > > > > > > >>> > > case,
> > > > > > > > >>> > > > > only
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the
> > > > behavior
> > > > > > of
> > > > > > > > the
> > > > > > > > >>> rest
> > > > > > > > >>> > > of
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> markers
> > > > > > > > >>> > > > > > > > >> > > where
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of
> the
> > > > > > > transaction
> > > > > > > > >>> > > records +
> > > > > > > > >>> > > > > 1.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You
> are
> > > > > correct
> > > > > > > > that
> > > > > > > > >>> we
> > > > > > > > >>> > > don't
> > > > > > > > >>> > > > > > need
> > > > > > > > >>> > > > > > > to
> > > > > > > > >>> > > > > > > > >> > write
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the
> same.
> > > > > > However,
> > > > > > > I
> > > > > > > > >>> was
> > > > > > > > >>> > > > writing
> > > > > > > > >>> > > > > it
> > > > > > > > >>> > > > > > > so
> > > > > > > > >>> > > > > > > > >> that
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old
> clients
> > > > where
> > > > > we
> > > > > > > > don't
> > > > > > > > >>> > have
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > ability
> > > > > > > > >>> > > > > > > > >> do
> > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients
> that
> > > > can.
> > > > > > (Old
> > > > > > > > >>> clients
> > > > > > > > >>> > > > don't
> > > > > > > > >>> > > > > > > bump
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for
> sure
> > > the
> > > > > > write
> > > > > > > > >>> belongs
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > given
> > > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> > request
> > > > > from
> > > > > > a
> > > > > > > > new
> > > > > > > > >>> > > client,
> > > > > > > > >>> > > > we
> > > > > > > > >>> > > > > > > will
> > > > > > > > >>> > > > > > > > >> fill
> > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any
> > > EndTxn
> > > > > > > requests
> > > > > > > > >>> with
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > same
> > > > > > > > >>> > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > are
> > > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > > > > transaction.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In
> > > prepare
> > > > > > phase,
> > > > > > > > we
> > > > > > > > >>> have
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > same
> > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is
> > the
> > > > > > > producer
> > > > > > > > >>> ID and
> > > > > > > > >>> > > > epoch
> > > > > > > > >>> > > > > > > that
> > > > > > > > >>> > > > > > > > >> are
> > > > > > > > >>> > > > > > > > >> > on
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we
> > stay
> > > > the
> > > > > > > same
> > > > > > > > >>> unless
> > > > > > > > >>> > > it
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the
> > > producer
> > > > > ID
> > > > > > to
> > > > > > > > >>> the new
> > > > > > > > >>> > > one
> > > > > > > > >>> > > > > we
> > > > > > > > >>> > > > > > > > >> generated
> > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete.
> > This
> > > > is
> > > > > > for
> > > > > > > > >>> > downgrade
> > > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > > >>> > > > > > > > >> > > The
> > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> > guards
> > > > for
> > > > > > > > retries
> > > > > > > > >>> and
> > > > > > > > >>> > > > > > failovers.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare
> > > phase
> > > > > for
> > > > > > > > epoch
> > > > > > > > >>> > > > overflow
> > > > > > > > >>> > > > > > case
> > > > > > > > >>> > > > > > > > >> only
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This
> is
> > > for
> > > > > the
> > > > > > > case
> > > > > > > > >>> where
> > > > > > > > >>> > > we
> > > > > > > > >>> > > > > > reload
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare
> > state.
> > > > > Once
> > > > > > > the
> > > > > > > > >>> > > > transaction
> > > > > > > > >>> > > > > is
> > > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> > client
> > > > > > already
> > > > > > > is
> > > > > > > > >>> > using.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit
> > > phase,
> > > > > we
> > > > > > > > store
> > > > > > > > >>> the
> > > > > > > > >>> > > > > previous
> > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it
> > is
> > > > > easier
> > > > > > > to
> > > > > > > > >>> think
> > > > > > > > >>> > of
> > > > > > > > >>> > > > it
> > > > > > > > >>> > > > > as
> > > > > > > > >>> > > > > > > > just
> > > > > > > > >>> > > > > > > > >> how
> > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch
> > > > before,
> > > > > > with
> > > > > > > > >>> some
> > > > > > > > >>> > > extra
> > > > > > > > >>> > > > > > > > bookeeping
> > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> > fields.
> > > > We
> > > > > > have
> > > > > > > > to
> > > > > > > > >>> do
> > > > > > > > >>> > it
> > > > > > > > >>> > > > this
> > > > > > > > >>> > > > > > way
> > > > > > > > >>> > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next
> > > > producer
> > > > > > ID
> > > > > > > is
> > > > > > > > >>> for
> > > > > > > > >>> > > > prepare
> > > > > > > > >>> > > > > > > > status
> > > > > > > > >>> > > > > > > > >> and
> > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > > complete.
> > > > > The
> > > > > > > > >>> reason why
> > > > > > > > >>> > > we
> > > > > > > > >>> > > > > need
> > > > > > > > >>> > > > > > > two
> > > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > > > > > > compatibility.
> > > > > > > > >>> We
> > > > > > > > >>> > need
> > > > > > > > >>> > > > to
> > > > > > > > >>> > > > > > keep
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > > field
> > > > > in
> > > > > > > case
> > > > > > > > >>> we
> > > > > > > > >>> > > > > downgrade.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We
> set
> > > the
> > > > > > fields
> > > > > > > > as
> > > > > > > > >>> we
> > > > > > > > >>> > do
> > > > > > > > >>> > > in
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this
> for
> > > > > > > > compatibility
> > > > > > > > >>> --
> > > > > > > > >>> > if
> > > > > > > > >>> > > we
> > > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > > > fields)
> > > > > > It
> > > > > > > > >>> will be
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > old
> > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > >>> > > > > > > > >> > ID
> > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this
> > > helps.
> > > > > Let
> > > > > > > me
> > > > > > > > >>> know
> > > > > > > > >>> > if
> > > > > > > > >>> > > > you
> > > > > > > > >>> > > > > > have
> > > > > > > > >>> > > > > > > > >> > further
> > > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed,
> Dec
> > > 20,
> > > > > 2023
> > > > > > > at
> > > > > > > > >>> > 3:33 PM
> > > > > > > > >>> > > > Jun
> > > > > > > > >>> > > > > > Rao
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> > Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems
> > > that
> > > > > you
> > > > > > > have
> > > > > > > > >>> made
> > > > > > > > >>> > > some
> > > > > > > > >>> > > > > > > changes
> > > > > > > > >>> > > > > > > > >> to
> > > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > particular,
> > > we
> > > > > are
> > > > > > > > >>> changing
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > format
> > > > > > > > >>> > > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments
> > > > related
> > > > > > to
> > > > > > > > >>> that.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100.
> Just
> > to
> > > > be
> > > > > > > clear.
> > > > > > > > >>> The
> > > > > > > > >>> > > > > overflow
> > > > > > > > >>> > > > > > > case
> > > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> generated)
> > > is
> > > > > when
> > > > > > > the
> > > > > > > > >>> > current
> > > > > > > > >>> > > > > epoch
> > > > > > > > >>> > > > > > > > >> equals
> > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For
> > the
> > > > > "not
> > > > > > > > epoch
> > > > > > > > >>> > > > overflow"
> > > > > > > > >>> > > > > > > case,
> > > > > > > > >>> > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > write
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged
> > field
> > > > in
> > > > > > the
> > > > > > > > >>> complete
> > > > > > > > >>> > > > > phase.
> > > > > > > > >>> > > > > > Do
> > > > > > > > >>> > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > need
> > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't
> > > change
> > > > > in
> > > > > > > this
> > > > > > > > >>> case?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It
> > > seems
> > > > > that
> > > > > > > the
> > > > > > > > >>> > meaning
> > > > > > > > >>> > > > for
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> ProducerId/ProducerEpoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > TransactionLogValue
> > > > > > > > >>> changes
> > > > > > > > >>> > > > > > depending
> > > > > > > > >>> > > > > > > on
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > TransactionStatus
> > > > > > > > is
> > > > > > > > >>> > > > ongoing,
> > > > > > > > >>> > > > > > they
> > > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > current
> > > > > > > > >>> ProducerEpoch.
> > > > > > > > >>> > > When
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > > >>> > > they
> > > > > > > > >>> > > > > > > > represent
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > > ProducerEpoch.
> > > > > > > > >>> When the
> > > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further
> > > depend
> > > > > on
> > > > > > > > >>> whether
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > epoch
> > > > > > > > >>> > > > > > > > >> overflows
> > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> overflow,
> > > they
> > > > > > > > represent
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > current
> > > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > ProducerEpoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > > > > Otherwise,
> > > > > > > they
> > > > > > > > >>> > > represent
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > newly
> > > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > ProducerEpoch
> > > > of
> > > > > > 0.
> > > > > > > Is
> > > > > > > > >>> that
> > > > > > > > >>> > > > right?
> > > > > > > > >>> > > > > > > This
> > > > > > > > >>> > > > > > > > >> seems
> > > > > > > > >>> > > > > > > > >> > > not
> > > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> provide
> > > > some
> > > > > > > > examples
> > > > > > > > >>> > like
> > > > > > > > >>> > > > what
> > > > > > > > >>> > > > > > > Artem
> > > > > > > > >>> > > > > > > > >> has
> > > > > > > > >>> > > > > > > > >> > > done
> > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > considered a
> > > > > > simpler
> > > > > > > > >>> design
> > > > > > > > >>> > > > where
> > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same
> > > value
> > > > > > (e.g.
> > > > > > > > >>> for the
> > > > > > > > >>> > > > > current
> > > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > TransactionStatus
> > > > > > > and
> > > > > > > > >>> epoch
> > > > > > > > >>> > > > > > overflow?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103.
> It's
> > > not
> > > > > > clear
> > > > > > > to
> > > > > > > > >>> me
> > > > > > > > >>> > why
> > > > > > > > >>> > > we
> > > > > > > > >>> > > > > > need
> > > > > > > > >>> > > > > > > 3
> > > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > PrevProducerId,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > NextProducerId.
> > > > > > > Could
> > > > > > > > we
> > > > > > > > >>> > just
> > > > > > > > >>> > > > have
> > > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > > >>> > > > > if
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set
> the
> > > > > > > producerId
> > > > > > > > >>> and
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > >
> > > > > > > > >>> > >
> > > > > > > > >>> >
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Calvin Liu <ca...@confluent.io.INVALID>]

Hey team,
In order to address the performance issue in the
https://issues.apache.org/jira/browse/KAFKA-18654. We are making a change
to retry the AddPartitionToTxn requests on the server side when hitting the
CONCURRENT_TRANSACTION error. This error is common after the transaction is
committed but before the markers are fully propagated to all the involved
partitions. With the server side retry, the client can avoid the frequent
and expensive retry for the produce request. In order to better control the
retry behavior, we proposed to add 2 following configs.

   - *add.partitions.to.txn.max.timeout.ms
   <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum retry
   timeout when the server attempts to add the partition to the transaction.
   - *add.partitions.to.txn.retry.backoff.ms
   <http://add.partitions.to.txn.retry.backoff.ms>* defines how frequently
   the server will retry the AddPartitionToTxn.

Thanks!

On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan <jo...@confluent.io.invalid>
wrote:

> Hey folks, just wanted to share another quick update with KIP-890.
>
> Making the change to bump epoch after every transaction means that we no
> longer need to call InitProducerId during the life of the producer to fence
> requests from previous transactions. With KIP-890 part 2, the new client
> will no longer call InitProducerId except on startup to fence a previous
> instance.
>
> With this change, some other calls to InitProducerId were inspected
> including the call after receiving an InvalidPidMappingException. This
> exception was changed to abortable as part of KIP-360: Improve reliability
> of idempotent/transactional producer
> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> >.
> However, this change means that we can violate EOS guarantees. As an
> example:
>
> Consider an application that is copying data from one partition to another
>
>    - Application instance A processes to offset 4
>    - Application instance B comes up and fences application instance A
>    - Application instance B processes to offset 5
>    - Application instances A and B are idle for
> transaction.id.expiration.ms,
>    transaction id expires on server
>    - Application instance A attempts to process offset 5 (since in its
>    view, that is next) -- if we recover from invalid pid mapping, we can
>    duplicate this processing
>
> Thus, INVALID_PID_MAPPING should be fatal to the producer and as part of
> KIP-890 the new client in 4.0 will treat the exception as fatal.
>
> This is consistent with KIP-1050: Consistent error handling for
> Transactions
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> >
> where
> errors that are fatal to the producer are in the "application recoverable"
> category. This is a grouping that indicates to the client that the producer
> needs to restart and recovery on the application side is necessary.
> KIP-1050 is approved so we are consistent with that decision.
>
> Let me know if there are any questions. I've updated the KIP to reflect
> this change.
>
> Justine
>
> On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Sounds good, Justine. It would be useful to document that in the KIP.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > <jo...@confluent.io.invalid>
> > wrote:
> >
> > > The client will send the newest EndTxn request version if and only if
> > both
> > > the client and the server support kip-890 part 2.
> > > We set the value in the record based on the EndTxn version.
> > >
> > > Justine
> > >
> > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >
> > > > Hi, Justine,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 120. If the broker sends TV Y for the finalized version in
> > > > ApiVersionResponse, but the client doesn't support Y, how does the
> > broker
> > > > know the TV that the client supports?
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > <jo...@confluent.io.invalid>
> > > > wrote:
> > > >
> > > > > Hey Jun,
> > > > >
> > > > > No worries. Work on this KIP has been blocked for a bit anyways --
> > > > catching
> > > > > up and rereading what I wrote :)
> > > > >
> > > > > 120. ClientTransactionProtocolVersion is the transaction version as
> > > > defined
> > > > > by the highest transaction version (feature version value)
> supported
> > by
> > > > the
> > > > > client and the server. This works by the broker sending an
> > > > > ApiVersionsRequest to the client with the finalized version.
> Assuming
> > > > > kip-890 part 2 is enabled by transaction version Y, if this request
> > > > > contains finalized version Y and the client has the logic to set
> this
> > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2 not
> > > enable)
> > > > > the client will send Y - 1, even though the client has the ability
> to
> > > > > support kip-890 part 2.
> > > > >
> > > > > 121. You are correct that this is not needed. However, currently
> that
> > > > field
> > > > > is already being set in memory -- just not written to disk. I think
> > it
> > > is
> > > > > ok to write it to disk though. Let me know if you think otherwise.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <ju...@confluent.io.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi, Justine,
> > > > > >
> > > > > > Thanks for the update and sorry for the late reply.
> > > > > >
> > > > > > 120. I am wondering what value is used for
> > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > EndTxnRequest?
> > > > > >
> > > > > > 121. Earlier, you made the change to set lastProducerId in
> PREPARE
> > to
> > > > > > indicate that the marker is written for the new client. With the
> > new
> > > > > > ClientTransactionProtocolVersion field, it seems this is no
> longer
> > > > > > necessary.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > <jo...@confluent.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi there -- another update!
> > > > > > >
> > > > > > > When looking into the implementation for the safe epoch bumps I
> > > > > realized
> > > > > > > that we are already populating previousProducerID in memory as
> > part
> > > > of
> > > > > > > KIP-360.
> > > > > > > If we are to start using flexible fields, it is better to
> always
> > > use
> > > > > this
> > > > > > > information and have an explicit (tagged) field to indicate
> > whether
> > > > the
> > > > > > > client supports KIP-890 part 2.
> > > > > > >
> > > > > > > I've included the extra field and how it is set in the KIP.
> I've
> > > also
> > > > > > > updated the KIP to explain that we will be setting the tagged
> > > fields
> > > > > when
> > > > > > > they are available for all transitions.
> > > > > > >
> > > > > > > Finally, I added clearer text about the transaction protocol
> > > versions
> > > > > > > included with this KIP. 1 for flexible transaction state
> records
> > > and
> > > > 2
> > > > > > for
> > > > > > > KIP-890 part 2 enablement.
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > jolshan@confluent.io
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey there -- small update to the KIP,
> > > > > > > >
> > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > > > TxnOffsetCommit
> > > > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > AbortableTransactionException to match the pattern we had for
> > > other
> > > > > > > errors.
> > > > > > > > I also mentioned bumping all 6 transactional APIs so we can
> > > future
> > > > > > > > proof/support the error on the client going forward. If a
> > future
> > > > > change
> > > > > > > > wants to have an error scenario that requires us to abort the
> > > > > > > transaction,
> > > > > > > > we can rely on the 3.8+ clients to support it. We ran into
> > issues
> > > > > > finding
> > > > > > > > good/generic error codes that older clients could support
> while
> > > > > working
> > > > > > > on
> > > > > > > > this KIP, so this should help in the future.
> > > > > > > >
> > > > > > > > The features discussion is still ongoing in KIP-1022. Will
> > update
> > > > > again
> > > > > > > > here when that concludes.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > jolshan@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> I don't think AddPartitions is a good example since we
> > currenly
> > > > > don't
> > > > > > > >> gate the version on TV or MV. (We only set a different flag
> > > > > depending
> > > > > > on
> > > > > > > >> the TV)
> > > > > > > >>
> > > > > > > >> Even if we did want to gate it on TV, I think the idea is to
> > > move
> > > > > away
> > > > > > > >> from MV gating inter broker protocols. Ideally we can get
> to a
> > > > state
> > > > > > > where
> > > > > > > >> MV is just used for metadata changes.
> > > > > > > >>
> > > > > > > >> I think some of this discussion might fit more with the
> > feature
> > > > > > version
> > > > > > > >> KIP, so I can try to open that up soon. Until we settle
> that,
> > > some
> > > > > of
> > > > > > > the
> > > > > > > >> work in KIP-890 is blocked.
> > > > > > > >>
> > > > > > > >> Justine
> > > > > > > >>
> > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Hi, Justine,
> > > > > > > >>>
> > > > > > > >>> Thanks for the reply.
> > > > > > > >>>
> > > > > > > >>> Since AddPartitions is an inter broker request, will its
> > > version
> > > > be
> > > > > > > gated
> > > > > > > >>> only by TV or other features like MV too? For example, if
> we
> > > need
> > > > > to
> > > > > > > >>> change
> > > > > > > >>> the protocol for AddPartitions for reasons other than txn
> > > > > > verification
> > > > > > > in
> > > > > > > >>> the future, will the new version be gated by a new MV? If
> so,
> > > > does
> > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > >>>
> > > > > > > >>> Jun
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > involved,
> > > > only
> > > > > > the
> > > > > > > >>> > transactional records that should only be gated by TV)
> > > > > > > >>> > Another TV gates the ability to turn on kip-890 part 2.
> > This
> > > > > would
> > > > > > > >>> gate the
> > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > transactions),
> > > > > > and
> > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the version
> > is
> > > > > > already
> > > > > > > >>> used
> > > > > > > >>> > without TV.
> > > > > > > >>> >
> > > > > > > >>> > I think the only concern is the Produce request and we
> > could
> > > > > > consider
> > > > > > > >>> work
> > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > >>> >
> > > > > > > >>> > Justine
> > > > > > > >>> >
> > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > <jun@confluent.io.invalid
> > > > > >
> > > > > > > >>> wrote:
> > > > > > > >>> >
> > > > > > > >>> > > Hi, Justine,
> > > > > > > >>> > >
> > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> forward,
> > > will
> > > > > > those
> > > > > > > >>> > > PRC/record protocols only be guarded by TV and not by
> > other
> > > > > > > features
> > > > > > > >>> like
> > > > > > > >>> > > MV?
> > > > > > > >>> > >
> > > > > > > >>> > > Thanks,
> > > > > > > >>> > >
> > > > > > > >>> > > Jun
> > > > > > > >>> > >
> > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > >>> > > >
> > > > > > > >>> > > wrote:
> > > > > > > >>> > >
> > > > > > > >>> > > > Hi Jun,
> > > > > > > >>> > > >
> > > > > > > >>> > > > Sorry I think I misunderstood your question or
> answered
> > > > > > > >>> incorrectly.
> > > > > > > >>> > The
> > > > > > > >>> > > TV
> > > > > > > >>> > > > version should ideally be fully independent from MV.
> > > > > > > >>> > > > At least for the changes I proposed, TV should not
> > affect
> > > > MV
> > > > > > and
> > > > > > > MV
> > > > > > > >>> > > should
> > > > > > > >>> > > > not affect TV/
> > > > > > > >>> > > >
> > > > > > > >>> > > > I think if we downgrade TV, only that feature should
> > > > > downgrade.
> > > > > > > >>> > Likewise
> > > > > > > >>> > > > the same with MV. The finalizedFeatures should just
> > > reflect
> > > > > the
> > > > > > > >>> feature
> > > > > > > >>> > > > downgrade we made.
> > > > > > > >>> > > >
> > > > > > > >>> > > > I also plan to write a new KIP for managing the disk
> > > format
> > > > > and
> > > > > > > >>> upgrade
> > > > > > > >>> > > > tool as we will need new flags to support these
> > features.
> > > > > That
> > > > > > > >>> should
> > > > > > > >>> > > help
> > > > > > > >>> > > > clarify some things.
> > > > > > > >>> > > >
> > > > > > > >>> > > > Justine
> > > > > > > >>> > > >
> > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > <jun@confluent.io.invalid
> > > > > > > >
> > > > > > > >>> > > wrote:
> > > > > > > >>> > > >
> > > > > > > >>> > > > > Hi, Justine,
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> downgrade
> > > > > another
> > > > > > > >>> feature
> > > > > > > >>> > > > (say
> > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would we
> > > return
> > > > > for
> > > > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in
> > that
> > > > > case?
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > Thanks,
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > Jun
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > wrote:
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > > Hey Jun,
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > (transaction
> > > > > > > version)
> > > > > > > >>> we
> > > > > > > >>> > > will
> > > > > > > >>> > > > > stop
> > > > > > > >>> > > > > > using the add partitions to txn optimization and
> > stop
> > > > > > writing
> > > > > > > >>> the
> > > > > > > >>> > > > > flexible
> > > > > > > >>> > > > > > feature version of the log.
> > > > > > > >>> > > > > > In the compatibility section I included some
> > > > explanations
> > > > > > on
> > > > > > > >>> how
> > > > > > > >>> > this
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > done.
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > Thanks,
> > > > > > > >>> > > > > > Justine
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > >>> > > > > wrote:
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > If we ever downgrade the transaction feature,
> any
> > > > > feature
> > > > > > > >>> > depending
> > > > > > > >>> > > > on
> > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > changes
> > > > > > > made
> > > > > > > >>> in
> > > > > > > >>> > > > KIP-890
> > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > Jun
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > >>> > > > > > > wrote:
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > questions
> > > > > about
> > > > > > > >>> MV/IBP.
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > Looking at the options, I think it makes the
> > most
> > > > > sense
> > > > > > > to
> > > > > > > >>> > > create a
> > > > > > > >>> > > > > > > > separate feature for transactions and use
> that
> > to
> > > > > > version
> > > > > > > >>> gate
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > features
> > > > > > > >>> > > > > > > > we need to version gate (flexible
> transactional
> > > > state
> > > > > > > >>> records
> > > > > > > >>> > and
> > > > > > > >>> > > > > using
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > new protocol)
> > > > > > > >>> > > > > > > > I've updated the KIP to include this change.
> > > > > Hopefully
> > > > > > > >>> that's
> > > > > > > >>> > > > > > everything
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > Justine
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> Olshan
> > <
> > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > > > wrote:
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > I will update the KIP with the prev field
> for
> > > > > prepare
> > > > > > > as
> > > > > > > >>> > well.
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty or z
> if
> > y
> > > > will
> > > > > > > >>> overflow
> > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > >>> > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > >>> > > > > > > > >> (1) My point was that the coordinator
> could
> > > fail
> > > > > > right
> > > > > > > >>> after
> > > > > > > >>> > > > > writing
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> coordinator
> > > > > > generates
> > > > > > > >>> the
> > > > > > > >>> > > > > complete
> > > > > > > >>> > > > > > > > marker
> > > > > > > >>> > > > > > > > >> after the failover, it needs some field
> from
> > > the
> > > > > > > prepare
> > > > > > > >>> > > marker
> > > > > > > >>> > > > to
> > > > > > > >>> > > > > > > > >> determine whether it's written by the new
> > > > client.
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> (2) The changing of the behavior sounds
> good
> > > to
> > > > > me.
> > > > > > We
> > > > > > > >>> only
> > > > > > > >>> > > want
> > > > > > > >>> > > > > to
> > > > > > > >>> > > > > > > > return
> > > > > > > >>> > > > > > > > >> success if the prepare state is written by
> > the
> > > > new
> > > > > > > >>> client.
> > > > > > > >>> > So,
> > > > > > > >>> > > > in
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we also
> > need
> > > > sth
> > > > > in
> > > > > > > the
> > > > > > > >>> > > prepare
> > > > > > > >>> > > > > > > marker
> > > > > > > >>> > > > > > > > to
> > > > > > > >>> > > > > > > > >> tell us whether it's written by the new
> > > client.
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That
> sounds
> > > > good
> > > > > to
> > > > > > > me.
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> Jun
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine
> > > Olshan
> > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually have
> two
> > > > > > questions.
> > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need
> to
> > > > write
> > > > > > the
> > > > > > > >>> > > previous
> > > > > > > >>> > > > > > > produce
> > > > > > > >>> > > > > > > > Id
> > > > > > > >>> > > > > > > > >> > tagged field in the end maker so that we
> > > know
> > > > if
> > > > > > the
> > > > > > > >>> > marker
> > > > > > > >>> > > is
> > > > > > > >>> > > > > > from
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> new
> > > > > > > >>> > > > > > > > >> > client. Since the end maker is derived
> > from
> > > > the
> > > > > > > >>> prepare
> > > > > > > >>> > > > marker,
> > > > > > > >>> > > > > > > should
> > > > > > > >>> > > > > > > > >> we
> > > > > > > >>> > > > > > > > >> > write the previous produce Id in the
> > prepare
> > > > > > marker
> > > > > > > >>> field
> > > > > > > >>> > > too?
> > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > >>> > > > > > > > >> > we will lose this information when
> > deriving
> > > > the
> > > > > > end
> > > > > > > >>> > marker.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in the
> > normal
> > > > > > producer
> > > > > > > >>> ID
> > > > > > > >>> > > field.
> > > > > > > >>> > > > > So
> > > > > > > >>> > > > > > > yes,
> > > > > > > >>> > > > > > > > >> we
> > > > > > > >>> > > > > > > > >> > need it in prepare and that was always
> the
> > > > plan.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> > > enumerate
> > > > > the
> > > > > > > >>> fields
> > > > > > > >>> > and
> > > > > > > >>> > > > add
> > > > > > > >>> > > > > > > them
> > > > > > > >>> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch y.
> > When
> > > we
> > > > > > > >>> overflow
> > > > > > > >>> > > epoch
> > > > > > > >>> > > > y
> > > > > > > >>> > > > > we
> > > > > > > >>> > > > > > > get
> > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field):
> > > empty
> > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty or
> z
> > > if y
> > > > > > will
> > > > > > > >>> > overflow
> > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field):
> x
> > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> overflowed
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry
> and
> > > see
> > > > > > epoch
> > > > > > > -
> > > > > > > >>> 1 +
> > > > > > > >>> > ID
> > > > > > > >>> > > > in
> > > > > > > >>> > > > > > last
> > > > > > > >>> > > > > > > > >> seen
> > > > > > > >>> > > > > > > > >> > fields and are issuing the same command
> > (ie
> > > > > commit
> > > > > > > not
> > > > > > > >>> > > abort),
> > > > > > > >>> > > > > we
> > > > > > > >>> > > > > > > > return
> > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890 seems
> to
> > > > > return
> > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> making
> > > this
> > > > > > > change?
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer if
> the
> > > > epoch
> > > > > is
> > > > > > > >>> bumped
> > > > > > > >>> > > and
> > > > > > > >>> > > > we
> > > > > > > >>> > > > > > > get a
> > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally
> > > > adding
> > > > > > this
> > > > > > > >>> to
> > > > > > > >>> > > > prevent
> > > > > > > >>> > > > > > > > fencing.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > 112. We already merged the code that
> adds
> > > the
> > > > > > > >>> VerifyOnly
> > > > > > > >>> > > field
> > > > > > > >>> > > > > in
> > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an
> > inter
> > > > > > broker
> > > > > > > >>> > request.
> > > > > > > >>> > > > It
> > > > > > > >>> > > > > > > seems
> > > > > > > >>> > > > > > > > >> that
> > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do
> you
> > > > know
> > > > > > why?
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> interbroker
> > > > > requests
> > > > > > > as
> > > > > > > >>> > > > > ApiVersions
> > > > > > > >>> > > > > > > > should
> > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> errors
> > > > > > correctly
> > > > > > > >>> if we
> > > > > > > >>> > > > > receive
> > > > > > > >>> > > > > > > them
> > > > > > > >>> > > > > > > > >> in
> > > > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually have
> > two
> > > > > > > questions.
> > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need
> to
> > > > write
> > > > > > the
> > > > > > > >>> > > previous
> > > > > > > >>> > > > > > > produce
> > > > > > > >>> > > > > > > > Id
> > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so that
> we
> > > > know
> > > > > if
> > > > > > > the
> > > > > > > >>> > > marker
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > > from
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > new
> > > > > > > >>> > > > > > > > >> > > client. Since the end maker is derived
> > > from
> > > > > the
> > > > > > > >>> prepare
> > > > > > > >>> > > > > marker,
> > > > > > > >>> > > > > > > > >> should we
> > > > > > > >>> > > > > > > > >> > > write the previous produce Id in the
> > > prepare
> > > > > > > marker
> > > > > > > >>> > field
> > > > > > > >>> > > > too?
> > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > >>> > > > > > > > >> > > we will lose this information when
> > > deriving
> > > > > the
> > > > > > > end
> > > > > > > >>> > > marker.
> > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we retry
> > and
> > > > see
> > > > > > > epoch
> > > > > > > >>> - 1
> > > > > > > >>> > +
> > > > > > > >>> > > ID
> > > > > > > >>> > > > > in
> > > > > > > >>> > > > > > > last
> > > > > > > >>> > > > > > > > >> seen
> > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> command
> > > (ie
> > > > > > commit
> > > > > > > >>> not
> > > > > > > >>> > > > abort),
> > > > > > > >>> > > > > > we
> > > > > > > >>> > > > > > > > >> return
> > > > > > > >>> > > > > > > > >> > > success. The logic before KIP-890
> seems
> > to
> > > > > > return
> > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > >>> > > > > > > > >> > > in this case. Are we intentionally
> > making
> > > > this
> > > > > > > >>> change?
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > 112. We already merged the code that
> > adds
> > > > the
> > > > > > > >>> VerifyOnly
> > > > > > > >>> > > > field
> > > > > > > >>> > > > > > in
> > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an
> > > inter
> > > > > > > broker
> > > > > > > >>> > > request.
> > > > > > > >>> > > > > It
> > > > > > > >>> > > > > > > > seems
> > > > > > > >>> > > > > > > > >> > that
> > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do
> > you
> > > > > know
> > > > > > > why?
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> Justine
> > > > Olshan
> > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to
> > > "current
> > > > > > > >>> producer id
> > > > > > > >>> > > and
> > > > > > > >>> > > > > > > epoch"
> > > > > > > >>> > > > > > > > if
> > > > > > > >>> > > > > > > > >> > that
> > > > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am still
> > > > > > finalizing
> > > > > > > >>> the
> > > > > > > >>> > > > design.
> > > > > > > >>> > > > > I
> > > > > > > >>> > > > > > > will
> > > > > > > >>> > > > > > > > >> > update
> > > > > > > >>> > > > > > > > >> > > > the KIP once I get the information
> > > > > finalized.
> > > > > > > >>> Sorry
> > > > > > > >>> > for
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > delays.
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun
> > Rao
> > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case,
> the
> > > > > previous
> > > > > > > ID
> > > > > > > >>> is
> > > > > > > >>> > the
> > > > > > > >>> > > > > same
> > > > > > > >>> > > > > > as
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker too,
> but
> > we
> > > > set
> > > > > > the
> > > > > > > >>> > > previous
> > > > > > > >>> > > > ID
> > > > > > > >>> > > > > > in
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned that
> > > this
> > > > is
> > > > > > to
> > > > > > > >>> know
> > > > > > > >>> > > that
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> marker is
> > > > > > > >>> > > > > > > > >> > > > > written by the new client so that
> we
> > > > could
> > > > > > > >>> return
> > > > > > > >>> > > > success
> > > > > > > >>> > > > > on
> > > > > > > >>> > > > > > > > >> retried
> > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was trying
> to
> > > > > > understand
> > > > > > > >>> why
> > > > > > > >>> > > this
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > not
> > > > > > > >>> > > > > > > > >> needed
> > > > > > > >>> > > > > > > > >> > > for
> > > > > > > >>> > > > > > > > >> > > > > the prepare marker since retry can
> > > > happen
> > > > > in
> > > > > > > the
> > > > > > > >>> > > prepare
> > > > > > > >>> > > > > > state
> > > > > > > >>> > > > > > > > >> too.
> > > > > > > >>> > > > > > > > >> > Is
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare state,
> we
> > > > > return
> > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> > > requests?
> > > > > If
> > > > > > > so,
> > > > > > > >>> > should
> > > > > > > >>> > > > we
> > > > > > > >>> > > > > > > change
> > > > > > > >>> > > > > > > > >> "If
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in
> last
> > > > seen
> > > > > > > >>> fields and
> > > > > > > >>> > > are
> > > > > > > >>> > > > > > > issuing
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > same
> > > > > > > >>> > > > > > > > >> > > > > command (ie commit not abort) we
> can
> > > > > return
> > > > > > > >>> (with
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > epoch)"
> > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a
> > delayed
> > > > > > endMaker
> > > > > > > >>> > request
> > > > > > > >>> > > > > > carries
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > same
> > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This
> can
> > > > > > > >>> commit/abort a
> > > > > > > >>> > > > future
> > > > > > > >>> > > > > > > > >> transaction
> > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we
> > have
> > > > > seen
> > > > > > > >>> this in
> > > > > > > >>> > > > > practice
> > > > > > > >>> > > > > > > > >> though.
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be
> useful
> > > to
> > > > > make
> > > > > > > it
> > > > > > > >>> > clear
> > > > > > > >>> > > > that
> > > > > > > >>> > > > > > we
> > > > > > > >>> > > > > > > > can
> > > > > > > >>> > > > > > > > >> now
> > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field from
> the
> > > log
> > > > > > > >>> reliably.
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > >>> > > > > since
> > > > > > > >>> > > > > > > > it's
> > > > > > > >>> > > > > > > > >> > > called
> > > > > > > >>> > > > > > > > >> > > > > across brokers and we are changing
> > its
> > > > > > schema.
> > > > > > > >>> Are
> > > > > > > >>> > you
> > > > > > > >>> > > > > > saying
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > >> > don't
> > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that we
> > already
> > > > > > > >>> implemented
> > > > > > > >>> > the
> > > > > > > >>> > > > > > server
> > > > > > > >>> > > > > > > > side
> > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > >>> > > > > across
> > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM
> > > Justine
> > > > > > Olshan
> > > > > > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the previous
> ID
> > > in
> > > > > the
> > > > > > > >>> Prepare
> > > > > > > >>> > > > field
> > > > > > > >>> > > > > > > since
> > > > > > > >>> > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > >>> > > > > > > > >> > > > > > it. It is the same producer ID
> as
> > > the
> > > > > main
> > > > > > > >>> > producer
> > > > > > > >>> > > ID
> > > > > > > >>> > > > > > > field.
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> reread
> > > your
> > > > > > > message
> > > > > > > >>> > about
> > > > > > > >>> > > > > > delayed
> > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn marker
> > > after
> > > > > the
> > > > > > > >>> > > transaction
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > > > already
> > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > transaction
> > > > > > early
> > > > > > > >>> > without
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > fixes
> > > > > > > >>> > > > > > > > >> in
> > > > > > > >>> > > > > > > > >> > > part
> > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was
> > used
> > > > in
> > > > > a
> > > > > > > >>> previous
> > > > > > > >>> > > KIP
> > > > > > > >>> > > > > and
> > > > > > > >>> > > > > > > > never
> > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log --
> only
> > in
> > > > > > memory
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> > > protocol
> > > > > are
> > > > > > > you
> > > > > > > >>> > > > referring
> > > > > > > >>> > > > > > to?
> > > > > > > >>> > > > > > > I
> > > > > > > >>> > > > > > > > am
> > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> remove
> > > the
> > > > > > extra
> > > > > > > >>> add
> > > > > > > >>> > > > > partitions
> > > > > > > >>> > > > > > > > call
> > > > > > > >>> > > > > > > > >> > and I
> > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I have
> > yet
> > > to
> > > > > > > update
> > > > > > > >>> that
> > > > > > > >>> > > > > section
> > > > > > > >>> > > > > > > as
> > > > > > > >>> > > > > > > > I
> > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > >>> > > > > > > > >> > > > > > the design so please stay tuned.
> > Was
> > > > > there
> > > > > > > >>> > anything
> > > > > > > >>> > > > else
> > > > > > > >>> > > > > > you
> > > > > > > >>> > > > > > > > >> > thought
> > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM
> > Jun
> > > > Rao
> > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create any
> > issue.
> > > > It
> > > > > > just
> > > > > > > >>> makes
> > > > > > > >>> > > it
> > > > > > > >>> > > > a
> > > > > > > >>> > > > > > bit
> > > > > > > >>> > > > > > > > >> hard to
> > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged produce
> id
> > > > field
> > > > > > > >>> means. We
> > > > > > > >>> > > are
> > > > > > > >>> > > > > > > > >> essentially
> > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> (completing
> > a
> > > > txn
> > > > > > and
> > > > > > > >>> init a
> > > > > > > >>> > > new
> > > > > > > >>> > > > > > > produce
> > > > > > > >>> > > > > > > > >> Id)
> > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this may
> be
> > > fine
> > > > > > too.
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> comments.
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that
> we
> > > > only
> > > > > > set
> > > > > > > >>> the
> > > > > > > >>> > > > previous
> > > > > > > >>> > > > > > > > >> produce id
> > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > >>> > > > > > > > >> > > > > > > field in the complete marker,
> > but
> > > > not
> > > > > in
> > > > > > > the
> > > > > > > >>> > > prepare
> > > > > > > >>> > > > > > > marker,
> > > > > > > >>> > > > > > > > >> is
> > > > > > > >>> > > > > > > > >> > > that
> > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> return
> > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > >>> > > > > > on
> > > > > > > >>> > > > > > > > >> > retried
> > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your second
> > point
> > > is
> > > > > > > >>> mentioned
> > > > > > > >>> > in
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > KIP. I
> > > > > > > >>> > > > > > > > >> can
> > > > > > > >>> > > > > > > > >> > > add
> > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message case can
> > > also
> > > > > > > violate
> > > > > > > >>> EOS
> > > > > > > >>> > if
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > delayed
> > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > addPartitionsToTxn
> > > > > > > >>> > request
> > > > > > > >>> > > > > comes
> > > > > > > >>> > > > > > > in.
> > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > previous
> > > > > > > (aborted)
> > > > > > > >>> > > > > transaction
> > > > > > > >>> > > > > > > > become
> > > > > > > >>> > > > > > > > >> > part
> > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > The above is the case when a
> > > delayed
> > > > > > > >>> message is
> > > > > > > >>> > > > > appended
> > > > > > > >>> > > > > > > to
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > data
> > > > > > > >>> > > > > > > > >> > > > > > > partition. What I mentioned
> is a
> > > > > > slightly
> > > > > > > >>> > > different
> > > > > > > >>> > > > > case
> > > > > > > >>> > > > > > > > when
> > > > > > > >>> > > > > > > > >> a
> > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > > > transaction
> > > > > > log
> > > > > > > >>> > > partition.
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we
> move
> > > past
> > > > > the
> > > > > > > >>> Prepare
> > > > > > > >>> > > and
> > > > > > > >>> > > > > > > > Complete
> > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about
> > lastSeen
> > > > > > fields
> > > > > > > >>> and
> > > > > > > >>> > > clear
> > > > > > > >>> > > > > > them,
> > > > > > > >>> > > > > > > > just
> > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is
> the
> > > > > lastSeen
> > > > > > > >>> field
> > > > > > > >>> > the
> > > > > > > >>> > > > > same
> > > > > > > >>> > > > > > as
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > > > >>> TransactionLogValue?
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> > > > > > > inter-broker
> > > > > > > >>> > > > protocol,
> > > > > > > >>> > > > > > > should
> > > > > > > >>> > > > > > > > >> we
> > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> feature
> > > only
> > > > > for
> > > > > > > the
> > > > > > > >>> > KRaft
> > > > > > > >>> > > > > mode?
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> 11:13 AM
> > > > > Justine
> > > > > > > >>> Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > <jolshan@confluent.io.invalid
> >
> > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> > > > > convergence
> > > > > > > on
> > > > > > > >>> the
> > > > > > > >>> > > > > design.
> > > > > > > >>> > > > > > :)
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it seems
> a
> > > > little
> > > > > > > >>> "weird".
> > > > > > > >>> > > I'm
> > > > > > > >>> > > > > not
> > > > > > > >>> > > > > > > sure
> > > > > > > >>> > > > > > > > >> what
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra record
> to
> > > the
> > > > > log.
> > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> > > describe
> > > > > > > >>> transactions
> > > > > > > >>> > > > won't
> > > > > > > >>> > > > > > > work
> > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to calculate
> > the
> > > > > time
> > > > > > > >>> since
> > > > > > > >>> > the
> > > > > > > >>> > > > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like
> this,
> > > it
> > > > is
> > > > > > > >>> enough to
> > > > > > > >>> > > > > > convince
> > > > > > > >>> > > > > > > me
> > > > > > > >>> > > > > > > > >> we
> > > > > > > >>> > > > > > > > >> > > need
> > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems like
> it
> > > > would
> > > > > > be
> > > > > > > >>> > > replacing
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > record
> > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this
> > correct?
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at
> > 5:14 PM
> > > > Jun
> > > > > > Rao
> > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> explanation.
> > I
> > > > > > > >>> understand the
> > > > > > > >>> > > > > > intention
> > > > > > > >>> > > > > > > > >> now.
> > > > > > > >>> > > > > > > > >> > In
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> non-tagged
> > > > field
> > > > > to
> > > > > > > >>> the old
> > > > > > > >>> > > pid
> > > > > > > >>> > > > > > (and
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> max
> > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that we
> > > could
> > > > > > > >>> correctly
> > > > > > > >>> > > write
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> marker to
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> > > > > downgrades.
> > > > > > > When
> > > > > > > >>> > > writing
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> complete
> > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> already
> > > been
> > > > > > > >>> written to
> > > > > > > >>> > > the
> > > > > > > >>> > > > > data
> > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the
> new
> > > pid
> > > > to
> > > > > > > avoid
> > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to work.
> > It's
> > > > > just a
> > > > > > > bit
> > > > > > > >>> > > > > > inconsistent
> > > > > > > >>> > > > > > > > for
> > > > > > > >>> > > > > > > > >> a
> > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker to
> use
> > > > > > different
> > > > > > > >>> pids
> > > > > > > >>> > in
> > > > > > > >>> > > > > this
> > > > > > > >>> > > > > > > > >> special
> > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> complete
> > > > > marker,
> > > > > > it
> > > > > > > >>> seems
> > > > > > > >>> > > > that
> > > > > > > >>> > > > > we
> > > > > > > >>> > > > > > > > will
> > > > > > > >>> > > > > > > > >> > never
> > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete marker
> > with
> > > > the
> > > > > > old
> > > > > > > >>> pid.
> > > > > > > >>> > > Not
> > > > > > > >>> > > > > sure
> > > > > > > >>> > > > > > > if
> > > > > > > >>> > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit
> > > weird.
> > > > > > > >>> Instead of
> > > > > > > >>> > > > > writing
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid, could we
> > > write
> > > > > two
> > > > > > > >>> > records:
> > > > > > > >>> > > a
> > > > > > > >>> > > > > > > complete
> > > > > > > >>> > > > > > > > >> > marker
> > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > > > >>> TransactionLogValue
> > > > > > > >>> > with
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > new
> > > > > > > >>> > > > > > > > pid
> > > > > > > >>> > > > > > > > >> > and
> > > > > > > >>> > > > > > > > >> > > an
> > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make the
> two
> > > > > records
> > > > > > > in
> > > > > > > >>> the
> > > > > > > >>> > > same
> > > > > > > >>> > > > > > batch
> > > > > > > >>> > > > > > > > so
> > > > > > > >>> > > > > > > > >> > that
> > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> atomically.
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at
> > > 5:40 PM
> > > > > > > Justine
> > > > > > > >>> > Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > <jolshan@confluent.io.invalid
> > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker
> is
> > > > > written,
> > > > > > > >>> but the
> > > > > > > >>> > > > > endTxn
> > > > > > > >>> > > > > > > > >> response
> > > > > > > >>> > > > > > > > >> > is
> > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when the
> > > server
> > > > > > > >>> downgrades
> > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker
> is
> > > > > > written,
> > > > > > > >>> the
> > > > > > > >>> > > endTxn
> > > > > > > >>> > > > > > > > response
> > > > > > > >>> > > > > > > > >> is
> > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the server
> > > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a
> > little
> > > > > > > confused.
> > > > > > > >>> In
> > > > > > > >>> > > both
> > > > > > > >>> > > > of
> > > > > > > >>> > > > > > > these
> > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has the
> > old
> > > > > > producer
> > > > > > > >>> ID.
> > > > > > > >>> > We
> > > > > > > >>> > > > > don't
> > > > > > > >>> > > > > > > > write
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare marker's
> > non
> > > > > tagged
> > > > > > > >>> fields.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server downgrades
> > > now,
> > > > it
> > > > > > > would
> > > > > > > >>> > read
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > records
> > > > > > > >>> > > > > > > > >> not
> > > > > > > >>> > > > > > > > >> > > in
> > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the complete
> > > marker
> > > > > > will
> > > > > > > >>> also
> > > > > > > >>> > > have
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > old
> > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the new
> > > > producer
> > > > > > ID,
> > > > > > > >>> we
> > > > > > > >>> > > would
> > > > > > > >>> > > > > not
> > > > > > > >>> > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since the
> > > producer
> > > > > id
> > > > > > > >>> doesn't
> > > > > > > >>> > > > match
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> correct
> > > on
> > > > > the
> > > > > > > data
> > > > > > > >>> > > > > partition.)
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow case,
> I'd
> > > > expect
> > > > > > the
> > > > > > > >>> > > following
> > > > > > > >>> > > > > to
> > > > > > > >>> > > > > > > > >> happen on
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry
> EndTxn
> > > --
> > > > it
> > > > > > is
> > > > > > > >>> the
> > > > > > > >>> > same
> > > > > > > >>> > > > > > > producer
> > > > > > > >>> > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > and
> > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry
> > > > EndTxn
> > > > > > and
> > > > > > > >>> use
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > >> producer id
> > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > InvalidPidMappingException
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> special
> > > > > handling
> > > > > > > for
> > > > > > > >>> > when
> > > > > > > >>> > > a
> > > > > > > >>> > > > > > server
> > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could get
> an
> > > API
> > > > > > > version
> > > > > > > >>> > > request
> > > > > > > >>> > > > > > > showing
> > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In that
> > case,
> > > > we
> > > > > > can
> > > > > > > >>> call
> > > > > > > >>> > > > > > > > initProducerId
> > > > > > > >>> > > > > > > > >> to
> > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> > > overflow
> > > > > > case,
> > > > > > > >>> this
> > > > > > > >>> > > > > correctly
> > > > > > > >>> > > > > > > > gives
> > > > > > > >>> > > > > > > > >> > us a
> > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> corresponding
> > > case
> > > > > > would
> > > > > > > >>> be
> > > > > > > >>> > > where
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> *complete
> > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn
> is
> > > not
> > > > > > > >>> received by
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > client
> > > > > > > >>> > > > > > > > and
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would
> > > result
> > > > in
> > > > > > the
> > > > > > > >>> > > > transaction
> > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old one.
> > If
> > > > the
> > > > > > > client
> > > > > > > >>> > > > retries,
> > > > > > > >>> > > > > it
> > > > > > > >>> > > > > > > > will
> > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > InvalidPidMappingException.
> > > > The
> > > > > > > >>> > > InitProducerId
> > > > > > > >>> > > > > > > > scenario
> > > > > > > >>> > > > > > > > >> > above
> > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > compatibility
> > > > > > story
> > > > > > > is
> > > > > > > >>> > meant
> > > > > > > >>> > > > to
> > > > > > > >>> > > > > > > > support
> > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > > > transactional
> > > > > > > >>> > > correctness.
> > > > > > > >>> > > > > > > Keeping
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not
> the
> > > > > > priority.
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can
> > also
> > > > add
> > > > > > text
> > > > > > > >>> in
> > > > > > > >>> > the
> > > > > > > >>> > > > KIP
> > > > > > > >>> > > > > > > about
> > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes some
> edge
> > > > > cases.
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at
> > > > 4:10 PM
> > > > > > Jun
> > > > > > > >>> Rao
> > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we don't
> > need
> > > > to
> > > > > > > >>> optimize
> > > > > > > >>> > for
> > > > > > > >>> > > > > > fencing
> > > > > > > >>> > > > > > > > >> during
> > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding consistency,
> > > there
> > > > > are
> > > > > > > two
> > > > > > > >>> > > > possible
> > > > > > > >>> > > > > > > cases:
> > > > > > > >>> > > > > > > > >> (1)
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but the
> > endTxn
> > > > > > > response
> > > > > > > >>> is
> > > > > > > >>> > not
> > > > > > > >>> > > > > > > received
> > > > > > > >>> > > > > > > > by
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)
> > > the
> > > > > > > prepare
> > > > > > > >>> > marker
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > > > written,
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the
> > client
> > > > when
> > > > > > the
> > > > > > > >>> > server
> > > > > > > >>> > > > > > > > downgrades.
> > > > > > > >>> > > > > > > > >> In
> > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old
> > produce
> > > Id
> > > > > and
> > > > > > > in
> > > > > > > >>> (2),
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > client
> > > > > > > >>> > > > > > > > >> will
> > > > > > > >>> > > > > > > > >> > > have
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> > > downgrade
> > > > > > right
> > > > > > > >>> after
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > prepare
> > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both (1)
> > and
> > > > (2)
> > > > > > > >>> since we
> > > > > > > >>> > > can
> > > > > > > >>> > > > > only
> > > > > > > >>> > > > > > > put
> > > > > > > >>> > > > > > > > >> one
> > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id
> > field.
> > > > > It's
> > > > > > > >>> also not
> > > > > > > >>> > > > clear
> > > > > > > >>> > > > > > > which
> > > > > > > >>> > > > > > > > >> case
> > > > > > > >>> > > > > > > > >> > > is
> > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could probably
> be
> > > > > > consistent
> > > > > > > >>> with
> > > > > > > >>> > > > either
> > > > > > > >>> > > > > > > case.
> > > > > > > >>> > > > > > > > >> By
> > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the
> > prepare
> > > > > > marker,
> > > > > > > >>> we
> > > > > > > >>> > are
> > > > > > > >>> > > > > > > consistent
> > > > > > > >>> > > > > > > > >> with
> > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the slight
> > > benefit
> > > > > that
> > > > > > > the
> > > > > > > >>> > > produce
> > > > > > > >>> > > > > > field
> > > > > > > >>> > > > > > > > in
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker are
> > > > consistent
> > > > > > in
> > > > > > > >>> the
> > > > > > > >>> > > > overflow
> > > > > > > >>> > > > > > > case.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 3:11 PM
> > > > > > > >>> Justine
> > > > > > > >>> > > > Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > <jolshan@confluent.io.invalid
> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> > > describe,
> > > > we
> > > > > > > would
> > > > > > > >>> > need
> > > > > > > >>> > > to
> > > > > > > >>> > > > > > have
> > > > > > > >>> > > > > > > a
> > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn,
> > and a
> > > > > > > >>> successful
> > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > >>> > > > > > > > >> > > and
> > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> request
> > > go
> > > > > > > through
> > > > > > > >>> for
> > > > > > > >>> > a
> > > > > > > >>> > > > > given
> > > > > > > >>> > > > > > > > >> producer.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure
> > out
> > > > if
> > > > > it
> > > > > > > is
> > > > > > > >>> > > possible
> > > > > > > >>> > > > > for
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> > client
> > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous request
> is
> > > > > delayed
> > > > > > > >>> > somewhere.
> > > > > > > >>> > > > But
> > > > > > > >>> > > > > > > yes,
> > > > > > > >>> > > > > > > > in
> > > > > > > >>> > > > > > > > >> > this
> > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the
> > client.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow
> > > case.
> > > > > In
> > > > > > > the
> > > > > > > >>> > > overflow
> > > > > > > >>> > > > > > case,
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are different
> on
> > > the
> > > > > > > marker
> > > > > > > >>> and
> > > > > > > >>> > on
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > new
> > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use
> the
> > > max
> > > > > > epoch
> > > > > > > >>> but
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> epoch 0
> > > in
> > > > > the
> > > > > > > >>> > > > transactional
> > > > > > > >>> > > > > > > state.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > downgrade
> > > > > case,
> > > > > > we
> > > > > > > >>> want
> > > > > > > >>> > to
> > > > > > > >>> > > > see
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> > producer
> > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client will
> > > have.
> > > > > If
> > > > > > we
> > > > > > > >>> > > complete
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> commit,
> > > > > > > >>> > > > > > > > >> > and
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded,
> we
> > > need
> > > > > the
> > > > > > > new
> > > > > > > >>> > > > producer
> > > > > > > >>> > > > > ID
> > > > > > > >>> > > > > > > in
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > state
> > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid producer
> ID
> > > > > > mapping.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server downgrade
> > > cases
> > > > > are
> > > > > > > >>> > > considering
> > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing from
> > > > previous
> > > > > > > >>> behavior
> > > > > > > >>> > --
> > > > > > > >>> > > > and
> > > > > > > >>> > > > > > are
> > > > > > > >>> > > > > > > > not
> > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> safety
> > > from
> > > > > > > fencing
> > > > > > > >>> > > retries
> > > > > > > >>> > > > > (as
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > >> have
> > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > support).
> > > > > > Perhaps
> > > > > > > >>> this
> > > > > > > >>> > is
> > > > > > > >>> > > a
> > > > > > > >>> > > > > > trade
> > > > > > > >>> > > > > > > > off,
> > > > > > > >>> > > > > > > > >> > but
> > > > > > > >>> > > > > > > > >> > > I
> > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > downgrades,
> > > > > it
> > > > > > > will
> > > > > > > >>> > have
> > > > > > > >>> > > > > > > restarted
> > > > > > > >>> > > > > > > > >> and
> > > > > > > >>> > > > > > > > >> > it
> > > > > > > >>> > > > > > > > >> > > is
> > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new producer
> ID
> > > > too).
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 11:42 AM
> > > > > > > >>> Jun
> > > > > > > >>> > Rao
> > > > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the
> > reply.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the
> marker
> > > is
> > > > > > > written
> > > > > > > >>> by
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > >> client, we
> > > > > > > >>> > > > > > > > >> > > can
> > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> > > guarantee
> > > > > > that
> > > > > > > >>> any
> > > > > > > >>> > > EndTxn
> > > > > > > >>> > > > > > > > requests
> > > > > > > >>> > > > > > > > >> > with
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same
> > producer
> > > > and
> > > > > > the
> > > > > > > >>> same
> > > > > > > >>> > > > > > > transaction.
> > > > > > > >>> > > > > > > > >> Then
> > > > > > > >>> > > > > > > > >> > > we
> > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced
> > error
> > > > but
> > > > > > can
> > > > > > > >>> handle
> > > > > > > >>> > > > > > > gracefully
> > > > > > > >>> > > > > > > > as
> > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed
> > EndTnx
> > > > > > request
> > > > > > > is
> > > > > > > >>> > > > > processed,
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> txn
> > > > > > > >>> > > > > > > > >> > > state
> > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next txn.
> I
> > > > guess
> > > > > in
> > > > > > > >>> this
> > > > > > > >>> > case
> > > > > > > >>> > > > we
> > > > > > > >>> > > > > > > still
> > > > > > > >>> > > > > > > > >> > return
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> request?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> > > question
> > > > > was
> > > > > > > >>> > > inaccurate.
> > > > > > > >>> > > > > What
> > > > > > > >>> > > > > > > you
> > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > > > compatibility I
> > > > > > > >>> > mention
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > that
> > > > > > > >>> > > > > > > > we
> > > > > > > >>> > > > > > > > >> > keep
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the
> > main
> > > > > > > >>> (non-tagged)
> > > > > > > >>> > > > fields
> > > > > > > >>> > > > > as
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > >> did
> > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If
> we
> > > want
> > > > > to
> > > > > > do
> > > > > > > >>> this,
> > > > > > > >>> > > it
> > > > > > > >>> > > > > > seems
> > > > > > > >>> > > > > > > > >> that we
> > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current produce Id
> > and
> > > > max
> > > > > > > >>> epoch in
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > existing
> > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch
> fields
> > > for
> > > > > > both
> > > > > > > >>> the
> > > > > > > >>> > > > prepare
> > > > > > > >>> > > > > > and
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can
> > > happen
> > > > > > after
> > > > > > > >>> the
> > > > > > > >>> > > > > complete
> > > > > > > >>> > > > > > > > >> marker is
> > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described, the
> > > > > > downgraded
> > > > > > > >>> > > > coordinator
> > > > > > > >>> > > > > > will
> > > > > > > >>> > > > > > > > see
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the old
> > > one.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 10:44 AM
> > > > > > > >>> > > Justine
> > > > > > > >>> > > > > > > Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > <jolshan@confluent.io.invalid
> > > > > > > >
> > > > > > > >>> > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> > > > > > > description.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your
> > > second
> > > > > > point
> > > > > > > is
> > > > > > > >>> > > > mentioned
> > > > > > > >>> > > > > > in
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > >>> > > > > > > > >> > > I
> > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is
> > > helpful.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed
> > > message
> > > > > case
> > > > > > > can
> > > > > > > >>> > also
> > > > > > > >>> > > > > > violate
> > > > > > > >>> > > > > > > > EOS
> > > > > > > >>> > > > > > > > >> if
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after
> the
> > > > next
> > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > >>> > > > > > > > >> request
> > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> message
> > > > from a
> > > > > > > >>> previous
> > > > > > > >>> > > > > > (aborted)
> > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> transaction.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is
> > > > written
> > > > > > by
> > > > > > > >>> the
> > > > > > > >>> > new
> > > > > > > >>> > > > > > client,
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > >> can
> > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > guarantee
> > > > > that
> > > > > > > any
> > > > > > > >>> > EndTxn
> > > > > > > >>> > > > > > > requests
> > > > > > > >>> > > > > > > > >> with
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> producer
> > > and
> > > > > the
> > > > > > > same
> > > > > > > >>> > > > > > transaction.
> > > > > > > >>> > > > > > > > >> Then we
> > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but
> > can
> > > > > > handle
> > > > > > > >>> > > gracefully
> > > > > > > >>> > > > > as
> > > > > > > >>> > > > > > > > >> described
> > > > > > > >>> > > > > > > > >> > > in
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a
> > > > boolean
> > > > > is
> > > > > > > >>> useful
> > > > > > > >>> > > > since
> > > > > > > >>> > > > > it
> > > > > > > >>> > > > > > > is
> > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or
> lack
> > of
> > > > the
> > > > > > > >>> tagged
> > > > > > > >>> > > field
> > > > > > > >>> > > > > > being
> > > > > > > >>> > > > > > > > >> > written.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare
> > > marker
> > > > we
> > > > > > > will
> > > > > > > >>> have
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > same
> > > > > > > >>> > > > > > > > >> > producer
> > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the
> > > Complete
> > > > > > state
> > > > > > > >>> we
> > > > > > > >>> > may
> > > > > > > >>> > > > not.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why
> > the
> > > > > > ongoing
> > > > > > > >>> state
> > > > > > > >>> > > > > matters
> > > > > > > >>> > > > > > > for
> > > > > > > >>> > > > > > > > >> this
> > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> what
> > > you
> > > > > are
> > > > > > > >>> > referring
> > > > > > > >>> > > to
> > > > > > > >>> > > > > > about
> > > > > > > >>> > > > > > > > >> > writing
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> prepare
> > > > > marker.
> > > > > > > >>> This is
> > > > > > > >>> > > not
> > > > > > > >>> > > > in
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow
> > > case,
> > > > we
> > > > > > > >>> write the
> > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > >>> > > > > > > > >> > in
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we
> know
> > > > what
> > > > > we
> > > > > > > >>> > assigned
> > > > > > > >>> > > > when
> > > > > > > >>> > > > > > we
> > > > > > > >>> > > > > > > > >> reload
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> complete,
> > we
> > > > > > > >>> transition
> > > > > > > >>> > this
> > > > > > > >>> > > > ID
> > > > > > > >>> > > > > to
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> main
> > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> previous
> > > > > producer
> > > > > > > ID
> > > > > > > >>> > field
> > > > > > > >>> > > > > filled
> > > > > > > >>> > > > > > > in.
> > > > > > > >>> > > > > > > > >> This
> > > > > > > >>> > > > > > > > >> > > is
> > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case
> > the
> > > > > > > operation
> > > > > > > >>> > > > completed
> > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The
> > > > downgrade
> > > > > > > >>> > > compatibility
> > > > > > > >>> > > > I
> > > > > > > >>> > > > > > > > mention
> > > > > > > >>> > > > > > > > >> is
> > > > > > > >>> > > > > > > > >> > > that
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and
> > > epoch
> > > > in
> > > > > > the
> > > > > > > >>> main
> > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > >>> > > > > > > > >> > fields
> > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the
> server
> > > > side.
> > > > > > If
> > > > > > > >>> the
> > > > > > > >>> > > server
> > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses
> > both
> > > > the
> > > > > > > >>> prepare
> > > > > > > >>> > and
> > > > > > > >>> > > > > > complete
> > > > > > > >>> > > > > > > > >> state
> > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > >>> 10:21 AM
> > > > > > > >>> > Jun
> > > > > > > >>> > > > Rao
> > > > > > > >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the
> > > > reply.
> > > > > > > Sorry
> > > > > > > >>> for
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > delay. I
> > > > > > > >>> > > > > > > > >> > have a
> > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think
> the
> > > > > > > motivation
> > > > > > > >>> > > section
> > > > > > > >>> > > > > > could
> > > > > > > >>> > > > > > > be
> > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations
> > listed
> > > > by
> > > > > > the
> > > > > > > >>> KIP is
> > > > > > > >>> > > > "This
> > > > > > > >>> > > > > > can
> > > > > > > >>> > > > > > > > >> happen
> > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due
> > to
> > > > > > > networking
> > > > > > > >>> > > issues
> > > > > > > >>> > > > > or a
> > > > > > > >>> > > > > > > > >> network
> > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and
> then
> > > the
> > > > > > > delayed
> > > > > > > >>> > > message
> > > > > > > >>> > > > > > > finally
> > > > > > > >>> > > > > > > > >> > comes
> > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate.
> > > > Without
> > > > > > > >>> KIP-890,
> > > > > > > >>> > > > > > currently,
> > > > > > > >>> > > > > > > > if
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an
> > ongoing
> > > > > > > >>> transaction,
> > > > > > > >>> > it
> > > > > > > >>> > > > > > already
> > > > > > > >>> > > > > > > > >> bumps
> > > > > > > >>> > > > > > > > >> > up
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents
> > the
> > > > > > delayed
> > > > > > > >>> > produce
> > > > > > > >>> > > > > > message
> > > > > > > >>> > > > > > > > >> from
> > > > > > > >>> > > > > > > > >> > > being
> > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition.
> What
> > > can
> > > > > > cause
> > > > > > > a
> > > > > > > >>> > > hanging
> > > > > > > >>> > > > > > > > >> transaction
> > > > > > > >>> > > > > > > > >> > is
> > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes
> > (either
> > > > > aborts
> > > > > > > or
> > > > > > > >>> > > > commits) a
> > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack
> > on
> > > > > > messages
> > > > > > > >>> > > published
> > > > > > > >>> > > > > in
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> same
> > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for
> the
> > > > > delayed
> > > > > > > >>> message
> > > > > > > >>> > > to
> > > > > > > >>> > > > be
> > > > > > > >>> > > > > > > > >> appended
> > > > > > > >>> > > > > > > > >> > to
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> causing
> > a
> > > > > > > >>> transaction to
> > > > > > > >>> > > > hang.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar
> issue
> > > (not
> > > > > > > >>> mentioned
> > > > > > > >>> > in
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> > > > > > > coordinator's
> > > > > > > >>> log.
> > > > > > > >>> > > For
> > > > > > > >>> > > > > > > > example,
> > > > > > > >>> > > > > > > > >> > it's
> > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest
> to
> > > be
> > > > > > > delayed
> > > > > > > >>> on
> > > > > > > >>> > the
> > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > >>> > > > > > > > >> > By
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest
> is
> > > > > > > processed,
> > > > > > > >>> it's
> > > > > > > >>> > > > > > possible
> > > > > > > >>> > > > > > > > that
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed and
> a
> > > new
> > > > > txn
> > > > > > > has
> > > > > > > >>> > > started.
> > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > >>> > > > > > > > >> > > since
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on
> every
> > > txn,
> > > > > the
> > > > > > > >>> delayed
> > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > >>> > > > > > > > >> > will
> > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker
> > > (and
> > > > > > > >>> eventually a
> > > > > > > >>> > > > > > complete
> > > > > > > >>> > > > > > > > >> marker)
> > > > > > > >>> > > > > > > > >> > > to
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause
> the
> > > > > > > transaction
> > > > > > > >>> to
> > > > > > > >>> > > hang,
> > > > > > > >>> > > > > but
> > > > > > > >>> > > > > > > it
> > > > > > > >>> > > > > > > > >> will
> > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal
> in
> > > this
> > > > > KIP
> > > > > > > >>> will
> > > > > > > >>> > > > address
> > > > > > > >>> > > > > > this
> > > > > > > >>> > > > > > > > >> issue
> > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> "However, I
> > > was
> > > > > > > >>> writing it
> > > > > > > >>> > so
> > > > > > > >>> > > > > that
> > > > > > > >>> > > > > > we
> > > > > > > >>> > > > > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients
> > where
> > > we
> > > > > > don't
> > > > > > > >>> have
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > ability
> > > > > > > >>> > > > > > > > do
> > > > > > > >>> > > > > > > > >> > this
> > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that
> > can.
> > > > (Old
> > > > > > > >>> clients
> > > > > > > >>> > > don't
> > > > > > > >>> > > > > > bump
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure
> the
> > > > write
> > > > > > > >>> belongs
> > > > > > > >>> > to
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > given
> > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> > > wondering
> > > > > why
> > > > > > > we
> > > > > > > >>> need
> > > > > > > >>> > > to
> > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by the
> > old
> > > > and
> > > > > > the
> > > > > > > >>> new
> > > > > > > >>> > > > client.
> > > > > > > >>> > > > > > > Could
> > > > > > > >>> > > > > > > > >> you
> > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently if
> > we
> > > > know
> > > > > > the
> > > > > > > >>> > marker
> > > > > > > >>> > > is
> > > > > > > >>> > > > > > > written
> > > > > > > >>> > > > > > > > >> by
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do
> > > need
> > > > a
> > > > > > way
> > > > > > > to
> > > > > > > >>> > > > > distinguish
> > > > > > > >>> > > > > > > > >> whether
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and
> the
> > > new
> > > > > > > client.
> > > > > > > >>> > Would
> > > > > > > >>> > > it
> > > > > > > >>> > > > > be
> > > > > > > >>> > > > > > > > >> simpler
> > > > > > > >>> > > > > > > > >> > to
> > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field instead
> of
> > > > > > > indirectly
> > > > > > > >>> > > through
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> previous
> > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not
> > > clear
> > > > > to
> > > > > > me
> > > > > > > >>> why
> > > > > > > >>> > we
> > > > > > > >>> > > > only
> > > > > > > >>> > > > > > add
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete
> > > marker,
> > > > > but
> > > > > > > >>> not in
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > prepare
> > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a
> marker
> > > is
> > > > > > > written
> > > > > > > >>> by
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > client
> > > > > > > >>> > > > > > > > >> or
> > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > > > consistently
> > > > > > > for
> > > > > > > >>> all
> > > > > > > >>> > > > > markers.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What
> about
> > > the
> > > > > > > >>> > > > > TransactionLogValue
> > > > > > > >>> > > > > > > > >> record
> > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should
> we
> > > > also
> > > > > > > >>> > distinguish
> > > > > > > >>> > > > > > whether
> > > > > > > >>> > > > > > > > it's
> > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the
> > > overflow
> > > > > > case,
> > > > > > > >>> it's
> > > > > > > >>> > > > still
> > > > > > > >>> > > > > > not
> > > > > > > >>> > > > > > > > >> clear
> > > > > > > >>> > > > > > > > >> > to
> > > > > > > >>> > > > > > > > >> > > me
> > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> produce
> > > Id
> > > > in
> > > > > > the
> > > > > > > >>> > prepare
> > > > > > > >>> > > > > > marker
> > > > > > > >>> > > > > > > > >> while
> > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> complete
> > > > > marker.
> > > > > > > You
> > > > > > > >>> > > > mentioned
> > > > > > > >>> > > > > > that
> > > > > > > >>> > > > > > > > >> it's
> > > > > > > >>> > > > > > > > >> > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > downgrade
> > > > > with
> > > > > > > >>> either
> > > > > > > >>> > the
> > > > > > > >>> > > > > > prepare
> > > > > > > >>> > > > > > > > >> marker
> > > > > > > >>> > > > > > > > >> > > or
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either
> case,
> > > the
> > > > > > > >>> downgraded
> > > > > > > >>> > > > > > coordinator
> > > > > > > >>> > > > > > > > >> should
> > > > > > > >>> > > > > > > > >> > > see
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the
> > > > previous
> > > > > > > >>> produce
> > > > > > > >>> > > Id),
> > > > > > > >>> > > > > > right?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec
> 20,
> > > 2023
> > > > > at
> > > > > > > >>> 6:00 PM
> > > > > > > >>> > > > > Justine
> > > > > > > >>> > > > > > > > Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for
> > > taking
> > > > a
> > > > > > look
> > > > > > > >>> at
> > > > > > > >>> > the
> > > > > > > >>> > > > KIP
> > > > > > > >>> > > > > > > again.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the
> > > epoch
> > > > > > > >>> overflow
> > > > > > > >>> > > case,
> > > > > > > >>> > > > > only
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> > marker
> > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the
> > > behavior
> > > > > of
> > > > > > > the
> > > > > > > >>> rest
> > > > > > > >>> > > of
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> markers
> > > > > > > >>> > > > > > > > >> > > where
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the
> > > > > > transaction
> > > > > > > >>> > > records +
> > > > > > > >>> > > > > 1.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are
> > > > correct
> > > > > > > that
> > > > > > > >>> we
> > > > > > > >>> > > don't
> > > > > > > >>> > > > > > need
> > > > > > > >>> > > > > > > to
> > > > > > > >>> > > > > > > > >> > write
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same.
> > > > > However,
> > > > > > I
> > > > > > > >>> was
> > > > > > > >>> > > > writing
> > > > > > > >>> > > > > it
> > > > > > > >>> > > > > > > so
> > > > > > > >>> > > > > > > > >> that
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients
> > > where
> > > > we
> > > > > > > don't
> > > > > > > >>> > have
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > ability
> > > > > > > >>> > > > > > > > >> do
> > > > > > > >>> > > > > > > > >> > > this
> > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that
> > > can.
> > > > > (Old
> > > > > > > >>> clients
> > > > > > > >>> > > > don't
> > > > > > > >>> > > > > > > bump
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure
> > the
> > > > > write
> > > > > > > >>> belongs
> > > > > > > >>> > > to
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > given
> > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> request
> > > > from
> > > > > a
> > > > > > > new
> > > > > > > >>> > > client,
> > > > > > > >>> > > > we
> > > > > > > >>> > > > > > > will
> > > > > > > >>> > > > > > > > >> fill
> > > > > > > >>> > > > > > > > >> > > this
> > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any
> > EndTxn
> > > > > > requests
> > > > > > > >>> with
> > > > > > > >>> > > the
> > > > > > > >>> > > > > same
> > > > > > > >>> > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > are
> > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > > > transaction.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In
> > prepare
> > > > > phase,
> > > > > > > we
> > > > > > > >>> have
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > same
> > > > > > > >>> > > > > > > > >> > producer
> > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is
> the
> > > > > > producer
> > > > > > > >>> ID and
> > > > > > > >>> > > > epoch
> > > > > > > >>> > > > > > > that
> > > > > > > >>> > > > > > > > >> are
> > > > > > > >>> > > > > > > > >> > on
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we
> stay
> > > the
> > > > > > same
> > > > > > > >>> unless
> > > > > > > >>> > > it
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the
> > producer
> > > > ID
> > > > > to
> > > > > > > >>> the new
> > > > > > > >>> > > one
> > > > > > > >>> > > > > we
> > > > > > > >>> > > > > > > > >> generated
> > > > > > > >>> > > > > > > > >> > > and
> > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete.
> This
> > > is
> > > > > for
> > > > > > > >>> > downgrade
> > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > >>> > > > > > > > >> > > The
> > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> guards
> > > for
> > > > > > > retries
> > > > > > > >>> and
> > > > > > > >>> > > > > > failovers.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare
> > phase
> > > > for
> > > > > > > epoch
> > > > > > > >>> > > > overflow
> > > > > > > >>> > > > > > case
> > > > > > > >>> > > > > > > > >> only
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is
> > for
> > > > the
> > > > > > case
> > > > > > > >>> where
> > > > > > > >>> > > we
> > > > > > > >>> > > > > > reload
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare
> state.
> > > > Once
> > > > > > the
> > > > > > > >>> > > > transaction
> > > > > > > >>> > > > > is
> > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> client
> > > > > already
> > > > > > is
> > > > > > > >>> > using.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit
> > phase,
> > > > we
> > > > > > > store
> > > > > > > >>> the
> > > > > > > >>> > > > > previous
> > > > > > > >>> > > > > > > > >> producer
> > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it
> is
> > > > easier
> > > > > > to
> > > > > > > >>> think
> > > > > > > >>> > of
> > > > > > > >>> > > > it
> > > > > > > >>> > > > > as
> > > > > > > >>> > > > > > > > just
> > > > > > > >>> > > > > > > > >> how
> > > > > > > >>> > > > > > > > >> > > we
> > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch
> > > before,
> > > > > with
> > > > > > > >>> some
> > > > > > > >>> > > extra
> > > > > > > >>> > > > > > > > bookeeping
> > > > > > > >>> > > > > > > > >> > and
> > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> fields.
> > > We
> > > > > have
> > > > > > > to
> > > > > > > >>> do
> > > > > > > >>> > it
> > > > > > > >>> > > > this
> > > > > > > >>> > > > > > way
> > > > > > > >>> > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next
> > > producer
> > > > > ID
> > > > > > is
> > > > > > > >>> for
> > > > > > > >>> > > > prepare
> > > > > > > >>> > > > > > > > status
> > > > > > > >>> > > > > > > > >> and
> > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > complete.
> > > > The
> > > > > > > >>> reason why
> > > > > > > >>> > > we
> > > > > > > >>> > > > > need
> > > > > > > >>> > > > > > > two
> > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > > > > > compatibility.
> > > > > > > >>> We
> > > > > > > >>> > need
> > > > > > > >>> > > > to
> > > > > > > >>> > > > > > keep
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > same
> > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > field
> > > > in
> > > > > > case
> > > > > > > >>> we
> > > > > > > >>> > > > > downgrade.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set
> > the
> > > > > fields
> > > > > > > as
> > > > > > > >>> we
> > > > > > > >>> > do
> > > > > > > >>> > > in
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> > > > > > > compatibility
> > > > > > > >>> --
> > > > > > > >>> > if
> > > > > > > >>> > > we
> > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > > fields)
> > > > > It
> > > > > > > >>> will be
> > > > > > > >>> > > the
> > > > > > > >>> > > > > old
> > > > > > > >>> > > > > > > > >> producer
> > > > > > > >>> > > > > > > > >> > ID
> > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this
> > helps.
> > > > Let
> > > > > > me
> > > > > > > >>> know
> > > > > > > >>> > if
> > > > > > > >>> > > > you
> > > > > > > >>> > > > > > have
> > > > > > > >>> > > > > > > > >> > further
> > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec
> > 20,
> > > > 2023
> > > > > > at
> > > > > > > >>> > 3:33 PM
> > > > > > > >>> > > > Jun
> > > > > > > >>> > > > > > Rao
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems
> > that
> > > > you
> > > > > > have
> > > > > > > >>> made
> > > > > > > >>> > > some
> > > > > > > >>> > > > > > > changes
> > > > > > > >>> > > > > > > > >> to
> > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> particular,
> > we
> > > > are
> > > > > > > >>> changing
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > format
> > > > > > > >>> > > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments
> > > related
> > > > > to
> > > > > > > >>> that.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just
> to
> > > be
> > > > > > clear.
> > > > > > > >>> The
> > > > > > > >>> > > > > overflow
> > > > > > > >>> > > > > > > case
> > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated)
> > is
> > > > when
> > > > > > the
> > > > > > > >>> > current
> > > > > > > >>> > > > > epoch
> > > > > > > >>> > > > > > > > >> equals
> > > > > > > >>> > > > > > > > >> > to
> > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For
> the
> > > > "not
> > > > > > > epoch
> > > > > > > >>> > > > overflow"
> > > > > > > >>> > > > > > > case,
> > > > > > > >>> > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > write
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged
> field
> > > in
> > > > > the
> > > > > > > >>> complete
> > > > > > > >>> > > > > phase.
> > > > > > > >>> > > > > > Do
> > > > > > > >>> > > > > > > > we
> > > > > > > >>> > > > > > > > >> > need
> > > > > > > >>> > > > > > > > >> > > to
> > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't
> > change
> > > > in
> > > > > > this
> > > > > > > >>> case?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It
> > seems
> > > > that
> > > > > > the
> > > > > > > >>> > meaning
> > > > > > > >>> > > > for
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > TransactionLogValue
> > > > > > > >>> changes
> > > > > > > >>> > > > > > depending
> > > > > > > >>> > > > > > > on
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > TransactionStatus
> > > > > > > is
> > > > > > > >>> > > > ongoing,
> > > > > > > >>> > > > > > they
> > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> current
> > > > > > > >>> ProducerEpoch.
> > > > > > > >>> > > When
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > >>> > > they
> > > > > > > >>> > > > > > > > represent
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > ProducerEpoch.
> > > > > > > >>> When the
> > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > >>> > > > > > > > >> > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further
> > depend
> > > > on
> > > > > > > >>> whether
> > > > > > > >>> > the
> > > > > > > >>> > > > > epoch
> > > > > > > >>> > > > > > > > >> overflows
> > > > > > > >>> > > > > > > > >> > > or
> > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow,
> > they
> > > > > > > represent
> > > > > > > >>> > the
> > > > > > > >>> > > > > > current
> > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> ProducerEpoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > > > Otherwise,
> > > > > > they
> > > > > > > >>> > > represent
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > newly
> > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > ProducerEpoch
> > > of
> > > > > 0.
> > > > > > Is
> > > > > > > >>> that
> > > > > > > >>> > > > right?
> > > > > > > >>> > > > > > > This
> > > > > > > >>> > > > > > > > >> seems
> > > > > > > >>> > > > > > > > >> > > not
> > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide
> > > some
> > > > > > > examples
> > > > > > > >>> > like
> > > > > > > >>> > > > what
> > > > > > > >>> > > > > > > Artem
> > > > > > > >>> > > > > > > > >> has
> > > > > > > >>> > > > > > > > >> > > done
> > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> considered a
> > > > > simpler
> > > > > > > >>> design
> > > > > > > >>> > > > where
> > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same
> > value
> > > > > (e.g.
> > > > > > > >>> for the
> > > > > > > >>> > > > > current
> > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > TransactionStatus
> > > > > > and
> > > > > > > >>> epoch
> > > > > > > >>> > > > > > overflow?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's
> > not
> > > > > clear
> > > > > > to
> > > > > > > >>> me
> > > > > > > >>> > why
> > > > > > > >>> > > we
> > > > > > > >>> > > > > > need
> > > > > > > >>> > > > > > > 3
> > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > PrevProducerId,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > NextProducerId.
> > > > > > Could
> > > > > > > we
> > > > > > > >>> > just
> > > > > > > >>> > > > have
> > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > >>> > > > > > > > >> > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > >>> > > > > if
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the
> > > > > > producerId
> > > > > > > >>> and
> > > > > > > >>> > the
> > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > >
> > > > > > > >>> > > >
> > > > > > > >>> > >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>