[VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi All,

We have been discussing in the below thread and final changes have been made to the KIP wiki based on these discussions.

We would now like to put to the vote the following KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+Add+Compaction+Tombstone+Flag

This kip is for having a distinct compaction attribute “tombstone” flag instead of relying on null value, allowing non-null value delete messages.

Many thanks,
Michael



On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com> wrote:

    Hi Mayuresh,
    
    LGTM. Ive just made one small adjustment updating the wire protocol to show the magic byte bump.
    
    Do we think we’re good to put to a vote? Is there any other bits needing discussion?
    
    Cheers
    Mike
    
    On 21/11/2016, 18:26, "Mayuresh Gharat" <gh...@gmail.com> wrote:
    
        Hi Michael,
    
        I have updated the migration section of the KIP. Can you please take a look?
    
        Thanks,
    
        Mayuresh
    
        On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com
        > wrote:
    
        > Hi Michael,
        >
        > That whilst sending tombstone and non null value, the consumer can expect
        > only to receive the non-null message only in step (3) is this correct?
        > ---> I do agree with you here.
        >
        > Becket, Ismael : can you guys review the migration plan listed above using
        > magic byte?
        >
        > Thanks,
        >
        > Mayuresh
        >
        > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <Mi...@ig.com>
        > wrote:
        >
        >> Many thanks for this Mayuresh. I don't have any objections.
        >>
        >> I assume we should state:
        >>
        >> That whilst sending tombstone and non null value, the consumer can expect
        >> only to receive the non-null message only in step (3) is this correct?
        >>
        >> Cheers
        >> Mike
        >>
        >>
        >>
        >> Sent using OWA for iPhone
        >> ________________________________________
        >> From: Mayuresh Gharat <gh...@gmail.com>
        >> Sent: Thursday, November 17, 2016 5:18:41 PM
        >> To: dev@kafka.apache.org
        >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
        >>
        >> Hi Ismael,
        >>
        >> Thanks for the explanation.
        >> Specially I like this part where in you mentioned we can get rid of the
        >> older null value support for log compaction later on, here :
        >> We can't change semantics of the message format without having a long
        >> transition period. And we can't rely
        >> on people reading documentation or acting on a warning for something so
        >> fundamental. As such, my take is that we need to bump the magic byte. The
        >> good news is
        >> that we don't have to support all versions forever. We have said that we
        >> will support direct upgrades for 2 years. That means that message format
        >> version n could, in theory, be removed 2 years after the it's introduced.
        >>
        >> Just a heads up, I would like to mention that even without bumping magic
        >> byte, we will *NOT* loose zero copy as in the client(x+1) in my
        >> explanation
        >> above will convert internally a null value to have a tombstone bit set and
        >> a tombstone bit set to have a null value automatically internally and by
        >> the time we move to version (x+2), the clients would have upgraded.
        >> Obviously if we support a request from consumer(x), we will loose zero
        >> copy
        >> but that is the same case with magic byte.
        >>
        >> But if magic byte bump makes life easier for transition for the above
        >> reasons that you explained, I am OK with it since we are going to meet the
        >> end goal down the road :)
        >>
        >> On a side note can we update the doc here on magic byte to say that "*it
        >> should be bumped whenever the message format is changed or the
        >> interpretation of message format (usage of the reserved bits as well) is
        >> changed*".
        >>
        >>
        >> Hi Michael,
        >>
        >> Here is the update plan that we discussed offline yesterday :
        >>
        >> Currently the magic-byte which corresponds to the "message.format.version"
        >> is set to 1.
        >>
        >> 1) On broker it will be set to 1 initially.
        >>
        >> 2) When a producer client sends a message with magic-byte = 2, since the
        >> broker is on magic-byte = 1, we will down convert it, which means if the
        >> tombstone bit is set, the value will be set to null. A consumer
        >> understanding magic-byte = 1, will still work with this. A consumer
        >> working
        >> with magic-byte =2 will also be able to understand this, since it
        >> understands the tombstone.
        >> Now there is still the question of supporting a non-tombstone and null
        >> value from producer client with magic-byte = 2.* (I am not sure if we
        >> should support this. Ismael/Becket can comment here)*
        >>
        >> 3) When almost all the clients have upgraded, the message.format.version
        >> on
        >> the broker can be changed to 2, where in the down conversion in the above
        >> step will not happen. If at this point we get a consumer request from a
        >> older consumer, we might have to down convert where in we loose zero copy,
        >> but these cases should be rare.
        >>
        >> Becket can you review this plan and add more details if I have
        >> missed/wronged something, before we put it on KIP.
        >>
        >> Thanks,
        >>
        >> Mayuresh
        >>
        >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <Mi...@ig.com>
        >> wrote:
        >>
        >> > Thanks guys, for discussing this offline and getting some consensus.
        >> >
        >> > So its clear for myself and others what is proposed now (i think i
        >> > understand, but want to make sure)
        >> >
        >> > Could i ask either directly update the kip to detail the migration
        >> > strategy, or (re-)state your offline discussed and agreed migration
        >> > strategy based on a magic byte is in this thread.
        >> >
        >> >
        >> > The main original driver for the KIP was to support compaction where
        >> value
        >> > isn't null, based off the discussions on KIP-82 thread.
        >> >
        >> > We should be able to support non-tombstone + null value by the
        >> completion
        >> > of the KIP, as we noted when discussing this kip, having logic based on
        >> a
        >> > null value isn't very clean and also separates the concerns.
        >> >
        >> > As discussed already though we can split this into KIP-87a and KIP-87b
        >> >
        >> > Where we look to deliver KIP-87a on a compacted topic (to address the
        >> > immediate issues)
        >> > * tombstone + null value
        >> > * tombstone + non-null value
        >> > * non-tombstone + non-null value
        >> >
        >> > Then we can discuss once KIP-87a is completed options later and how we
        >> > support the second part KIP-87b to deliver:
        >> > * non-tombstone + null value
        >> >
        >> > Cheers
        >> > Mike
        >> >
        >> >
        >> >
        >> > ________________________________________
        >> > From: Becket Qin <be...@gmail.com>
        >> > Sent: Thursday, November 17, 2016 1:43 AM
        >> > To: dev@kafka.apache.org
        >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
        >> >
        >> > Renu, Mayuresh and I had an offline discussion, and following is a brief
        >> > summary.
        >> >
        >> > 1. We agreed that not bumping up magic value may result in losing zero
        >> copy
        >> > during migration.
        >> > 2. Given that bumping up magic value is almost free and has benefit of
        >> > avoiding potential performance issue. It is probably worth doing.
        >> >
        >> > One issue we still need to think about is whether we want to support a
        >> > non-tombstone message with null value.
        >> > Currently it is not supported by Kafka. If we allow a non-tombstone null
        >> > value message to exist after KIP-87. The problem is that such message
        >> will
        >> > not be supported by the consumers prior to KIP-87. Because a null value
        >> > will always be interpreted to a tombstone.
        >> >
        >> > One option is that we keep the current way, i.e. do not support such
        >> > message. It would be good to know if there is a concrete use case for
        >> such
        >> > message. If there is not, we can probably just not support it.
        >> >
        >> > Thanks,
        >> >
        >> > JIangjie (Becket) Qin
        >> >
        >> >
        >> >
        >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
        >> > gharatmayuresh15@gmail.com
        >> > > wrote:
        >> >
        >> > > Hi Ismael,
        >> > >
        >> > > This is something I can think of for migration plan:
        >> > > So the migration plan can look something like this, with up
        >> conversion :
        >> > >
        >> > > 1) Currently lets say we have Broker at version x.
        >> > > 2) Currently we have clients at version x.
        >> > > 3) a) We move the version to Broker(x+1) : supports both tombstone and
        >> > null
        >> > > for log compaction.
        >> > >     b) We upgrade the client to version client(x+1) : if in the
        >> producer
        >> > > client(x+1) the value is set to null, we will automatically set the
        >> > > Tombstone bit internally. If the producer client(x+1) sets the
        >> tombstone
        >> > > itself, well and good. For producer client(x), the broker will up
        >> convert
        >> > > to have the tombstone bit. Broker(x+1) is supporting both. Consumer
        >> > > client(x+1) will be aware of this and should be able to handle this.
        >> For
        >> > > consumer client(x) we will down convert the message on the broker
        >> side.
        >> > >     c) At this point we will have to specify a warning or clearly
        >> specify
        >> > > in docs that this behavior is about to be changed for log compaction.
        >> > > 4) a) In next release of the Broker(x+2), we say that only Tombstone
        >> is
        >> > > used for log compaction on the Broker side. Clients(x+1) still is
        >> > > supported.
        >> > >     b) We upgrade the client to version client(x+2) : if value is set
        >> to
        >> > > null, tombstone will not be set automatically. The client will have to
        >> > call
        >> > > setTombstone() to actually set the tombstone.
        >> > >
        >> > > We should compare this migration plan with the migration plan for
        >> magic
        >> > > byte bump and do whatever looks good.
        >> > > I am just worried that if we go down magic byte route, unless I am
        >> > missing
        >> > > something, it sounds like kafka will be stuck with supporting both
        >> null
        >> > > value and tombstone bit for log compaction for life long, which does
        >> not
        >> > > look like a good end state.
        >> > >
        >> > > Thanks,
        >> > >
        >> > > Mayuresh
        >> > >
        >> > >
        >> > >
        >> > >
        >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
        >> > > gharatmayuresh15@gmail.com
        >> > > > wrote:
        >> > >
        >> > > > Hi Ismael,
        >> > > >
        >> > > > That's a very good point which I might have not considered earlier.
        >> > > >
        >> > > > Here is a plan that I can think of:
        >> > > >
        >> > > > Stage 1) The broker from now on, up converts the message to have the
        >> > > > tombstone marker. The log compaction thread does log compaction
        >> based
        >> > on
        >> > > > both null and tombstone marker. This is our transition period.
        >> > > > Stage 2) The next release we only say that log compaction is based
        >> on
        >> > > > tombstone marker. (Open source kafka makes this as a policy). By
        >> this
        >> > > time,
        >> > > > the organization which is moving to this release will be sure that
        >> they
        >> > > > have gone through the entire transition period.
        >> > > >
        >> > > > My only goal of doing this is that Kafka clearly specifies the end
        >> > state
        >> > > > about what log compaction means (is it null value or a tombstone
        >> > marker,
        >> > > > but not both).
        >> > > >
        >> > > > What do you think?
        >> > > >
        >> > > > Thanks,
        >> > > >
        >> > > > Mayuresh
        >> > > > .
        >> > > >
        >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <is...@juma.me.uk>
        >> > wrote:
        >> > > >
        >> > > >> One comment below.
        >> > > >>
        >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
        >> > > >> gharatmayuresh15@gmail.com
        >> > > >> > wrote:
        >> > > >>
        >> > > >> >    - If we don't bump up the magic byte, on the broker side, the
        >> > > broker
        >> > > >> >    will always have to look at both tombstone bit and the value
        >> when
        >> > > do
        >> > > >> the
        >> > > >> >    compaction. Assuming we do not bump up the magic byte,
        >> > > >> >    imagine the broker sees a message which does not have a
        >> tombstone
        >> > > bit
        >> > > >> >    set. The broker does not know when the message was produced
        >> (i.e.
        >> > > >> > whether
        >> > > >> >    the message has been up converted or not), it has to take a
        >> > further
        >> > > >> > look at
        >> > > >> >    the value to see if it is null or not in order to determine
        >> if it
        >> > > is
        >> > > >> a
        >> > > >> >    tombstone. The same logic has to be put on the consumer as
        >> well
        >> > > >> because
        >> > > >> > the
        >> > > >> >    consumer does not know if the message has been up converted or
        >> > not.
        >> > > >> >       - If we upconvert while appending, this is not the case,
        >> > right?
        >> > > >>
        >> > > >>
        >> > > >> If I understand you correctly, this is not sufficient because the
        >> log
        >> > > may
        >> > > >> have messages appended before it was upgraded to include KIP-87.
        >> > > >>
        >> > > >> Ismael
        >> > > >>
        >> > > >
        >> > > >
        >> > > >
        >> > > > --
        >> > > > -Regards,
        >> > > > Mayuresh R. Gharat
        >> > > > (862) 250-7125
        >> > > >
        >> > >
        >> > >
        >> > >
        >> > > --
        >> > > -Regards,
        >> > > Mayuresh R. Gharat
        >> > > (862) 250-7125
        >> > >
        >> > The information contained in this email is strictly confidential and for
        >> > the use of the addressee only, unless otherwise indicated. If you are
        >> not
        >> > the intended recipient, please do not read, copy, use or disclose to
        >> others
        >> > this message or any attachment. Please also notify the sender by
        >> replying
        >> > to this email or by telephone (+44(020 7896 0011) and then delete the
        >> email
        >> > and any copies of it. Opinions, conclusion (etc) that do not relate to
        >> the
        >> > official business of this company shall be understood as neither given
        >> nor
        >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
        >> > registered in England and Wales, company number 04008957) and IG Index
        >> > Limited (a company registered in England and Wales, company number
        >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
        >> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
        >> > Index Limited (register number 114059) are authorised and regulated by
        >> the
        >> > Financial Conduct Authority.
        >> >
        >>
        >>
        >>
        >> --
        >> -Regards,
        >> Mayuresh R. Gharat
        >> (862) 250-7125
        >> The information contained in this email is strictly confidential and for
        >> the use of the addressee only, unless otherwise indicated. If you are not
        >> the intended recipient, please do not read, copy, use or disclose to others
        >> this message or any attachment. Please also notify the sender by replying
        >> to this email or by telephone (+44(020 7896 0011) and then delete the email
        >> and any copies of it. Opinions, conclusion (etc) that do not relate to the
        >> official business of this company shall be understood as neither given nor
        >> endorsed by it. IG is a trading name of IG Markets Limited (a company
        >> registered in England and Wales, company number 04008957) and IG Index
        >> Limited (a company registered in England and Wales, company number
        >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
        >> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
        >> Index Limited (register number 114059) are authorised and regulated by the
        >> Financial Conduct Authority.
        >>
        >
        >
        >
        > --
        > -Regards,
        > Mayuresh R. Gharat
        > (862) 250-7125
        >
    
    
    
        --
        -Regards,
        Mayuresh R. Gharat
        (862) 250-7125
    
    
    The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
    

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi All,

Based on KIP-82 result.

I am no longer actively going to push further on this.
Based on this, if no one actively wants to take the helm on this, I will move KIP-87 to the Dormant/Inactive KIP section for now, I will do this next week, if no one steps in.

Cheers
Mike

On 05/01/2017, 20:29, "Michael Pearce" <Mi...@ig.com> wrote:

    @Jay.

    On KIP-82 are you and Jun now on board that Headers are the better of the two options/kips?

    If so could you reply on that thread this is the case.

    We really would like to get KIP-82 going/moving on an active discussion around the last few implementation details, rather than where we’re at that seems were held up on getting consensus that the concept of KIP-82 is worth it.

    If we don’t push on this one.

    Either way we really need a solution of 82 or 87 Apr/May time release (or when ever the next one is after the imminent on in Feb)

    Cheers
    Mike

    On 20/12/2016, 05:55, "Michael Pearce" <Mi...@ig.com> wrote:

        Agreed. As I said there is still a use case just I wouldn't be pushing for it, the. Need for it reduces for me.

        Sent using OWA for iPhone
        ________________________________________
        From: radai <ra...@gmail.com>
        Sent: Tuesday, December 20, 2016 3:51:33 AM
        To: dev@kafka.apache.org
        Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

        i didnt mean to sound as insisting. what i actually mean is it would still
        be a valid issue but of much less concern.

        On Mon, Dec 19, 2016 at 7:50 PM, radai <ra...@gmail.com> wrote:

        > this kip fixes a "bug" (quirk?) that arises when people implement headers
        > "in V" (in the payload part of a message).
        > if you have proper headers you obviously dont need to to stick them in V
        > and so wont run into this, but its still a valid issue.
        >
        > On Mon, Dec 19, 2016 at 3:06 PM, Jay Kreps <ja...@confluent.io> wrote:
        >
        >> Makes sense!
        >>
        >> -Jay
        >>
        >> On Mon, Dec 19, 2016 at 2:40 PM, Michael Pearce <Mi...@ig.com>
        >> wrote:
        >>
        >> > Wow just read that def over tired. Hopefully it makes sense. Or you get
        >> > the gist at least.
        >> >
        >> > ________________________________________
        >> > From: Michael Pearce <Mi...@ig.com>
        >> > Sent: Monday, December 19, 2016 9:19:02 PM
        >> > To: dev@kafka.apache.org
        >> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
        >> >
        >> > Hi Jay,
        >> >
        >> > Agreed this stemmed as offshoot from KIP-82.
        >> >
        >> > Which our main driver for was to be able to have some headers for a null
        >> > value as such for our routing, audit, tracing and a few other bits which
        >> > currently we are forced to do with a message wrapper, if we all agreed
        >> on
        >> > KIP-82 that we need native headers and look to implement that the push
        >> for
        >> > this would dissipate.
        >> >
        >> > This KIP would allow for though one use case that comes to mind we could
        >> > see which is to have business data with a delete. Though as said this
        >> isn't
        >> > something we are pushing for think really we would have.
        >> >
        >> > As such in summary yes, if you want to fully support KIP-82 and we can
        >> get
        >> > that agreed in principle and a target release version, I think quite a
        >> few
        >> > guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.
        >> >
        >> > Cheers
        >> > Mike
        >> >
        >> > Sent using OWA for iPhone
        >> > ________________________________________
        >> > From: Jay Kreps <ja...@confluent.io>
        >> > Sent: Monday, December 19, 2016 8:51:23 PM
        >> > To: dev@kafka.apache.org
        >> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
        >> >
        >> > Hey Michael,
        >> >
        >> > Here is the compatibility concern I have:
        >> >
        >> >    1. You have a consumer app that relies on value == null to indicate a
        >> >    delete (current semantics).
        >> >    2. You upgrade Kafka and your clients.
        >> >    3. Some producer starts using the tombstone field in combination with
        >> >    non-null.
        >> >
        >> > I share Ismael's dislike of setting tombstones on records with null
        >> values.
        >> > This makes sense as a transitional state, but as an end state its a bit
        >> > weird. You'd expect to be able to mix null values and tombstones, and
        >> have
        >> > the null values remain and the tombstones get compacted. However what
        >> will
        >> > happen is both will be compacted and upon debugging this you'll learn
        >> that
        >> > we sometimes use null in the value to indicate tombstone. Ismael's
        >> solution
        >> > is a bigger compatibility break, though, so not sure if that is better.
        >> >
        >> > My other question is the relationship to KIP-82. My read is that this
        >> KIP
        >> > solves some but not all of the problems KIP-82 is intended for. KIP-82,
        >> on
        >> > the other hand, seems to address most of the motivating uses for this
        >> KIP.
        >> > The exception is maybe item (5) on the list where you want to
        >> simultaneous
        >> > delete and convey some information to subscribers, but I couldn't
        >> construct
        >> > a concrete examples for that one. Do we need to rationalize these two
        >> KIPs?
        >> > That is, do you still advocate this proposal if we do KIP-82 and vice
        >> > versa? As you may have noticed I'm somewhat emotionally invested in the
        >> > simplicity of the core data model, so my default position is let's try
        >> to
        >> > avoid stuffing more stuff in, but if we have to add stuff I like each of
        >> > these individually more than doing both. :-)
        >> >
        >> > -Jay
        >> >
        >> >
        >> >
        >> >
        >> > On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Michael.Pearce@ig.com
        >> >
        >> > wrote:
        >> >
        >> > > Hi Jay
        >> > >
        >> > > I disagree here that we are breaking any compatibility, we went
        >> through
        >> > > this on the discussion thread in fact with the help of that thread is
        >> how
        >> > > the kip came to the solution.
        >> > >
        >> > > Also on the supported combinations front you mention, we are not
        >> taking
        >> > > anything away afaik.
        >> > >
        >> > > Currently supported are only are:
        >> > > Null value = delete
        >> > > Non-null value = non delete
        >> > >
        >> > > With this kip we would support
        >> > > Null value + tombstone = delete
        >> > > Non null value + tombstone = delete
        >> > > Non null value + no tombstone = non delete
        >> > >
        >> > > As for the alternative idea, this is simply a new policy, how can
        >> there
        >> > be
        >> > > confusion here? For this policy it would be explicit that tombstone
        >> > marker
        >> > > would need to be set for a delete.
        >> > >
        >> > > I'm going to vent a little now as starting to get quite frustrated.
        >> > >
        >> > > We are going round in circles on kip-82 as per use cases there is now
        >> > many
        >> > > use cases, how many more are needed? just because confluent don't see
        >> > these
        >> > > doesn't mean they aren't real use cases other have, this is the point
        >> of
        >> > > the Apache foundation, it shouldn't be the view of just one
        >> organisation.
        >> > > It really is getting a feeling of the NIH syndrome. Rather than it
        >> being
        >> > > constructive on discussion of the implementation detail.
        >> > >
        >> > > kip-87 spawned from as on the kip call we all agreed this was needed.
        >> And
        >> > > would at least allow a custom wrapper be supported in a compacted
        >> topic,
        >> > > allowing meta data. Which again now I feel we are spinning wheels, and
        >> > > simply finding reasons not support it.
        >> > >
        >> > > Cheers
        >> > > Mike
        >> > >
        >> > >
        >> > >
        >> > > Sent using OWA for iPad
        >> > > ________________________________________
        >> > > From: Jay Kreps <ja...@confluent.io>
        >> > > Sent: Friday, December 16, 2016 7:09:23 PM
        >> > > To: dev@kafka.apache.org
        >> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
        >> > >
        >> > > Hey Michael,
        >> > >
        >> > > I do think it might have been better had we started with a separate
        >> > concept
        >> > > of null vs delete. Given that we are where we are, I'm not sure that
        >> the
        >> > > possible cures we explored so far are better than the disease.
        >> > >
        >> > > I apologize for coming to this late, but I didn't really understand
        >> the
        >> > > implications of the KIP and that we'd be breaking compatibility with
        >> > > existing apps until the vote had begun, and in my defense I'm not sure
        >> > the
        >> > > other folks voting did either.
        >> > >
        >> > > I think we all agree there are many existing apps that are built with
        >> the
        >> > > assumption of "null value non-tombstone" and it isn't possible to
        >> > > disambiguate these from tombstones on the producer. It isn't that
        >> anyone
        >> > is
        >> > > saying we have to support all four possibilities at once, it's that we
        >> > > simply can't orphan one of the existing combinations or our users will
        >> > eat
        >> > > us!
        >> > >
        >> > > If I've understood your alternate solution of adding another setting
        >> for
        >> > > compaction, I think this does fix the compatibility problem, but it
        >> adds
        >> > an
        >> > > odd mode the user has to add on all their topics. While the current
        >> state
        >> > > is easily explainable, the resulting state where the meaning of
        >> tombstone
        >> > > and null are overlapping and ambiguous and dependent on a compaction
        >> > > setting that could change out of band or not be in sync with your
        >> code in
        >> > > some environment seems worse to me then where we currently are. I
        >> think
        >> > the
        >> > > question is how would this combination be explained to users and does
        >> it
        >> > > make sense?
        >> > >
        >> > > -Jay
        >> > >
        >> > >
        >> > >
        >> > > On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <
        >> Michael.Pearce@ig.com>
        >> > > wrote:
        >> > >
        >> > > > Hi Chaps,
        >> > > >
        >> > > > Can we either get one more +1 binding (we have 2 already) on the
        >> > > existing?
        >> > > >
        >> > > > Or have some response on the below possible alternative. We are
        >> keen to
        >> > > > get working on this, so we make next feature release.
        >> > > >
        >> > > > Cheers
        >> > > > Mike
        >> > > >
        >> > > >
        >> > > > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com>
        >> wrote:
        >> > > >
        >> > > >     Hi Ismael
        >> > > >
        >> > > >     Did you see our email this morning, what's your thoughts on this
        >> > > > approach to instead we simply have a brand new policy?
        >> > > >
        >> > > >     Cheers
        >> > > >     Mike
        >> > > >
        >> > > >
        >> > > >     Sent using OWA for iPhone
        >> > > >     ________________________________________
        >> > > >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
        >> > > Juma <
        >> > > > ismael@juma.me.uk>
        >> > > >     Sent: Tuesday, December 13, 2016 11:30:05 AM
        >> > > >     To: dev@kafka.apache.org
        >> > > >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
        >> > > >
        >> > > >     Yes, this is actually tricky to do in a way where we both have
        >> the
        >> > > > desired
        >> > > >     semantics and maintain compatibility. When someone creates a
        >> > > >     `ProducerRecord` with a `null` value today, the producer doesn't
        >> > know
        >> > > > if
        >> > > >     it's meant to be a tombstone or not. For V3 messages, it's easy
        >> > when
        >> > > > the
        >> > > >     constructor that takes a tombstone is used. However, if any
        >> other
        >> > > >     constructor is used, it's not clear. A couple of options I can
        >> > think
        >> > > > of,
        >> > > >     none of them particularly nice:
        >> > > >
        >> > > >     1. Have a third state where tombstone = unknown and the broker
        >> > would
        >> > > > set
        >> > > >     the tombstone bit if the value was null and the topic was
        >> > compacted.
        >> > > > People
        >> > > >     that wanted to pass a non-null value for the tombstone would
        >> have
        >> > to
        >> > > > use
        >> > > >     the constructor that takes a tombstone. The drawbacks: third
        >> state
        >> > > for
        >> > > >     tombstone in message format, message conversion at the broker
        >> for a
        >> > > > common
        >> > > >     case.
        >> > > >
        >> > > >     2. Extend MetadataResponse to optionally include topic configs,
        >> > which
        >> > > > would
        >> > > >     make it possible for the producer to be smarter about setting
        >> the
        >> > > >     tombstone. It would only do it if a tombstone was not passed
        >> > > > explicitly,
        >> > > >     the value was null and the topic was compacted. The main
        >> drawback
        >> > is
        >> > > > that
        >> > > >     the producer would be getting a bit more data for each topic
        >> even
        >> > > > though it
        >> > > >     probably won't use it most of the time. Extending
        >> MetadataResponse
        >> > to
        >> > > >     return topic configs would be useful for other reasons as well,
        >> so
        >> > > that
        >> > > >     part seems OK.
        >> > > >
        >> > > >     In addition, for both proposals, we could consider adding
        >> warnings
        >> > to
        >> > > > the
        >> > > >     documentation that the behaviour of the constructors that don't
        >> > take
        >> > > a
        >> > > >     tombstone would change in the next major release so that
        >> tombstone
        >> > =
        >> > > > false.
        >> > > >     Not sure if this would be worth it though.
        >> > > >
        >> > > >     Ismael
        >> > > >
        >> > > >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
        >> > > > ewen@confluent.io>
        >> > > >     wrote:
        >> > > >
        >> > > >     > Michael,
        >> > > >     >
        >> > > >     > It kind of depends on how you want to interpret the tombstone
        >> > flag.
        >> > > > If it's
        >> > > >     > purely a producer-facing Kafka-level thing that we treat as
        >> > > internal
        >> > > > to the
        >> > > >     > broker and log cleaner once the record is sent, then your
        >> > approach
        >> > > > makes
        >> > > >     > sense. You're just moving copying the null-indicates-delete
        >> > > behavior
        >> > > > of the
        >> > > >     > old constructor into the tombstone flag.
        >> > > >     >
        >> > > >     > However, if you want this change to more generally decouple
        >> the
        >> > > idea
        >> > > > of
        >> > > >     > deletion and null values, then you are sometimes converting
        >> what
        >> > > > might be a
        >> > > >     > completely valid null value that doesn't indicate deletion
        >> into a
        >> > > >     > tombstone. Downstream applications could potentially handle
        >> these
        >> > > > cases
        >> > > >     > differently given the separation of deletion from value.
        >> > > >     >
        >> > > >     > I guess the question is if we want to try to support the
        >> latter
        >> > > even
        >> > > > for
        >> > > >     > topics where we have older produce requests. An example where
        >> > this
        >> > > > could
        >> > > >     > come up is in something like a CDC Connector. If we try to
        >> > support
        >> > > > the
        >> > > >     > semantic difference, a connector might write changes to Kafka
        >> > using
        >> > > > the
        >> > > >     > tombstone flag to indicate when a row was truly deleted (vs an
        >> > > > update that
        >> > > >     > sets it to null but still present; this probably makes more
        >> sense
        >> > > > for CDC
        >> > > >     > from document stores or extracting single columns). There are
        >> > > various
        >> > > >     > reasons we might want to maintain the full log and not turn
        >> > > > compaction on
        >> > > >     > (or just use a time-based retention policy), but downstream
        >> > > > applications
        >> > > >     > might care to know the difference between a delete and a null
        >> > > value.
        >> > > > In
        >> > > >     > fact both versions of the same log (compacted and
        >> time-retention)
        >> > > > could be
        >> > > >     > useful and I don't think it'll be uncommon to maintain both or
        >> > use
        >> > > > KIP-71
        >> > > >     > to maintain a hybrid compacted/retention topic.
        >> > > >     >
        >> > > >     > -Ewen
        >> > > >     >
        >> > > >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
        >> > > > Michael.Pearce@ig.com>
        >> > > >     > wrote:
        >> > > >     >
        >> > > >     > > Hi Jay,
        >> > > >     > >
        >> > > >     > > Why wouldn't that work, the tombstone value is only looked
        >> at
        >> > by
        >> > > > the
        >> > > >     > > broker, on a topic configured for compaction as such is
        >> benign
        >> > on
        >> > > > non
        >> > > >     > > compacted topics. This is as much as sending a null value
        >> > > currently
        >> > > >     > >
        >> > > >     > >
        >> > > >     > > Regards
        >> > > >     > > Mike
        >> > > >     > >
        >> > > >     > >
        >> > > >     > >
        >> > > >     > > Sent using OWA for iPhone
        >> > > >     > > ________________________________________
        >> > > >     > > From: Jay Kreps <ja...@confluent.io>
        >> > > >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
        >> > > >     > > To: dev@kafka.apache.org
        >> > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
        >> > > >     > >
        >> > > >     > > Hey Michael,
        >> > > >     > >
        >> > > >     > > I'm not quite sure that works as that would translate ALL
        >> null
        >> > > > values to
        >> > > >     > > tombstones, even for non-compacted topics that use null as
        >> an
        >> > > > acceptable
        >> > > >     > > value sent by the producer and expected by the consumer.
        >> > > >     > >
        >> > > >     > > -Jay
        >> > > >     > >
        >> > > >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
        >> > > > Michael.Pearce@ig.com>
        >> > > >     > > wrote:
        >> > > >     > >
        >> > > >     > > > Hi Ewen,
        >> > > >     > > >
        >> > > >     > > > I think the easiest way to show this is with code.
        >> > > >     > > >
        >> > > >     > > > As you can see we keep the existing behaviour for
        >> > code/binaries
        >> > > > calling
        >> > > >     > > > the pre-existing constructors, whereby if the value is
        >> null
        >> > the
        >> > > >     > tombstone
        >> > > >     > > > is set to true.
        >> > > >     > > >
        >> > > >     > > > Regards
        >> > > >     > > > Mike
        >> > > >     > > >
        >> > > >     > > >
        >> > > >     > > >
        >> > > >     > > >     /**
        >> > > >     > > >      * Creates a record with a specified timestamp to be
        >> sent
        >> > > to
        >> > > > a
        >> > > >     > > > specified topic and partition
        >> > > >     > > >      *
        >> > > >     > > >      * @param topic The topic the record will be appended
        >> to
        >> > > >     > > >      * @param partition The partition to which the record
        >> > > should
        >> > > > be
        >> > > >     > sent
        >> > > >     > > >      * @param timestamp The timestamp of the record
        >> > > >     > > >      * @param tombstone if the record should be treated
        >> as a
        >> > > > tombstone
        >> > > >     > if
        >> > > >     > > > the topic is compacted
        >> > > >     > > >      * @param key The key that will be included in the
        >> record
        >> > > >     > > >      * @param value The record contents
        >> > > >     > > >      */
        >> > > >     > > >     public ProducerRecord(String topic, Integer partition,
        >> > > > Boolean
        >> > > >     > > > tombstone, Long timestamp, K key, V value) {
        >> > > >     > > >         if (topic == null)
        >> > > >     > > >             throw new IllegalArgumentException("Topic
        >> cannot
        >> > > be
        >> > > >     > null.");
        >> > > >     > > >         if (timestamp != null && timestamp < 0)
        >> > > >     > > >             throw new IllegalArgumentException(
        >> > > >     > > >                     String.format("Invalid timestamp: %d.
        >> > > > Timestamp
        >> > > >     > > should
        >> > > >     > > > always be non-negative or null.", timestamp));
        >> > > >     > > >         if (partition != null && partition < 0)
        >> > > >     > > >             throw new IllegalArgumentException(
        >> > > >     > > >                     String.format("Invalid partition: %d.
        >> > > > Partition
        >> > > >     > > number
        >> > > >     > > > should always be non-negative or null.", partition));
        >> > > >     > > >         if (!tombstone && value == null){
        >> > > >     > > >             throw new IllegalArgumentException(
        >> > > >     > > >                     String.format("Tombstone must be true
        >> if
        >> > > null
        >> > > >     > > value"));
        >> > > >     > > >         }
        >> > > >     > > >         this.topic = topic;
        >> > > >     > > >         this.partition = partition;
        >> > > >     > > >         this.tombstone = tombstone;
        >> > > >     > > >         this.key = key;
        >> > > >     > > >         this.value = value;
        >> > > >     > > >         this.timestamp = timestamp;
        >> > > >     > > >     }
        >> > > >     > > >
        >> > > >     > > >     public ProducerRecord(String topic, Integer partition,
        >> > Long
        >> > > >     > > timestamp,
        >> > > >     > > > K key, V value) {
        >> > > >     > > >         this(topic, partition, value == null, timestamp,
        >> key,
        >> > > > value);
        >> > > >     > > >     }
        >> > > >     > > > ________________________________________
        >> > > >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
        >> > > >     > > > Sent: Sunday, December 11, 2016 5:45 AM
        >> > > >     > > > To: dev@kafka.apache.org
        >> > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
        >> > > >     > > >
        >> > > >     > > > It seemed like this was addressed in the migration
        >> section,
        >> > > > wasn't it?
        >> > > >     > > The
        >> > > >     > > > V2 vs V3 requests and the fact that the broker will
        >> downgrade
        >> > > the
        >> > > >     > message
        >> > > >     > > > format (losing zero copy) if you issues a V2 request to a
        >> > > broker
        >> > > > using
        >> > > >     > V3
        >> > > >     > > > format handles compatibility, does it not? The existing
        >> > > > consumers won't
        >> > > >     > > see
        >> > > >     > > > the extra metadata in the value, but they will get a null
        >> > > > instead and
        >> > > >     > > treat
        >> > > >     > > > it as a tombstone. Certainly there is a performance
        >> impact,
        >> > but
        >> > > > it
        >> > > >     > seemed
        >> > > >     > > > compatible.
        >> > > >     > > >
        >> > > >     > > > I'm worried about this though:
        >> > > >     > > >
        >> > > >     > > >
        >> > > >     > > >    - *NOTE *: With the new version of producer client
        >> using
        >> > > >     > > ProduceRequest
        >> > > >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not
        >> > set)
        >> > > > and
        >> > > >     > null
        >> > > >     > > > value
        >> > > >     > > >    should not be allowed as the older version of consumer
        >> > using
        >> > > >     > > > FetchRequest
        >> > > >     > > >    V2 will think of this as a tombstone when its actually
        >> > not.
        >> > > >     > > >
        >> > > >     > > >
        >> > > >     > > > Unless I'm misunderstanding, this ends up breaking binary
        >> > > > compatibility
        >> > > >     > > for
        >> > > >     > > > the Java API. It sounds like this suggests that passing a
        >> > null
        >> > > > value to
        >> > > >     > > the
        >> > > >     > > > existing ProducerRecord constructors would generate an
        >> > > exception
        >> > > > since
        >> > > >     > > you
        >> > > >     > > > didn't explicitly enable the tombstone (via whatever new
        >> > > > constructor is
        >> > > >     > > > provided). But that means you can't swap in a newer client
        >> > jar
        >> > > > without
        >> > > >     > > > recompiling and get the same behavior if your app deletes
        >> > keys
        >> > > > using
        >> > > >     > the
        >> > > >     > > > current approach because your app will start throwing
        >> > > > exceptions. Maybe
        >> > > >     > > > this is a tradeoff we're ok with, but we've tried pretty
        >> hard
        >> > > to
        >> > > > avoid
        >> > > >     > > > breaking compatibility like this so far -- it makes
        >> picking
        >> > up
        >> > > > bug
        >> > > >     > fixes
        >> > > >     > > in
        >> > > >     > > > the clients harder for users of frameworks that have to
        >> pin
        >> > to
        >> > > > earlier
        >> > > >     > > > default versions for compatibility.
        >> > > >     > > >
        >> > > >     > > > But then later the KIP says:
        >> > > >     > > >
        >> > > >     > > >
        >> > > >     > > >    - The old constructors for ProducerRecord without this
        >> > > > parameter
        >> > > >     > will
        >> > > >     > > be
        >> > > >     > > >    kept but updated so that their default behaviour if
        >> > setting
        >> > > > null
        >> > > >     > value
        >> > > >     > > > will
        >> > > >     > > >    be the tombstone will be set to true to keep existing
        >> > > > behaviour.
        >> > > >     > > >
        >> > > >     > > >
        >> > > >     > > > So maybe I am misinterpreting something.
        >> > > >     > > >
        >> > > >     > > > And just a nit re: motivation section, item 6 would be
        >> > clearer
        >> > > > for a
        >> > > >     > > union
        >> > > >     > > > schema with null (or optional schemas in other formats),
        >> e.g.
        >> > > > [null,
        >> > > >     > > > string], in which case losing the schema truly is losing
        >> > > > information
        >> > > >     > > > (whereas null is already the only valid value for a pure
        >> null
        >> > > > schema).
        >> > > >     > > >
        >> > > >     > > > -Ewen
        >> > > >     > > >
        >> > > >     > > >
        >> > > >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
        >> > > > Michael.Pearce@ig.com
        >> > > >     > >
        >> > > >     > > > wrote:
        >> > > >     > > >
        >> > > >     > > > > Hi Jay,
        >> > > >     > > > >
        >> > > >     > > > > Good point this detail is missing in the KIP write up.
        >> Ive
        >> > > > added this
        >> > > >     > > > now.
        >> > > >     > > > >
        >> > > >     > > > > Essentially simply just upgrading the clients we do not
        >> > > expect
        >> > > > any
        >> > > >     > > client
        >> > > >     > > > > app code change needed.
        >> > > >     > > > >
        >> > > >     > > > > Cheers
        >> > > >     > > > > Mike
        >> > > >     > > > > ________________________________________
        >> > > >     > > > > From: Jay Kreps <ja...@confluent.io>
        >> > > >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
        >> > > >     > > > > To: dev@kafka.apache.org
        >> > > >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
        >> Flag
        >> > > >     > > > >
        >> > > >     > > > > Michael,
        >> > > >     > > > >
        >> > > >     > > > > The compatibility section goes through the migration
        >> path,
        >> > > but
        >> > > > isn't
        >> > > >     > > the
        >> > > >     > > > > bigger compatibility issue with existing apps? There are
        >> > many
        >> > > >     > (probably
        >> > > >     > > > > thousands) of apps in production that use this feature
        >> and
        >> > > > send null
        >> > > >     > to
        >> > > >     > > > > mean delete. It seems like this would break
        >> compatibility
        >> > > with
        >> > > > them,
        >> > > >     > > and
        >> > > >     > > > > they would have to be rewritten to right?
        >> > > >     > > > >
        >> > > >     > > > > -Jay
        >> > > >     > > > >
        >> > > >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
        >> > > >     > Michael.Pearce@ig.com
        >> > > >     > > >
        >> > > >     > > > > wrote:
        >> > > >     > > > >
        >> > > >     > > > > > Hi Jun,
        >> > > >     > > > > >
        >> > > >     > > > > > 4) On v3 we honour the tombstone. As such we expect
        >> it to
        >> > > be
        >> > > > set
        >> > > >     > > > > correctly
        >> > > >     > > > > > as per the KIP.
        >> > > >     > > > > >
        >> > > >     > > > > > 4.1) why would we want to produce an error when its
        >> v3?
        >> > > This
        >> > > > is the
        >> > > >     > > > exact
        >> > > >     > > > > > purpose to support non-null tombstone's
        >> > > >     > > > > > 4.2) again here im unclear on the question on the v3,
        >> > > produce
        >> > > >     > request
        >> > > >     > > > we
        >> > > >     > > > > > expect the tombstone flag to be set correctly.
        >> > > >     > > > > >
        >> > > >     > > > > > 4.4) compaction only occurs on compacted topics, the
        >> bit
        >> > > > makes no
        >> > > >     > > > > > difference and not looked at on un-compacted
        >> (time/size
        >> > > based
        >> > > >     > > > eviction).
        >> > > >     > > > > >
        >> > > >     > > > > >
        >> > > >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io>
        >> > wrote:
        >> > > >     > > > > >
        >> > > >     > > > > >     Hi, Michael,
        >> > > >     > > > > >
        >> > > >     > > > > >     4. Then, I think I misunderstood this point. Could
        >> > you
        >> > > > document
        >> > > >     > > the
        >> > > >     > > > > >     following points in the wiki?
        >> > > >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
        >> > > > non-null
        >> > > >     > value,
        >> > > >     > > > > does
        >> > > >     > > > > > the
        >> > > >     > > > > >     send() get an error or does the producer
        >> > automatically
        >> > > > set the
        >> > > >     > > > value
        >> > > >     > > > > to
        >> > > >     > > > > >     null?
        >> > > >     > > > > >     4.2 If producer V3 doesn't set tombstone, but
        >> > provides
        >> > > a
        >> > > > null
        >> > > >     > > > value,
        >> > > >     > > > > > does
        >> > > >     > > > > >     the send() get an error or does the producer
        >> > > > automatically sets
        >> > > >     > > the
        >> > > >     > > > > >     tombstone?
        >> > > >     > > > > >     4.3 Does the broker only expect messages that (a)
        >> > have
        >> > > no
        >> > > >     > > tombstone
        >> > > >     > > > > and
        >> > > >     > > > > >     non-null value; (b) have tombstone and null value
        >> and
        >> > > > reject
        >> > > >     > the
        >> > > >     > > > > > messages
        >> > > >     > > > > >     with an error code otherwise?
        >> > > >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic
        >> is
        >> > > > compacted
        >> > > >     > on
        >> > > >     > > > > not?
        >> > > >     > > > > >
        >> > > >     > > > > >     Thanks,
        >> > > >     > > > > >
        >> > > >     > > > > >     Jun
        >> > > >     > > > > >
        >> > > >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
        >> > > >     > > > > Michael.Pearce@ig.com
        >> > > >     > > > > > >
        >> > > >     > > > > >     wrote:
        >> > > >     > > > > >
        >> > > >     > > > > >     > Not at all.  This only acts on compacted topics
        >> > just
        >> > > > as what
        >> > > >     > > > occurs
        >> > > >     > > > > > today
        >> > > >     > > > > >     >
        >> > > >     > > > > >     > Sent using OWA for iPhone
        >> > > >     > > > > >     > ________________________________________
        >> > > >     > > > > >     > From: Jun Rao <ju...@confluent.io>
        >> > > >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
        >> > > >     > > > > >     > To: dev@kafka.apache.org
        >> > > >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction
        >> > Tombstone
        >> > > > Flag
        >> > > >     > > > > >     >
        >> > > >     > > > > >     > Hi, Michael,
        >> > > >     > > > > >     >
        >> > > >     > > > > >     > 4. Hmm, does that mean the new client library
        >> can
        >> > > > never send
        >> > > >     > a
        >> > > >     > > > null
        >> > > >     > > > > > message
        >> > > >     > > > > >     > even to a regular topic? This seems like a
        >> change
        >> > of
        >> > > > the
        >> > > >     > > existing
        >> > > >     > > > > > behavior.
        >> > > >     > > > > >     >
        >> > > >     > > > > >     > Thanks,
        >> > > >     > > > > >     >
        >> > > >     > > > > >     > Jun
        >> > > >     > > > > >     >
        >> > > >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
        >> > > >     > > > > > Michael.Pearce@ig.com>
        >> > > >     > > > > >     > wrote:
        >> > > >     > > > > >     >
        >> > > >     > > > > >     > > Hi Jun,
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > Re 4) That's because we expect the tombstone
        >> > value
        >> > > > to be
        >> > > >     > set
        >> > > >     > > > > > correctly if
        >> > > >     > > > > >     > > message bit is 2, as such if an older client
        >> > sends
        >> > > > in on
        >> > > >     > old
        >> > > >     > > > > > message the
        >> > > >     > > > > >     > > message is upcast and the bit is set
        >> correctly.
        >> > And
        >> > > > such no
        >> > > >     > > > > longer
        >> > > >     > > > > > need
        >> > > >     > > > > >     > to
        >> > > >     > > > > >     > > check the value. Mayuresh can you confirm my
        >> > > > thinking and
        >> > > >     > > > > > understanding
        >> > > >     > > > > >     > of
        >> > > >     > > > > >     > > what we've discussed?
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > The second point I understand what you're
        >> getting
        >> > > at
        >> > > > now my
        >> > > >     > > > > > apologies.
        >> > > >     > > > > >     > Yes
        >> > > >     > > > > >     > > this makes sense to save on touching the
        >> message,
        >> > > if
        >> > > > we're
        >> > > >     > > the
        >> > > >     > > > > > only kip
        >> > > >     > > > > >     > > going in, in this release.
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > Cheers
        >> > > >     > > > > >     > > Mike
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > Sent using OWA for iPhone
        >> > > >     > > > > >     > > ________________________________________
        >> > > >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
        >> > > >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
        >> > > >     > > > > >     > > To: dev@kafka.apache.org
        >> > > >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
        >> > > > Tombstone Flag
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > Hi, Michael,
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > 4. Is this updated in the wiki? The text "If
        >> the
        >> > > > magic byte
        >> > > >     > > on
        >> > > >     > > > > > message is
        >> > > >     > > > > >     > > 2, the broker should use the tombstone bit for
        >> > log
        >> > > >     > > compaction."
        >> > > >     > > > > > doesn't
        >> > > >     > > > > >     > > seem to have changed.
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > 2. My point is that if we change the message
        >> > format
        >> > > > just
        >> > > >     > for
        >> > > >     > > > this
        >> > > >     > > > > > KIP, we
        >> > > >     > > > > >     > > should consider whether it's worth optimizing
        >> the
        >> > > > down
        >> > > >     > > > conversion
        >> > > >     > > > > > path
        >> > > >     > > > > >     > > (i.e., decide whether a conversion is needed
        >> by
        >> > > just
        >> > > >     > looking
        >> > > >     > > at
        >> > > >     > > > > the
        >> > > >     > > > > >     > > tombstone bit in the wrapper message) since
        >> > > > tombstone will
        >> > > >     > be
        >> > > >     > > > > used
        >> > > >     > > > > >     > rarely.
        >> > > >     > > > > >     > > However, if the message format change here is
        >> > > > combined with
        >> > > >     > > > other
        >> > > >     > > > > > KIPs,
        >> > > >     > > > > >     > > then this optimization likely won't be needed.
        >> > The
        >> > > > latter
        >> > > >     > > > > probably
        >> > > >     > > > > > makes
        >> > > >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do
        >> you
        >> > > > think?
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > Other than those, +1 from me,
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > Thanks,
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > Jun
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael
        >> Pearce <
        >> > > >     > > > > > Michael.Pearce@ig.com>
        >> > > >     > > > > >     > > wrote:
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > > > Hi Jun
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > do we have your vote on this now?
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > Any other concerns?
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > Cheers
        >> > > >     > > > > >     > > > Mike
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > Sent using OWA for iPhone
        >> > > >     > > > > >     > > > ________________________________________
        >> > > >     > > > > >     > > > From: Michael Pearce <Michael.Pearce@ig.com
        >> >
        >> > > >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
        >> > > >     > > > > >     > > > To: dev@kafka.apache.org
        >> > > >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
        >> > > > Tombstone
        >> > > >     > Flag
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > Hi Jun,
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > Have updated. Thanks again for the feedback.
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > Agree yes we should align up when it gets to
        >> > > that,
        >> > > > I
        >> > > >     > assume
        >> > > >     > > > > > you've
        >> > > >     > > > > >     > > flagged
        >> > > >     > > > > >     > > > the same in the other KIP?
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > I think for now let's get this KIP approved,
        >> > then
        >> > > > we can
        >> > > >     > > > start
        >> > > >     > > > > > the work
        >> > > >     > > > > >     > > to
        >> > > >     > > > > >     > > > get an initial PR, then we can discuss how
        >> to
        >> > > > align the
        >> > > >     > two
        >> > > >     > > > if
        >> > > >     > > > > > needed.
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > Cheers,
        >> > > >     > > > > >     > > > Mike
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
        >> > > jun@confluent.io>
        >> > > >     > wrote:
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >     Hi, Michael,
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >     For 2), this is fine. Could you update
        >> the
        >> > > KIP
        >> > > > wiki
        >> > > >     > to
        >> > > >     > > > make
        >> > > >     > > > > > this
        >> > > >     > > > > >     > and
        >> > > >     > > > > >     > > > other
        >> > > >     > > > > >     > > >     points clearer? Other than that, the KIP
        >> > > looks
        >> > > > good
        >> > > >     > to
        >> > > >     > > > me.
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >     An orthogonal thing is that there are
        >> other
        >> > > > KIPs such
        >> > > >     > > as
        >> > > >     > > > > > KIP-98
        >> > > >     > > > > >     > that
        >> > > >     > > > > >     > > > also
        >> > > >     > > > > >     > > >     intend to change the message format. If
        >> > they
        >> > > > all get
        >> > > >     > > > > > approved, we
        >> > > >     > > > > >     > > > should
        >> > > >     > > > > >     > > >     think about whether it's better to just
        >> > bump
        >> > > > up the
        >> > > >     > > magic
        >> > > >     > > > > > byte once
        >> > > >     > > > > >     > > to
        >> > > >     > > > > >     > > >     incorporate multiple format changes
        >> like we
        >> > > > did in
        >> > > >     > > > > > KIP-31/KIP-32.
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >     Thanks,
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >     Jun
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
        >> > > Pearce
        >> > > > <
        >> > > >     > > > > >     > > Michael.Pearce@ig.com>
        >> > > >     > > > > >     > > >     wrote:
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >     > Hi Jao
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     > Thanks for the response. Sorry for
        >> slow
        >> > > > reply, both
        >> > > >     > > > with
        >> > > >     > > > > > personal
        >> > > >     > > > > >     > > > sickness
        >> > > >     > > > > >     > > >     > and also battling some critical issues
        >> > > > encountered
        >> > > >     > > > since
        >> > > >     > > > > >     > upgrading
        >> > > >     > > > > >     > > to
        >> > > >     > > > > >     > > >     > 0.10.1.0
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     > 1) Thans for spotting, Document error
        >> > where
        >> > > > we
        >> > > >     > > branched
        >> > > >     > > > > > this KIP
        >> > > >     > > > > >     > > from
        >> > > >     > > > > >     > > >     > KIP-82, will get that removed.
        >> > > >     > > > > >     > > >     > 2) Intent is to do this just at the
        >> > record
        >> > > > message
        >> > > >     > > > level.
        >> > > >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure
        >> this
        >> > is
        >> > > >     > > corrected.
        >> > > >     > > > > >     > > >     > 4) As per discussion thread we will
        >> > support
        >> > > >     > > tombstone +
        >> > > >     > > > > > null
        >> > > >     > > > > >     > value,
        >> > > >     > > > > >     > > >     > tombstone + non null value, no
        >> tombstone
        >> > +
        >> > > > null
        >> > > >     > > value.
        >> > > >     > > > > >     > > >     > 5) I believe this was in the
        >> discussion
        >> > > > thread,
        >> > > >     > > > @Mayuresh
        >> > > >     > > > > > is this
        >> > > >     > > > > >     > > >     > something we've overlooked? I thought
        >> we
        >> > > > would down
        >> > > >     > > > > > convert and
        >> > > >     > > > > >     > > > remove the
        >> > > >     > > > > >     > > >     > value so the old consumer had existing
        >> > > > behavior, or
        >> > > >     > > is
        >> > > >     > > > > > there
        >> > > >     > > > > >     > > > something we
        >> > > >     > > > > >     > > >     > haven't thought about?
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     > Cheers
        >> > > >     > > > > >     > > >     > Mike
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
        >> > > > jun@confluent.io>
        >> > > >     > > > > wrote:
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     Hi, Michael,
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     Thanks for the KIP. A few comments
        >> > > below.
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     1. The message format change
        >> contains
        >> > > >     > > > "HeadersLength
        >> > > >     > > > > >     > Headers".
        >> > > >     > > > > >     > > > Is that
        >> > > >     > > > > >     > > >     >     intended?
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     2. For compressed messageset, is
        >> the
        >> > > > tombstone
        >> > > >     > > bit
        >> > > >     > > > > > only set
        >> > > >     > > > > >     > at
        >> > > >     > > > > >     > > > the
        >> > > >     > > > > >     > > >     > shallow
        >> > > >     > > > > >     > > >     >     level? Do we always leave that
        >> bit in
        >> > > the
        >> > > >     > wrapper
        >> > > >     > > > > > message
        >> > > >     > > > > >     > > unset?
        >> > > >     > > > > >     > > > An
        >> > > >     > > > > >     > > >     >     alternative is to set the
        >> tombstone
        >> > bit
        >> > > > in the
        >> > > >     > > > > wrapper
        >> > > >     > > > > > if at
        >> > > >     > > > > >     > > > least one
        >> > > >     > > > > >     > > >     >     inner message has the tombstone
        >> bit
        >> > > set.
        >> > > > This
        >> > > >     > > makes
        >> > > >     > > > > > things a
        >> > > >     > > > > >     > > bit
        >> > > >     > > > > >     > > > more
        >> > > >     > > > > >     > > >     >     complicated, but we could
        >> potentially
        >> > > > exploit
        >> > > >     > > that
        >> > > >     > > > > for
        >> > > >     > > > > >     > > > optimizing down
        >> > > >     > > > > >     > > >     >     conversion. For example, we only
        >> need
        >> > > to
        >> > > >     > convert
        >> > > >     > > > > > messages
        >> > > >     > > > > >     > with
        >> > > >     > > > > >     > > > magic 2
        >> > > >     > > > > >     > > >     > to
        >> > > >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone
        >> > bit
        >> > > > is set
        >> > > >     > > > > > (conversion is
        >> > > >     > > > > >     > > > always
        >> > > >     > > > > >     > > >     > needed
        >> > > >     > > > > >     > > >     >     from magic 2 to magic 0). Not
        >> sure if
        >> > > the
        >> > > >     > > > > optimization
        >> > > >     > > > > > is
        >> > > >     > > > > >     > worth
        >> > > >     > > > > >     > > > the
        >> > > >     > > > > >     > > >     >     complexity though.
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     3. The referencing of the new
        >> version
        >> > > of
        >> > > >     > > > > >     > > > ProducerRequest/FetchRequest
        >> > > >     > > > > >     > > >     > is
        >> > > >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
        >> > > > convention
        >> > > >     > > > starts
        >> > > >     > > > > at
        >> > > >     > > > > >     > version
        >> > > >     > > > > >     > > > at 0, I
        >> > > >     > > > > >     > > >     >     think the new version would be 3.
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     4. "If the magic byte on message
        >> is
        >> > 2,
        >> > > > the
        >> > > >     > broker
        >> > > >     > > > > > should use
        >> > > >     > > > > >     > > the
        >> > > >     > > > > >     > > >     > tombstone
        >> > > >     > > > > >     > > >     >     bit for log compaction." What
        >> about
        >> > > null
        >> > > > value?
        >> > > >     > > My
        >> > > >     > > > > >     > > understanding
        >> > > >     > > > > >     > > > is
        >> > > >     > > > > >     > > >     > that
        >> > > >     > > > > >     > > >     >     null value will be treated the
        >> same
        >> > as
        >> > > > setting
        >> > > >     > > the
        >> > > >     > > > > > tombstone
        >> > > >     > > > > >     > > bit.
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     5. For the migration path, it
        >> would
        >> > be
        >> > > > useful
        >> > > >     > to
        >> > > >     > > > > > describe the
        >> > > >     > > > > >     > > > down
        >> > > >     > > > > >     > > >     >     conversion path to consumers
        >> (i.e.,
        >> > > > brokers on
        >> > > >     > > > > message
        >> > > >     > > > > > format
        >> > > >     > > > > >     > > > 0.10.2
        >> > > >     > > > > >     > > >     > and
        >> > > >     > > > > >     > > >     >     consumers on older version).
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     Thanks,
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     Jun
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
        >> > > Michael
        >> > > >     > Pearce <
        >> > > >     > > > > >     > > > Michael.Pearce@ig.com
        >> > > >     > > > > >     > > >     > >
        >> > > >     > > > > >     > > >     >     wrote:
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >     > Hi All,
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     > We have been discussing in the
        >> > below
        >> > > > thread
        >> > > >     > and
        >> > > >     > > > > final
        >> > > >     > > > > >     > changes
        >> > > >     > > > > >     > > > have
        >> > > >     > > > > >     > > >     > been
        >> > > >     > > > > >     > > >     >     > made to the KIP wiki based on
        >> these
        >> > > >     > > discussions.
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     > We would now like to put to the
        >> > vote
        >> > > > the
        >> > > >     > > > following
        >> > > >     > > > > > KIP:
        >> > > >     > > > > >     > > >     >     > https://cwiki.apache.org/
        >> > > >     > > > > > confluence/display/KAFKA/KIP-
        >> > > >     > > > > >     > 87+-+
        >> > > >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     > This kip is for having a
        >> distinct
        >> > > > compaction
        >> > > >     > > > > > attribute
        >> > > >     > > > > >     > > > "tombstone"
        >> > > >     > > > > >     > > >     > flag
        >> > > >     > > > > >     > > >     >     > instead of relying on null
        >> value,
        >> > > > allowing
        >> > > >     > > > non-null
        >> > > >     > > > > > value
        >> > > >     > > > > >     > > > delete
        >> > > >     > > > > >     > > >     > messages.
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     > Many thanks,
        >> > > >     > > > > >     > > >     >     > Michael
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
        >> > > Pearce"
        >> > > > <
        >> > > >     > > > > >     > > Michael.Pearce@ig.com>
        >> > > >     > > > > >     > > >     > wrote:
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >     Hi Mayuresh,
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >     LGTM. Ive just made one
        >> small
        >> > > > adjustment
        >> > > >     > > > > > updating the
        >> > > >     > > > > >     > > wire
        >> > > >     > > > > >     > > >     > protocol to
        >> > > >     > > > > >     > > >     >     > show the magic byte bump.
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >     Do we think we're good to
        >> put
        >> > to
        >> > > a
        >> > > > vote?
        >> > > >     > Is
        >> > > >     > > > > > there any
        >> > > >     > > > > >     > > > other bits
        >> > > >     > > > > >     > > >     >     > needing discussion?
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >     Cheers
        >> > > >     > > > > >     > > >     >     >     Mike
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >     On 21/11/2016, 18:26,
        >> "Mayuresh
        >> > > > Gharat" <
        >> > > >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
        >> > > >     > > > > >     > > >     >     > wrote:
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >         Hi Michael,
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >         I have updated the
        >> > migration
        >> > > > section
        >> > > >     > of
        >> > > >     > > > the
        >> > > >     > > > > > KIP.
        >> > > >     > > > > >     > Can
        >> > > >     > > > > >     > > > you
        >> > > >     > > > > >     > > >     > please
        >> > > >     > > > > >     > > >     >     > take a look?
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >         Thanks,
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >         Mayuresh
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at
        >> > 9:07
        >> > > > AM,
        >> > > >     > > Mayuresh
        >> > > >     > > > > > Gharat <
        >> > > >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
        >> > > >     > > > > >     > > >     >     >         > wrote:
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >         > Hi Michael,
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         > That whilst sending
        >> > > > tombstone and
        >> > > >     > non
        >> > > >     > > > > null
        >> > > >     > > > > > value,
        >> > > >     > > > > >     > > the
        >> > > >     > > > > >     > > >     > consumer
        >> > > >     > > > > >     > > >     >     > can expect
        >> > > >     > > > > >     > > >     >     >         > only to receive the
        >> > > non-null
        >> > > >     > message
        >> > > >     > > > only
        >> > > >     > > > > > in step
        >> > > >     > > > > >     > > > (3) is
        >> > > >     > > > > >     > > >     > this
        >> > > >     > > > > >     > > >     >     > correct?
        >> > > >     > > > > >     > > >     >     >         > ---> I do agree with
        >> you
        >> > > > here.
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         > Becket, Ismael : can
        >> you
        >> > > guys
        >> > > >     > review
        >> > > >     > > > the
        >> > > >     > > > > >     > migration
        >> > > >     > > > > >     > > > plan
        >> > > >     > > > > >     > > >     > listed
        >> > > >     > > > > >     > > >     >     > above using
        >> > > >     > > > > >     > > >     >     >         > magic byte?
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         > Thanks,
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         > Mayuresh
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016
        >> at
        >> > > 8:58
        >> > > > AM,
        >> > > >     > > > Michael
        >> > > >     > > > > > Pearce <
        >> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
        >> > > >     > > > > >     > > >     >     >         > wrote:
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         >> Many thanks for this
        >> > > > Mayuresh. I
        >> > > >     > > don't
        >> > > >     > > > > > have any
        >> > > >     > > > > >     > > >     > objections.
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> I assume we should
        >> > state:
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> That whilst sending
        >> > > > tombstone and
        >> > > >     > > non
        >> > > >     > > > > null
        >> > > >     > > > > >     > value,
        >> > > >     > > > > >     > > > the
        >> > > >     > > > > >     > > >     > consumer
        >> > > >     > > > > >     > > >     >     > can expect
        >> > > >     > > > > >     > > >     >     >         >> only to receive the
        >> > > non-null
        >> > > >     > message
        >> > > >     > > > > only
        >> > > >     > > > > > in
        >> > > >     > > > > >     > step
        >> > > >     > > > > >     > > > (3) is
        >> > > >     > > > > >     > > >     > this
        >> > > >     > > > > >     > > >     >     > correct?
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Cheers
        >> > > >     > > > > >     > > >     >     >         >> Mike
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Sent using OWA for
        >> > iPhone
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > > ______________________________
        >> > > >     > > > > __________
        >> > > >     > > > > >     > > >     >     >         >> From: Mayuresh
        >> Gharat <
        >> > > >     > > > > >     > gharatmayuresh15@gmail.com
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >     >     >         >> Sent: Thursday,
        >> November
        >> > > > 17, 2016
        >> > > >     > > > > 5:18:41
        >> > > >     > > > > > PM
        >> > > >     > > > > >     > > >     >     >         >> To:
        >> > dev@kafka.apache.org
        >> > > >     > > > > >     > > >     >     >         >> Subject: Re:
        >> [DISCUSS]
        >> > > > KIP-87 -
        >> > > >     > Add
        >> > > >     > > > > > Compaction
        >> > > >     > > > > >     > > > Tombstone
        >> > > >     > > > > >     > > >     > Flag
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Hi Ismael,
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Thanks for the
        >> > > explanation.
        >> > > >     > > > > >     > > >     >     >         >> Specially I like this
        >> > part
        >> > > > where
        >> > > >     > in
        >> > > >     > > > you
        >> > > >     > > > > >     > mentioned
        >> > > >     > > > > >     > > > we can
        >> > > >     > > > > >     > > >     > get
        >> > > >     > > > > >     > > >     >     > rid of the
        >> > > >     > > > > >     > > >     >     >         >> older null value
        >> support
        >> > > > for log
        >> > > >     > > > > > compaction
        >> > > >     > > > > >     > later
        >> > > >     > > > > >     > > > on,
        >> > > >     > > > > >     > > >     > here :
        >> > > >     > > > > >     > > >     >     >         >> We can't change
        >> > semantics
        >> > > > of the
        >> > > >     > > > message
        >> > > >     > > > > > format
        >> > > >     > > > > >     > > > without
        >> > > >     > > > > >     > > >     > having
        >> > > >     > > > > >     > > >     >     > a long
        >> > > >     > > > > >     > > >     >     >         >> transition period.
        >> And
        >> > we
        >> > > > can't
        >> > > >     > rely
        >> > > >     > > > > >     > > >     >     >         >> on people reading
        >> > > > documentation or
        >> > > >     > > > > acting
        >> > > >     > > > > > on a
        >> > > >     > > > > >     > > > warning for
        >> > > >     > > > > >     > > >     >     > something so
        >> > > >     > > > > >     > > >     >     >         >> fundamental. As
        >> such, my
        >> > > > take is
        >> > > >     > > that
        >> > > >     > > > we
        >> > > >     > > > > > need to
        >> > > >     > > > > >     > > > bump the
        >> > > >     > > > > >     > > >     > magic
        >> > > >     > > > > >     > > >     >     > byte. The
        >> > > >     > > > > >     > > >     >     >         >> good news is
        >> > > >     > > > > >     > > >     >     >         >> that we don't have to
        >> > > > support all
        >> > > >     > > > > versions
        >> > > >     > > > > >     > > forever.
        >> > > >     > > > > >     > > > We
        >> > > >     > > > > >     > > >     > have
        >> > > >     > > > > >     > > >     >     > said that we
        >> > > >     > > > > >     > > >     >     >         >> will support direct
        >> > > > upgrades for 2
        >> > > >     > > > > years.
        >> > > >     > > > > > That
        >> > > >     > > > > >     > > > means that
        >> > > >     > > > > >     > > >     >     > message format
        >> > > >     > > > > >     > > >     >     >         >> version n could, in
        >> > > theory,
        >> > > > be
        >> > > >     > > > removed 2
        >> > > >     > > > > > years
        >> > > >     > > > > >     > > > after the
        >> > > >     > > > > >     > > >     > it's
        >> > > >     > > > > >     > > >     >     > introduced.
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Just a heads up, I
        >> would
        >> > > > like to
        >> > > >     > > > mention
        >> > > >     > > > > > that
        >> > > >     > > > > >     > even
        >> > > >     > > > > >     > > > without
        >> > > >     > > > > >     > > >     >     > bumping magic
        >> > > >     > > > > >     > > >     >     >         >> byte, we will *NOT*
        >> > loose
        >> > > > zero
        >> > > >     > copy
        >> > > >     > > as
        >> > > >     > > > > in
        >> > > >     > > > > > the
        >> > > >     > > > > >     > > > client(x+1)
        >> > > >     > > > > >     > > >     > in my
        >> > > >     > > > > >     > > >     >     >         >> explanation
        >> > > >     > > > > >     > > >     >     >         >> above will convert
        >> > > > internally a
        >> > > >     > null
        >> > > >     > > > > > value to
        >> > > >     > > > > >     > > have a
        >> > > >     > > > > >     > > >     > tombstone
        >> > > >     > > > > >     > > >     >     > bit set and
        >> > > >     > > > > >     > > >     >     >         >> a tombstone bit set
        >> to
        >> > > have
        >> > > > a null
        >> > > >     > > > value
        >> > > >     > > > > >     > > > automatically
        >> > > >     > > > > >     > > >     >     > internally and by
        >> > > >     > > > > >     > > >     >     >         >> the time we move to
        >> > > version
        >> > > > (x+2),
        >> > > >     > > the
        >> > > >     > > > > > clients
        >> > > >     > > > > >     > > > would have
        >> > > >     > > > > >     > > >     >     > upgraded.
        >> > > >     > > > > >     > > >     >     >         >> Obviously if we
        >> support
        >> > a
        >> > > > request
        >> > > >     > > from
        >> > > >     > > > > >     > > consumer(x),
        >> > > >     > > > > >     > > > we
        >> > > >     > > > > >     > > >     > will
        >> > > >     > > > > >     > > >     >     > loose zero
        >> > > >     > > > > >     > > >     >     >         >> copy
        >> > > >     > > > > >     > > >     >     >         >> but that is the same
        >> > case
        >> > > > with
        >> > > >     > magic
        >> > > >     > > > > byte.
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> But if magic byte
        >> bump
        >> > > > makes life
        >> > > >     > > > easier
        >> > > >     > > > > > for
        >> > > >     > > > > >     > > > transition
        >> > > >     > > > > >     > > >     > for the
        >> > > >     > > > > >     > > >     >     > above
        >> > > >     > > > > >     > > >     >     >         >> reasons that you
        >> > > explained,
        >> > > > I am
        >> > > >     > OK
        >> > > >     > > > with
        >> > > >     > > > > > it
        >> > > >     > > > > >     > since
        >> > > >     > > > > >     > > > we are
        >> > > >     > > > > >     > > >     > going
        >> > > >     > > > > >     > > >     >     > to meet the
        >> > > >     > > > > >     > > >     >     >         >> end goal down the
        >> road
        >> > :)
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> On a side note can we
        >> > > > update the
        >> > > >     > doc
        >> > > >     > > > > here
        >> > > >     > > > > > on
        >> > > >     > > > > >     > magic
        >> > > >     > > > > >     > > > byte
        >> > > >     > > > > >     > > >     > to say
        >> > > >     > > > > >     > > >     >     > that "*it
        >> > > >     > > > > >     > > >     >     >         >> should be bumped
        >> > whenever
        >> > > > the
        >> > > >     > > message
        >> > > >     > > > > > format is
        >> > > >     > > > > >     > > > changed
        >> > > >     > > > > >     > > >     > or the
        >> > > >     > > > > >     > > >     >     >         >> interpretation of
        >> > message
        >> > > > format
        >> > > >     > > > (usage
        >> > > >     > > > > > of the
        >> > > >     > > > > >     > > > reserved
        >> > > >     > > > > >     > > >     > bits as
        >> > > >     > > > > >     > > >     >     > well) is
        >> > > >     > > > > >     > > >     >     >         >> changed*".
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Hi Michael,
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Here is the update
        >> plan
        >> > > > that we
        >> > > >     > > > > discussed
        >> > > >     > > > > >     > offline
        >> > > >     > > > > >     > > >     > yesterday :
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Currently the
        >> magic-byte
        >> > > > which
        >> > > >     > > > > > corresponds to
        >> > > >     > > > > >     > the
        >> > > >     > > > > >     > > >     >     > "message.format.version"
        >> > > >     > > > > >     > > >     >     >         >> is set to 1.
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> 1) On broker it will
        >> be
        >> > > set
        >> > > > to 1
        >> > > >     > > > > > initially.
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> 2) When a producer
        >> > client
        >> > > > sends a
        >> > > >     > > > > message
        >> > > >     > > > > > with
        >> > > >     > > > > >     > > > magic-byte
        >> > > >     > > > > >     > > >     > = 2,
        >> > > >     > > > > >     > > >     >     > since the
        >> > > >     > > > > >     > > >     >     >         >> broker is on
        >> magic-byte
        >> > =
        >> > > > 1, we
        >> > > >     > will
        >> > > >     > > > > down
        >> > > >     > > > > >     > convert
        >> > > >     > > > > >     > > > it,
        >> > > >     > > > > >     > > >     > which
        >> > > >     > > > > >     > > >     >     > means if the
        >> > > >     > > > > >     > > >     >     >         >> tombstone bit is set,
        >> > the
        >> > > > value
        >> > > >     > will
        >> > > >     > > > be
        >> > > >     > > > > > set to
        >> > > >     > > > > >     > > > null. A
        >> > > >     > > > > >     > > >     > consumer
        >> > > >     > > > > >     > > >     >     >         >> understanding
        >> > magic-byte =
        >> > > > 1, will
        >> > > >     > > > still
        >> > > >     > > > > > work
        >> > > >     > > > > >     > with
        >> > > >     > > > > >     > > > this. A
        >> > > >     > > > > >     > > >     >     > consumer
        >> > > >     > > > > >     > > >     >     >         >> working
        >> > > >     > > > > >     > > >     >     >         >> with magic-byte =2
        >> will
        >> > > > also be
        >> > > >     > able
        >> > > >     > > > to
        >> > > >     > > > > >     > understand
        >> > > >     > > > > >     > > > this,
        >> > > >     > > > > >     > > >     > since
        >> > > >     > > > > >     > > >     >     > it
        >> > > >     > > > > >     > > >     >     >         >> understands the
        >> > tombstone.
        >> > > >     > > > > >     > > >     >     >         >> Now there is still
        >> the
        >> > > > question of
        >> > > >     > > > > > supporting a
        >> > > >     > > > > >     > > >     > non-tombstone
        >> > > >     > > > > >     > > >     >     > and null
        >> > > >     > > > > >     > > >     >     >         >> value from producer
        >> > client
        >> > > > with
        >> > > >     > > > > > magic-byte = 2.*
        >> > > >     > > > > >     > > (I
        >> > > >     > > > > >     > > > am
        >> > > >     > > > > >     > > >     > not sure
        >> > > >     > > > > >     > > >     >     > if we
        >> > > >     > > > > >     > > >     >     >         >> should support this.
        >> > > > Ismael/Becket
        >> > > >     > > can
        >> > > >     > > > > > comment
        >> > > >     > > > > >     > > > here)*
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> 3) When almost all
        >> the
        >> > > > clients
        >> > > >     > have
        >> > > >     > > > > > upgraded,
        >> > > >     > > > > >     > the
        >> > > >     > > > > >     > > >     >     > message.format.version
        >> > > >     > > > > >     > > >     >     >         >> on
        >> > > >     > > > > >     > > >     >     >         >> the broker can be
        >> > changed
        >> > > > to 2,
        >> > > >     > > where
        >> > > >     > > > in
        >> > > >     > > > > > the
        >> > > >     > > > > >     > down
        >> > > >     > > > > >     > > >     > conversion in
        >> > > >     > > > > >     > > >     >     > the above
        >> > > >     > > > > >     > > >     >     >         >> step will not
        >> happen. If
        >> > > at
        >> > > > this
        >> > > >     > > point
        >> > > >     > > > > we
        >> > > >     > > > > > get a
        >> > > >     > > > > >     > > > consumer
        >> > > >     > > > > >     > > >     >     > request from a
        >> > > >     > > > > >     > > >     >     >         >> older consumer, we
        >> might
        >> > > > have to
        >> > > >     > > down
        >> > > >     > > > > > convert
        >> > > >     > > > > >     > > where
        >> > > >     > > > > >     > > > in we
        >> > > >     > > > > >     > > >     > loose
        >> > > >     > > > > >     > > >     >     > zero copy,
        >> > > >     > > > > >     > > >     >     >         >> but these cases
        >> should
        >> > be
        >> > > > rare.
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Becket can you review
        >> > this
        >> > > > plan
        >> > > >     > and
        >> > > >     > > > add
        >> > > >     > > > > > more
        >> > > >     > > > > >     > > > details if I
        >> > > >     > > > > >     > > >     > have
        >> > > >     > > > > >     > > >     >     >         >> missed/wronged
        >> > something,
        >> > > > before
        >> > > >     > we
        >> > > >     > > > put
        >> > > >     > > > > > it on
        >> > > >     > > > > >     > KIP.
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Thanks,
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> Mayuresh
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016
        >> at
        >> > > > 11:07 PM,
        >> > > >     > > > > Michael
        >> > > >     > > > > >     > Pearce <
        >> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
        >> > > >     > > > > >     > > >     >     >         >> wrote:
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> > Thanks guys, for
        >> > > > discussing this
        >> > > >     > > > > > offline and
        >> > > >     > > > > >     > > > getting
        >> > > >     > > > > >     > > >     > some
        >> > > >     > > > > >     > > >     >     > consensus.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > So its clear for
        >> > myself
        >> > > > and
        >> > > >     > others
        >> > > >     > > > > what
        >> > > >     > > > > > is
        >> > > >     > > > > >     > > > proposed now
        >> > > >     > > > > >     > > >     > (i
        >> > > >     > > > > >     > > >     >     > think i
        >> > > >     > > > > >     > > >     >     >         >> > understand, but
        >> want
        >> > to
        >> > > > make
        >> > > >     > sure)
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > Could i ask either
        >> > > > directly
        >> > > >     > update
        >> > > >     > > > the
        >> > > >     > > > > > kip to
        >> > > >     > > > > >     > > > detail the
        >> > > >     > > > > >     > > >     >     > migration
        >> > > >     > > > > >     > > >     >     >         >> > strategy, or
        >> > (re-)state
        >> > > > your
        >> > > >     > > offline
        >> > > >     > > > > > discussed
        >> > > >     > > > > >     > > and
        >> > > >     > > > > >     > > >     > agreed
        >> > > >     > > > > >     > > >     >     > migration
        >> > > >     > > > > >     > > >     >     >         >> > strategy based on a
        >> > > magic
        >> > > > byte
        >> > > >     > is
        >> > > >     > > in
        >> > > >     > > > > > this
        >> > > >     > > > > >     > > thread.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > The main original
        >> > driver
        >> > > > for the
        >> > > >     > > KIP
        >> > > >     > > > > > was to
        >> > > >     > > > > >     > > > support
        >> > > >     > > > > >     > > >     >     > compaction where
        >> > > >     > > > > >     > > >     >     >         >> value
        >> > > >     > > > > >     > > >     >     >         >> > isn't null, based
        >> off
        >> > > the
        >> > > >     > > > discussions
        >> > > >     > > > > on
        >> > > >     > > > > >     > KIP-82
        >> > > >     > > > > >     > > > thread.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > We should be able
        >> to
        >> > > > support
        >> > > >     > > > > > non-tombstone +
        >> > > >     > > > > >     > > null
        >> > > >     > > > > >     > > > value
        >> > > >     > > > > >     > > >     > by the
        >> > > >     > > > > >     > > >     >     >         >> completion
        >> > > >     > > > > >     > > >     >     >         >> > of the KIP, as we
        >> > noted
        >> > > > when
        >> > > >     > > > > discussing
        >> > > >     > > > > > this
        >> > > >     > > > > >     > > kip,
        >> > > >     > > > > >     > > > having
        >> > > >     > > > > >     > > >     >     > logic based on
        >> > > >     > > > > >     > > >     >     >         >> a
        >> > > >     > > > > >     > > >     >     >         >> > null value isn't
        >> very
        >> > > > clean and
        >> > > >     > > also
        >> > > >     > > > > > separates
        >> > > >     > > > > >     > > the
        >> > > >     > > > > >     > > >     > concerns.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > As discussed
        >> already
        >> > > > though we
        >> > > >     > can
        >> > > >     > > > > > split this
        >> > > >     > > > > >     > > into
        >> > > >     > > > > >     > > >     > KIP-87a
        >> > > >     > > > > >     > > >     >     > and KIP-87b
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > Where we look to
        >> > deliver
        >> > > > KIP-87a
        >> > > >     > > on
        >> > > >     > > > a
        >> > > >     > > > > >     > compacted
        >> > > >     > > > > >     > > > topic
        >> > > >     > > > > >     > > >     > (to
        >> > > >     > > > > >     > > >     >     > address the
        >> > > >     > > > > >     > > >     >     >         >> > immediate issues)
        >> > > >     > > > > >     > > >     >     >         >> > * tombstone + null
        >> > value
        >> > > >     > > > > >     > > >     >     >         >> > * tombstone +
        >> non-null
        >> > > > value
        >> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
        >> > > > non-null value
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > Then we can discuss
        >> > once
        >> > > > KIP-87a
        >> > > >     > > is
        >> > > >     > > > > > completed
        >> > > >     > > > > >     > > > options
        >> > > >     > > > > >     > > >     > later
        >> > > >     > > > > >     > > >     >     > and how we
        >> > > >     > > > > >     > > >     >     >         >> > support the second
        >> > part
        >> > > > KIP-87b
        >> > > >     > to
        >> > > >     > > > > > deliver:
        >> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
        >> null
        >> > > > value
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > Cheers
        >> > > >     > > > > >     > > >     >     >         >> > Mike
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > > ______________________________
        >> > > >     > > > > > __________
        >> > > >     > > > > >     > > >     >     >         >> > From: Becket Qin <
        >> > > >     > > > > becket.qin@gmail.com>
        >> > > >     > > > > >     > > >     >     >         >> > Sent: Thursday,
        >> > November
        >> > > > 17,
        >> > > >     > 2016
        >> > > >     > > > 1:43
        >> > > >     > > > > > AM
        >> > > >     > > > > >     > > >     >     >         >> > To:
        >> > > dev@kafka.apache.org
        >> > > >     > > > > >     > > >     >     >         >> > Subject: Re:
        >> [DISCUSS]
        >> > > > KIP-87 -
        >> > > >     > > Add
        >> > > >     > > > > > Compaction
        >> > > >     > > > > >     > > >     > Tombstone Flag
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and
        >> I
        >> > had
        >> > > > an
        >> > > >     > > offline
        >> > > >     > > > > >     > discussion,
        >> > > >     > > > > >     > > > and
        >> > > >     > > > > >     > > >     > following
        >> > > >     > > > > >     > > >     >     > is a brief
        >> > > >     > > > > >     > > >     >     >         >> > summary.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > 1. We agreed that
        >> not
        >> > > > bumping up
        >> > > >     > > > magic
        >> > > >     > > > > > value
        >> > > >     > > > > >     > may
        >> > > >     > > > > >     > > > result
        >> > > >     > > > > >     > > >     > in
        >> > > >     > > > > >     > > >     >     > losing zero
        >> > > >     > > > > >     > > >     >     >         >> copy
        >> > > >     > > > > >     > > >     >     >         >> > during migration.
        >> > > >     > > > > >     > > >     >     >         >> > 2. Given that
        >> bumping
        >> > up
        >> > > > magic
        >> > > >     > > value
        >> > > >     > > > > is
        >> > > >     > > > > > almost
        >> > > >     > > > > >     > > > free and
        >> > > >     > > > > >     > > >     > has
        >> > > >     > > > > >     > > >     >     > benefit of
        >> > > >     > > > > >     > > >     >     >         >> > avoiding potential
        >> > > > performance
        >> > > >     > > > issue.
        >> > > >     > > > > > It is
        >> > > >     > > > > >     > > > probably
        >> > > >     > > > > >     > > >     > worth
        >> > > >     > > > > >     > > >     >     > doing.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > One issue we still
        >> > need
        >> > > > to think
        >> > > >     > > > about
        >> > > >     > > > > > is
        >> > > >     > > > > >     > > whether
        >> > > >     > > > > >     > > > we
        >> > > >     > > > > >     > > >     > want to
        >> > > >     > > > > >     > > >     >     > support a
        >> > > >     > > > > >     > > >     >     >         >> > non-tombstone
        >> message
        >> > > > with null
        >> > > >     > > > value.
        >> > > >     > > > > >     > > >     >     >         >> > Currently it is not
        >> > > > supported by
        >> > > >     > > > > Kafka.
        >> > > >     > > > > > If we
        >> > > >     > > > > >     > > > allow a
        >> > > >     > > > > >     > > >     >     > non-tombstone null
        >> > > >     > > > > >     > > >     >     >         >> > value message to
        >> exist
        >> > > > after
        >> > > >     > > KIP-87.
        >> > > >     > > > > The
        >> > > >     > > > > >     > problem
        >> > > >     > > > > >     > > > is
        >> > > >     > > > > >     > > >     > that such
        >> > > >     > > > > >     > > >     >     > message
        >> > > >     > > > > >     > > >     >     >         >> will
        >> > > >     > > > > >     > > >     >     >         >> > not be supported by
        >> > the
        >> > > >     > consumers
        >> > > >     > > > > prior
        >> > > >     > > > > > to
        >> > > >     > > > > >     > > KIP-87.
        >> > > >     > > > > >     > > >     > Because a
        >> > > >     > > > > >     > > >     >     > null value
        >> > > >     > > > > >     > > >     >     >         >> > will always be
        >> > > > interpreted to a
        >> > > >     > > > > > tombstone.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > One option is that
        >> we
        >> > > > keep the
        >> > > >     > > > current
        >> > > >     > > > > > way,
        >> > > >     > > > > >     > i.e.
        >> > > >     > > > > >     > > > do not
        >> > > >     > > > > >     > > >     >     > support such
        >> > > >     > > > > >     > > >     >     >         >> > message. It would
        >> be
        >> > > good
        >> > > > to
        >> > > >     > know
        >> > > >     > > if
        >> > > >     > > > > > there is
        >> > > >     > > > > >     > a
        >> > > >     > > > > >     > > >     > concrete use
        >> > > >     > > > > >     > > >     >     > case for
        >> > > >     > > > > >     > > >     >     >         >> such
        >> > > >     > > > > >     > > >     >     >         >> > message. If there
        >> is
        >> > > not,
        >> > > > we can
        >> > > >     > > > > > probably just
        >> > > >     > > > > >     > > not
        >> > > >     > > > > >     > > >     > support it.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > Thanks,
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > JIangjie (Becket)
        >> Qin
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > On Wed, Nov 16,
        >> 2016
        >> > at
        >> > > > 1:28 PM,
        >> > > >     > > > > > Mayuresh
        >> > > >     > > > > >     > > Gharat <
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > > gharatmayuresh15@gmail.com
        >> > > >     > > > > >     > > >     >     >         >> > > wrote:
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >> > > Hi Ismael,
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > > This is
        >> something I
        >> > > can
        >> > > > think
        >> > > >     > of
        >> > > >     > > > for
        >> > > >     > > > > >     > migration
        >> > > >     > > > > >     > > > plan:
        >> > > >     > > > > >     > > >     >     >         >> > > So the migration
        >> > plan
        >> > > > can look
        >> > > >     > > > > > something
        >> > > >     > > > > >     > like
        >> > > >     > > > > >     > > > this,
        >> > > >     > > > > >     > > >     > with up
        >> > > >     > > > > >     > > >     >     >         >> conversion :
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > > 1) Currently lets
        >> > say
        >> > > > we have
        >> > > >     > > > Broker
        >> > > >     > > > > > at
        >> > > >     > > > > >     > > version
        >> > > >     > > > > >     > > > x.
        >> > > >     > > > > >     > > >     >     >         >> > > 2) Currently we
        >> have
        >> > > > clients
        >> > > >     > at
        >> > > >     > > > > > version x.
        >> > > >     > > > > >     > > >     >     >         >> > > 3) a) We move the
        >> > > > version to
        >> > > >     > > > > > Broker(x+1) :
        >> > > >     > > > > >     > > > supports
        >> > > >     > > > > >     > > >     > both
        >> > > >     > > > > >     > > >     >     > tombstone and
        >> > > >     > > > > >     > > >     >     >         >> > null
        >> > > >     > > > > >     > > >     >     >         >> > > for log
        >> compaction.
        >> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
        >> > the
        >> > > > client
        >> > > >     > to
        >> > > >     > > > > > version
        >> > > >     > > > > >     > > > client(x+1) :
        >> > > >     > > > > >     > > >     > if in
        >> > > >     > > > > >     > > >     >     > the
        >> > > >     > > > > >     > > >     >     >         >> producer
        >> > > >     > > > > >     > > >     >     >         >> > > client(x+1) the
        >> > value
        >> > > > is set
        >> > > >     > to
        >> > > >     > > > > null,
        >> > > >     > > > > > we
        >> > > >     > > > > >     > will
        >> > > >     > > > > >     > > >     > automatically
        >> > > >     > > > > >     > > >     >     > set the
        >> > > >     > > > > >     > > >     >     >         >> > > Tombstone bit
        >> > > > internally. If
        >> > > >     > the
        >> > > >     > > > > > producer
        >> > > >     > > > > >     > > > client(x+1)
        >> > > >     > > > > >     > > >     > sets
        >> > > >     > > > > >     > > >     >     > the
        >> > > >     > > > > >     > > >     >     >         >> tombstone
        >> > > >     > > > > >     > > >     >     >         >> > > itself, well and
        >> > good.
        >> > > > For
        >> > > >     > > > producer
        >> > > >     > > > > >     > client(x),
        >> > > >     > > > > >     > > > the
        >> > > >     > > > > >     > > >     > broker
        >> > > >     > > > > >     > > >     >     > will up
        >> > > >     > > > > >     > > >     >     >         >> convert
        >> > > >     > > > > >     > > >     >     >         >> > > to have the
        >> > tombstone
        >> > > > bit.
        >> > > >     > > > > > Broker(x+1) is
        >> > > >     > > > > >     > > > supporting
        >> > > >     > > > > >     > > >     > both.
        >> > > >     > > > > >     > > >     >     > Consumer
        >> > > >     > > > > >     > > >     >     >         >> > > client(x+1) will
        >> be
        >> > > > aware of
        >> > > >     > > this
        >> > > >     > > > > and
        >> > > >     > > > > > should
        >> > > >     > > > > >     > > be
        >> > > >     > > > > >     > > > able
        >> > > >     > > > > >     > > >     > to
        >> > > >     > > > > >     > > >     >     > handle this.
        >> > > >     > > > > >     > > >     >     >         >> For
        >> > > >     > > > > >     > > >     >     >         >> > > consumer
        >> client(x)
        >> > we
        >> > > > will
        >> > > >     > down
        >> > > >     > > > > > convert the
        >> > > >     > > > > >     > > > message
        >> > > >     > > > > >     > > >     > on the
        >> > > >     > > > > >     > > >     >     > broker
        >> > > >     > > > > >     > > >     >     >         >> side.
        >> > > >     > > > > >     > > >     >     >         >> > >     c) At this
        >> point
        >> > > we
        >> > > > will
        >> > > >     > > have
        >> > > >     > > > to
        >> > > >     > > > > >     > specify a
        >> > > >     > > > > >     > > >     > warning or
        >> > > >     > > > > >     > > >     >     > clearly
        >> > > >     > > > > >     > > >     >     >         >> specify
        >> > > >     > > > > >     > > >     >     >         >> > > in docs that this
        >> > > > behavior is
        >> > > >     > > > about
        >> > > >     > > > > > to be
        >> > > >     > > > > >     > > > changed for
        >> > > >     > > > > >     > > >     > log
        >> > > >     > > > > >     > > >     >     > compaction.
        >> > > >     > > > > >     > > >     >     >         >> > > 4) a) In next
        >> > release
        >> > > > of the
        >> > > >     > > > > > Broker(x+2), we
        >> > > >     > > > > >     > > > say that
        >> > > >     > > > > >     > > >     > only
        >> > > >     > > > > >     > > >     >     > Tombstone
        >> > > >     > > > > >     > > >     >     >         >> is
        >> > > >     > > > > >     > > >     >     >         >> > > used for log
        >> > > compaction
        >> > > > on the
        >> > > >     > > > > Broker
        >> > > >     > > > > > side.
        >> > > >     > > > > >     > > >     > Clients(x+1)
        >> > > >     > > > > >     > > >     >     > still is
        >> > > >     > > > > >     > > >     >     >         >> > > supported.
        >> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
        >> > the
        >> > > > client
        >> > > >     > to
        >> > > >     > > > > > version
        >> > > >     > > > > >     > > > client(x+2) :
        >> > > >     > > > > >     > > >     > if
        >> > > >     > > > > >     > > >     >     > value is set
        >> > > >     > > > > >     > > >     >     >         >> to
        >> > > >     > > > > >     > > >     >     >         >> > > null, tombstone
        >> will
        >> > > > not be
        >> > > >     > set
        >> > > >     > > > > >     > automatically.
        >> > > >     > > > > >     > > > The
        >> > > >     > > > > >     > > >     > client
        >> > > >     > > > > >     > > >     >     > will have to
        >> > > >     > > > > >     > > >     >     >         >> > call
        >> > > >     > > > > >     > > >     >     >         >> > > setTombstone() to
        >> > > > actually set
        >> > > >     > > the
        >> > > >     > > > > >     > tombstone.
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > > We should compare
        >> > this
        >> > > >     > migration
        >> > > >     > > > > plan
        >> > > >     > > > > > with
        >> > > >     > > > > >     > the
        >> > > >     > > > > >     > > >     > migration
        >> > > >     > > > > >     > > >     >     > plan for
        >> > > >     > > > > >     > > >     >     >         >> magic
        >> > > >     > > > > >     > > >     >     >         >> > > byte bump and do
        >> > > > whatever
        >> > > >     > looks
        >> > > >     > > > > good.
        >> > > >     > > > > >     > > >     >     >         >> > > I am just worried
        >> > that
        >> > > > if we
        >> > > >     > go
        >> > > >     > > > down
        >> > > >     > > > > > magic
        >> > > >     > > > > >     > > byte
        >> > > >     > > > > >     > > > route,
        >> > > >     > > > > >     > > >     >     > unless I am
        >> > > >     > > > > >     > > >     >     >         >> > missing
        >> > > >     > > > > >     > > >     >     >         >> > > something, it
        >> sounds
        >> > > > like
        >> > > >     > kafka
        >> > > >     > > > will
        >> > > >     > > > > > be
        >> > > >     > > > > >     > stuck
        >> > > >     > > > > >     > > > with
        >> > > >     > > > > >     > > >     >     > supporting both
        >> > > >     > > > > >     > > >     >     >         >> null
        >> > > >     > > > > >     > > >     >     >         >> > > value and
        >> tombstone
        >> > > bit
        >> > > > for
        >> > > >     > log
        >> > > >     > > > > > compaction
        >> > > >     > > > > >     > for
        >> > > >     > > > > >     > > > life
        >> > > >     > > > > >     > > >     > long,
        >> > > >     > > > > >     > > >     >     > which does
        >> > > >     > > > > >     > > >     >     >         >> not
        >> > > >     > > > > >     > > >     >     >         >> > > look like a good
        >> end
        >> > > > state.
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > > Thanks,
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > > Mayuresh
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16,
        >> 2016
        >> > > at
        >> > > > 9:32
        >> > > >     > AM,
        >> > > >     > > > > > Mayuresh
        >> > > >     > > > > >     > > > Gharat <
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > > gharatmayuresh15@gmail.com
        >> > > >     > > > > >     > > >     >     >         >> > > > wrote:
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > That's a very
        >> good
        >> > > > point
        >> > > >     > > which I
        >> > > >     > > > > > might
        >> > > >     > > > > >     > have
        >> > > >     > > > > >     > > > not
        >> > > >     > > > > >     > > >     >     > considered earlier.
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > Here is a plan
        >> > that
        >> > > I
        >> > > > can
        >> > > >     > > think
        >> > > >     > > > > of:
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > Stage 1) The
        >> > broker
        >> > > > from now
        >> > > >     > > on,
        >> > > >     > > > > up
        >> > > >     > > > > >     > converts
        >> > > >     > > > > >     > > > the
        >> > > >     > > > > >     > > >     > message
        >> > > >     > > > > >     > > >     >     > to have the
        >> > > >     > > > > >     > > >     >     >         >> > > > tombstone
        >> marker.
        >> > > The
        >> > > > log
        >> > > >     > > > > compaction
        >> > > >     > > > > >     > thread
        >> > > >     > > > > >     > > > does log
        >> > > >     > > > > >     > > >     >     > compaction
        >> > > >     > > > > >     > > >     >     >         >> based
        >> > > >     > > > > >     > > >     >     >         >> > on
        >> > > >     > > > > >     > > >     >     >         >> > > > both null and
        >> > > > tombstone
        >> > > >     > > marker.
        >> > > >     > > > > > This is
        >> > > >     > > > > >     > our
        >> > > >     > > > > >     > > >     > transition
        >> > > >     > > > > >     > > >     >     > period.
        >> > > >     > > > > >     > > >     >     >         >> > > > Stage 2) The
        >> next
        >> > > > release we
        >> > > >     > > > only
        >> > > >     > > > > > say that
        >> > > >     > > > > >     > > log
        >> > > >     > > > > >     > > >     > compaction
        >> > > >     > > > > >     > > >     >     > is based
        >> > > >     > > > > >     > > >     >     >         >> on
        >> > > >     > > > > >     > > >     >     >         >> > > > tombstone
        >> marker.
        >> > > > (Open
        >> > > >     > source
        >> > > >     > > > > > kafka makes
        >> > > >     > > > > >     > > > this as a
        >> > > >     > > > > >     > > >     >     > policy). By
        >> > > >     > > > > >     > > >     >     >         >> this
        >> > > >     > > > > >     > > >     >     >         >> > > time,
        >> > > >     > > > > >     > > >     >     >         >> > > > the
        >> organization
        >> > > > which is
        >> > > >     > > moving
        >> > > >     > > > > to
        >> > > >     > > > > > this
        >> > > >     > > > > >     > > > release
        >> > > >     > > > > >     > > >     > will be
        >> > > >     > > > > >     > > >     >     > sure that
        >> > > >     > > > > >     > > >     >     >         >> they
        >> > > >     > > > > >     > > >     >     >         >> > > > have gone
        >> through
        >> > > the
        >> > > > entire
        >> > > >     > > > > > transition
        >> > > >     > > > > >     > > > period.
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > My only goal of
        >> > > doing
        >> > > > this
        >> > > >     > is
        >> > > >     > > > that
        >> > > >     > > > > > Kafka
        >> > > >     > > > > >     > > > clearly
        >> > > >     > > > > >     > > >     >     > specifies the end
        >> > > >     > > > > >     > > >     >     >         >> > state
        >> > > >     > > > > >     > > >     >     >         >> > > > about what log
        >> > > > compaction
        >> > > >     > > means
        >> > > >     > > > > (is
        >> > > >     > > > > > it
        >> > > >     > > > > >     > null
        >> > > >     > > > > >     > > > value
        >> > > >     > > > > >     > > >     > or a
        >> > > >     > > > > >     > > >     >     > tombstone
        >> > > >     > > > > >     > > >     >     >         >> > marker,
        >> > > >     > > > > >     > > >     >     >         >> > > > but not both).
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > What do you
        >> think?
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > Thanks,
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh
        >> > > >     > > > > >     > > >     >     >         >> > > > .
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16,
        >> > 2016
        >> > > > at 9:17
        >> > > >     > > AM,
        >> > > >     > > > > > Ismael
        >> > > >     > > > > >     > > Juma <
        >> > > >     > > > > >     > > >     >     > ismael@juma.me.uk>
        >> > > >     > > > > >     > > >     >     >         >> > wrote:
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > >> One comment
        >> > below.
        >> > > >     > > > > >     > > >     >     >         >> > > >>
        >> > > >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov
        >> 16,
        >> > > 2016
        >> > > > at
        >> > > >     > 5:08
        >> > > >     > > > PM,
        >> > > >     > > > > > Mayuresh
        >> > > >     > > > > >     > > > Gharat <
        >> > > >     > > > > >     > > >     >     >         >> > > >>
        >> > > > gharatmayuresh15@gmail.com
        >> > > >     > > > > >     > > >     >     >         >> > > >> > wrote:
        >> > > >     > > > > >     > > >     >     >         >> > > >>
        >> > > >     > > > > >     > > >     >     >         >> > > >> >    - If we
        >> > don't
        >> > > > bump up
        >> > > >     > > the
        >> > > >     > > > > > magic
        >> > > >     > > > > >     > byte,
        >> > > >     > > > > >     > > > on the
        >> > > >     > > > > >     > > >     > broker
        >> > > >     > > > > >     > > >     >     > side, the
        >> > > >     > > > > >     > > >     >     >         >> > > broker
        >> > > >     > > > > >     > > >     >     >         >> > > >> >    will
        >> always
        >> > > > have to
        >> > > >     > look
        >> > > >     > > > at
        >> > > >     > > > > > both
        >> > > >     > > > > >     > > > tombstone
        >> > > >     > > > > >     > > >     > bit and
        >> > > >     > > > > >     > > >     >     > the value
        >> > > >     > > > > >     > > >     >     >         >> when
        >> > > >     > > > > >     > > >     >     >         >> > > do
        >> > > >     > > > > >     > > >     >     >         >> > > >> the
        >> > > >     > > > > >     > > >     >     >         >> > > >> >
        >> compaction.
        >> > > > Assuming
        >> > > >     > we
        >> > > >     > > do
        >> > > >     > > > > > not bump
        >> > > >     > > > > >     > up
        >> > > >     > > > > >     > > > the
        >> > > >     > > > > >     > > >     > magic
        >> > > >     > > > > >     > > >     >     > byte,
        >> > > >     > > > > >     > > >     >     >         >> > > >> >    imagine
        >> the
        >> > > > broker
        >> > > >     > sees
        >> > > >     > > a
        >> > > >     > > > > > message
        >> > > >     > > > > >     > > which
        >> > > >     > > > > >     > > > does
        >> > > >     > > > > >     > > >     > not
        >> > > >     > > > > >     > > >     >     > have a
        >> > > >     > > > > >     > > >     >     >         >> tombstone
        >> > > >     > > > > >     > > >     >     >         >> > > bit
        >> > > >     > > > > >     > > >     >     >         >> > > >> >    set. The
        >> > > broker
        >> > > > does
        >> > > >     > not
        >> > > >     > > > > know
        >> > > >     > > > > > when
        >> > > >     > > > > >     > the
        >> > > >     > > > > >     > > >     > message was
        >> > > >     > > > > >     > > >     >     > produced
        >> > > >     > > > > >     > > >     >     >         >> (i.e.
        >> > > >     > > > > >     > > >     >     >         >> > > >> > whether
        >> > > >     > > > > >     > > >     >     >         >> > > >> >    the
        >> message
        >> > > has
        >> > > > been
        >> > > >     > up
        >> > > >     > > > > > converted or
        >> > > >     > > > > >     > > > not), it
        >> > > >     > > > > >     > > >     > has
        >> > > >     > > > > >     > > >     >     > to take a
        >> > > >     > > > > >     > > >     >     >         >> > further
        >> > > >     > > > > >     > > >     >     >         >> > > >> > look at
        >> > > >     > > > > >     > > >     >     >         >> > > >> >    the
        >> value to
        >> > > > see if it
        >> > > >     > > is
        >> > > >     > > > > > null or
        >> > > >     > > > > >     > not
        >> > > >     > > > > >     > > in
        >> > > >     > > > > >     > > >     > order to
        >> > > >     > > > > >     > > >     >     > determine
        >> > > >     > > > > >     > > >     >     >         >> if it
        >> > > >     > > > > >     > > >     >     >         >> > > is
        >> > > >     > > > > >     > > >     >     >         >> > > >> a
        >> > > >     > > > > >     > > >     >     >         >> > > >> >
        >> tombstone.
        >> > The
        >> > > > same
        >> > > >     > > logic
        >> > > >     > > > > has
        >> > > >     > > > > > to be
        >> > > >     > > > > >     > > put
        >> > > >     > > > > >     > > > on the
        >> > > >     > > > > >     > > >     >     > consumer as
        >> > > >     > > > > >     > > >     >     >         >> well
        >> > > >     > > > > >     > > >     >     >         >> > > >> because
        >> > > >     > > > > >     > > >     >     >         >> > > >> > the
        >> > > >     > > > > >     > > >     >     >         >> > > >> >    consumer
        >> > does
        >> > > > not know
        >> > > >     > > if
        >> > > >     > > > > the
        >> > > >     > > > > >     > message
        >> > > >     > > > > >     > > > has
        >> > > >     > > > > >     > > >     > been up
        >> > > >     > > > > >     > > >     >     > converted or
        >> > > >     > > > > >     > > >     >     >         >> > not.
        >> > > >     > > > > >     > > >     >     >         >> > > >> >       - If
        >> we
        >> > > > upconvert
        >> > > >     > > while
        >> > > >     > > > > >     > appending,
        >> > > >     > > > > >     > > > this is
        >> > > >     > > > > >     > > >     > not
        >> > > >     > > > > >     > > >     >     > the case,
        >> > > >     > > > > >     > > >     >     >         >> > right?
        >> > > >     > > > > >     > > >     >     >         >> > > >>
        >> > > >     > > > > >     > > >     >     >         >> > > >>
        >> > > >     > > > > >     > > >     >     >         >> > > >> If I
        >> understand
        >> > you
        >> > > >     > > correctly,
        >> > > >     > > > > > this is
        >> > > >     > > > > >     > not
        >> > > >     > > > > >     > > >     > sufficient
        >> > > >     > > > > >     > > >     >     > because the
        >> > > >     > > > > >     > > >     >     >         >> log
        >> > > >     > > > > >     > > >     >     >         >> > > may
        >> > > >     > > > > >     > > >     >     >         >> > > >> have messages
        >> > > > appended
        >> > > >     > before
        >> > > >     > > > it
        >> > > >     > > > > > was
        >> > > >     > > > > >     > > > upgraded to
        >> > > >     > > > > >     > > >     > include
        >> > > >     > > > > >     > > >     >     > KIP-87.
        >> > > >     > > > > >     > > >     >     >         >> > > >>
        >> > > >     > > > > >     > > >     >     >         >> > > >> Ismael
        >> > > >     > > > > >     > > >     >     >         >> > > >>
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > > > --
        >> > > >     > > > > >     > > >     >     >         >> > > > -Regards,
        >> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh R.
        >> Gharat
        >> > > >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
        >> > > >     > > > > >     > > >     >     >         >> > > >
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > > --
        >> > > >     > > > > >     > > >     >     >         >> > > -Regards,
        >> > > >     > > > > >     > > >     >     >         >> > > Mayuresh R.
        >> Gharat
        >> > > >     > > > > >     > > >     >     >         >> > > (862) 250-7125
        >> > > >     > > > > >     > > >     >     >         >> > >
        >> > > >     > > > > >     > > >     >     >         >> > The information
        >> > > contained
        >> > > > in
        >> > > >     > this
        >> > > >     > > > > email
        >> > > >     > > > > > is
        >> > > >     > > > > >     > > > strictly
        >> > > >     > > > > >     > > >     >     > confidential and for
        >> > > >     > > > > >     > > >     >     >         >> > the use of the
        >> > addressee
        >> > > > only,
        >> > > >     > > > unless
        >> > > >     > > > > >     > otherwise
        >> > > >     > > > > >     > > >     > indicated. If
        >> > > >     > > > > >     > > >     >     > you are
        >> > > >     > > > > >     > > >     >     >         >> not
        >> > > >     > > > > >     > > >     >     >         >> > the intended
        >> > recipient,
        >> > > > please
        >> > > >     > do
        >> > > >     > > > not
        >> > > >     > > > > > read,
        >> > > >     > > > > >     > > copy,
        >> > > >     > > > > >     > > > use or
        >> > > >     > > > > >     > > >     >     > disclose to
        >> > > >     > > > > >     > > >     >     >         >> others
        >> > > >     > > > > >     > > >     >     >         >> > this message or any
        >> > > > attachment.
        >> > > >     > > > Please
        >> > > >     > > > > > also
        >> > > >     > > > > >     > > > notify the
        >> > > >     > > > > >     > > >     > sender
        >> > > >     > > > > >     > > >     >     > by
        >> > > >     > > > > >     > > >     >     >         >> replying
        >> > > >     > > > > >     > > >     >     >         >> > to this email or by
        >> > > > telephone
        >> > > >     > > > (+44(020
        >> > > >     > > > > > 7896
        >> > > >     > > > > >     > > 0011)
        >> > > >     > > > > >     > > > and
        >> > > >     > > > > >     > > >     > then
        >> > > >     > > > > >     > > >     >     > delete the
        >> > > >     > > > > >     > > >     >     >         >> email
        >> > > >     > > > > >     > > >     >     >         >> > and any copies of
        >> it.
        >> > > > Opinions,
        >> > > >     > > > > > conclusion
        >> > > >     > > > > >     > (etc)
        >> > > >     > > > > >     > > > that
        >> > > >     > > > > >     > > >     > do not
        >> > > >     > > > > >     > > >     >     > relate to
        >> > > >     > > > > >     > > >     >     >         >> the
        >> > > >     > > > > >     > > >     >     >         >> > official business
        >> of
        >> > > this
        >> > > >     > company
        >> > > >     > > > > shall
        >> > > >     > > > > > be
        >> > > >     > > > > >     > > > understood as
        >> > > >     > > > > >     > > >     >     > neither given
        >> > > >     > > > > >     > > >     >     >         >> nor
        >> > > >     > > > > >     > > >     >     >         >> > endorsed by it. IG
        >> is
        >> > a
        >> > > > trading
        >> > > >     > > name
        >> > > >     > > > > of
        >> > > >     > > > > > IG
        >> > > >     > > > > >     > > Markets
        >> > > >     > > > > >     > > >     > Limited (a
        >> > > >     > > > > >     > > >     >     > company
        >> > > >     > > > > >     > > >     >     >         >> > registered in
        >> England
        >> > > and
        >> > > > Wales,
        >> > > >     > > > > company
        >> > > >     > > > > >     > number
        >> > > >     > > > > >     > > >     > 04008957) and
        >> > > >     > > > > >     > > >     >     > IG Index
        >> > > >     > > > > >     > > >     >     >         >> > Limited (a company
        >> > > > registered in
        >> > > >     > > > > > England and
        >> > > >     > > > > >     > > > Wales,
        >> > > >     > > > > >     > > >     > company
        >> > > >     > > > > >     > > >     >     > number
        >> > > >     > > > > >     > > >     >     >         >> > 01190902).
        >> Registered
        >> > > > address at
        >> > > >     > > > > Cannon
        >> > > >     > > > > > Bridge
        >> > > >     > > > > >     > > > House, 25
        >> > > >     > > > > >     > > >     >     > Dowgate Hill,
        >> > > >     > > > > >     > > >     >     >         >> > London EC4R 2YA.
        >> Both
        >> > IG
        >> > > > Markets
        >> > > >     > > > > Limited
        >> > > >     > > > > >     > > (register
        >> > > >     > > > > >     > > >     > number
        >> > > >     > > > > >     > > >     >     > 195355) and IG
        >> > > >     > > > > >     > > >     >     >         >> > Index Limited
        >> > (register
        >> > > > number
        >> > > >     > > > 114059)
        >> > > >     > > > > > are
        >> > > >     > > > > >     > > > authorised
        >> > > >     > > > > >     > > >     > and
        >> > > >     > > > > >     > > >     >     > regulated by
        >> > > >     > > > > >     > > >     >     >         >> the
        >> > > >     > > > > >     > > >     >     >         >> > Financial Conduct
        >> > > > Authority.
        >> > > >     > > > > >     > > >     >     >         >> >
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >> --
        >> > > >     > > > > >     > > >     >     >         >> -Regards,
        >> > > >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
        >> > > >     > > > > >     > > >     >     >         >> (862) 250-7125
        >> > > >     > > > > >     > > >     >     >         >> The information
        >> > contained
        >> > > > in this
        >> > > >     > > > email
        >> > > >     > > > > is
        >> > > >     > > > > >     > > strictly
        >> > > >     > > > > >     > > >     >     > confidential and for
        >> > > >     > > > > >     > > >     >     >         >> the use of the
        >> addressee
        >> > > > only,
        >> > > >     > > unless
        >> > > >     > > > > > otherwise
        >> > > >     > > > > >     > > >     > indicated. If
        >> > > >     > > > > >     > > >     >     > you are not
        >> > > >     > > > > >     > > >     >     >         >> the intended
        >> recipient,
        >> > > > please do
        >> > > >     > > not
        >> > > >     > > > > > read,
        >> > > >     > > > > >     > copy,
        >> > > >     > > > > >     > > > use or
        >> > > >     > > > > >     > > >     >     > disclose to others
        >> > > >     > > > > >     > > >     >     >         >> this message or any
        >> > > > attachment.
        >> > > >     > > Please
        >> > > >     > > > > > also
        >> > > >     > > > > >     > notify
        >> > > >     > > > > >     > > > the
        >> > > >     > > > > >     > > >     > sender
        >> > > >     > > > > >     > > >     >     > by replying
        >> > > >     > > > > >     > > >     >     >         >> to this email or by
        >> > > > telephone
        >> > > >     > > (+44(020
        >> > > >     > > > > > 7896
        >> > > >     > > > > >     > 0011)
        >> > > >     > > > > >     > > > and then
        >> > > >     > > > > >     > > >     >     > delete the email
        >> > > >     > > > > >     > > >     >     >         >> and any copies of it.
        >> > > > Opinions,
        >> > > >     > > > > > conclusion (etc)
        >> > > >     > > > > >     > > > that do
        >> > > >     > > > > >     > > >     > not
        >> > > >     > > > > >     > > >     >     > relate to the
        >> > > >     > > > > >     > > >     >     >         >> official business of
        >> > this
        >> > > > company
        >> > > >     > > > shall
        >> > > >     > > > > be
        >> > > >     > > > > >     > > > understood as
        >> > > >     > > > > >     > > >     >     > neither given nor
        >> > > >     > > > > >     > > >     >     >         >> endorsed by it. IG
        >> is a
        >> > > > trading
        >> > > >     > name
        >> > > >     > > > of
        >> > > >     > > > > IG
        >> > > >     > > > > >     > Markets
        >> > > >     > > > > >     > > >     > Limited (a
        >> > > >     > > > > >     > > >     >     > company
        >> > > >     > > > > >     > > >     >     >         >> registered in England
        >> > and
        >> > > > Wales,
        >> > > >     > > > company
        >> > > >     > > > > > number
        >> > > >     > > > > >     > > > 04008957)
        >> > > >     > > > > >     > > >     > and
        >> > > >     > > > > >     > > >     >     > IG Index
        >> > > >     > > > > >     > > >     >     >         >> Limited (a company
        >> > > > registered in
        >> > > >     > > > England
        >> > > >     > > > > > and
        >> > > >     > > > > >     > > Wales,
        >> > > >     > > > > >     > > >     > company
        >> > > >     > > > > >     > > >     >     > number
        >> > > >     > > > > >     > > >     >     >         >> 01190902). Registered
        >> > > > address at
        >> > > >     > > > Cannon
        >> > > >     > > > > > Bridge
        >> > > >     > > > > >     > > > House, 25
        >> > > >     > > > > >     > > >     >     > Dowgate Hill,
        >> > > >     > > > > >     > > >     >     >         >> London EC4R 2YA.
        >> Both IG
        >> > > > Markets
        >> > > >     > > > Limited
        >> > > >     > > > > >     > (register
        >> > > >     > > > > >     > > > number
        >> > > >     > > > > >     > > >     >     > 195355) and IG
        >> > > >     > > > > >     > > >     >     >         >> Index Limited
        >> (register
        >> > > > number
        >> > > >     > > 114059)
        >> > > >     > > > > are
        >> > > >     > > > > >     > > > authorised and
        >> > > >     > > > > >     > > >     >     > regulated by the
        >> > > >     > > > > >     > > >     >     >         >> Financial Conduct
        >> > > Authority.
        >> > > >     > > > > >     > > >     >     >         >>
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >         > --
        >> > > >     > > > > >     > > >     >     >         > -Regards,
        >> > > >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
        >> > > >     > > > > >     > > >     >     >         > (862) 250-7125
        >> > > >     > > > > >     > > >     >     >         >
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >         --
        >> > > >     > > > > >     > > >     >     >         -Regards,
        >> > > >     > > > > >     > > >     >     >         Mayuresh R. Gharat
        >> > > >     > > > > >     > > >     >     >         (862) 250-7125
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >     The information contained in
        >> > this
        >> > > > email
        >> > > >     > is
        >> > > >     > > > > > strictly
        >> > > >     > > > > >     > > > confidential
        >> > > >     > > > > >     > > >     > and
        >> > > >     > > > > >     > > >     >     > for the use of the addressee
        >> only,
        >> > > > unless
        >> > > >     > > > otherwise
        >> > > >     > > > > >     > > indicated.
        >> > > >     > > > > >     > > > If
        >> > > >     > > > > >     > > >     > you are
        >> > > >     > > > > >     > > >     >     > not the intended recipient,
        >> please
        >> > do
        >> > > > not
        >> > > >     > read,
        >> > > >     > > > > > copy, use
        >> > > >     > > > > >     > or
        >> > > >     > > > > >     > > >     > disclose to
        >> > > >     > > > > >     > > >     >     > others this message or any
        >> > > attachment.
        >> > > > Please
        >> > > >     > > > also
        >> > > >     > > > > > notify
        >> > > >     > > > > >     > the
        >> > > >     > > > > >     > > > sender
        >> > > >     > > > > >     > > >     > by
        >> > > >     > > > > >     > > >     >     > replying to this email or by
        >> > > telephone
        >> > > >     > (+44(020
        >> > > >     > > > > 7896
        >> > > >     > > > > > 0011)
        >> > > >     > > > > >     > > and
        >> > > >     > > > > >     > > > then
        >> > > >     > > > > >     > > >     > delete
        >> > > >     > > > > >     > > >     >     > the email and any copies of it.
        >> > > > Opinions,
        >> > > >     > > > > conclusion
        >> > > >     > > > > > (etc)
        >> > > >     > > > > >     > > > that do
        >> > > >     > > > > >     > > >     > not
        >> > > >     > > > > >     > > >     >     > relate to the official business
        >> of
        >> > > this
        >> > > >     > company
        >> > > >     > > > > > shall be
        >> > > >     > > > > >     > > > understood
        >> > > >     > > > > >     > > >     > as
        >> > > >     > > > > >     > > >     >     > neither given nor endorsed by
        >> it.
        >> > IG
        >> > > > is a
        >> > > >     > > trading
        >> > > >     > > > > > name of
        >> > > >     > > > > >     > IG
        >> > > >     > > > > >     > > > Markets
        >> > > >     > > > > >     > > >     >     > Limited (a company registered in
        >> > > > England and
        >> > > >     > > > Wales,
        >> > > >     > > > > > company
        >> > > >     > > > > >     > > > number
        >> > > >     > > > > >     > > >     >     > 04008957) and IG Index Limited
        >> (a
        >> > > > company
        >> > > >     > > > > registered
        >> > > >     > > > > > in
        >> > > >     > > > > >     > > > England and
        >> > > >     > > > > >     > > >     > Wales,
        >> > > >     > > > > >     > > >     >     > company number 01190902).
        >> > Registered
        >> > > > address
        >> > > >     > at
        >> > > >     > > > > > Cannon
        >> > > >     > > > > >     > Bridge
        >> > > >     > > > > >     > > > House,
        >> > > >     > > > > >     > > >     > 25
        >> > > >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA.
        >> Both
        >> > > IG
        >> > > >     > Markets
        >> > > >     > > > > > Limited
        >> > > >     > > > > >     > > > (register
        >> > > >     > > > > >     > > >     > number
        >> > > >     > > > > >     > > >     >     > 195355) and IG Index Limited
        >> > > (register
        >> > > > number
        >> > > >     > > > > > 114059) are
        >> > > >     > > > > >     > > > authorised
        >> > > >     > > > > >     > > >     > and
        >> > > >     > > > > >     > > >     >     > regulated by the Financial
        >> Conduct
        >> > > > Authority.
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >     >
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >     > The information contained in this
        >> email
        >> > is
        >> > > > strictly
        >> > > >     > > > > > confidential
        >> > > >     > > > > >     > > and
        >> > > >     > > > > >     > > > for
        >> > > >     > > > > >     > > >     > the use of the addressee only, unless
        >> > > > otherwise
        >> > > >     > > > > indicated.
        >> > > >     > > > > > If you
        >> > > >     > > > > >     > > > are not
        >> > > >     > > > > >     > > >     > the intended recipient, please do not
        >> > read,
        >> > > > copy,
        >> > > >     > use
        >> > > >     > > > or
        >> > > >     > > > > > disclose
        >> > > >     > > > > >     > > to
        >> > > >     > > > > >     > > > others
        >> > > >     > > > > >     > > >     > this message or any attachment. Please
        >> > also
        >> > > > notify
        >> > > >     > > the
        >> > > >     > > > > > sender by
        >> > > >     > > > > >     > > > replying
        >> > > >     > > > > >     > > >     > to this email or by telephone (+44(020
        >> > 7896
        >> > > > 0011
        >> > > >     > <020%207896%200011>) and
        >> > > >     > > > > then
        >> > > >     > > > > > delete
        >> > > >     > > > > >     > > > the email
        >> > > >     > > > > >     > > >     > and any copies of it. Opinions,
        >> > conclusion
        >> > > > (etc)
        >> > > >     > that
        >> > > >     > > > do
        >> > > >     > > > > > not
        >> > > >     > > > > >     > relate
        >> > > >     > > > > >     > > > to the
        >> > > >     > > > > >     > > >     > official business of this company
        >> shall
        >> > be
        >> > > >     > understood
        >> > > >     > > > as
        >> > > >     > > > > > neither
        >> > > >     > > > > >     > > > given nor
        >> > > >     > > > > >     > > >     > endorsed by it. IG is a trading name
        >> of
        >> > IG
        >> > > > Markets
        >> > > >     > > > > Limited
        >> > > >     > > > > > (a
        >> > > >     > > > > >     > > company
        >> > > >     > > > > >     > > >     > registered in England and Wales,
        >> company
        >> > > > number
        >> > > >     > > > 04008957)
        >> > > >     > > > > > and IG
        >> > > >     > > > > >     > > > Index
        >> > > >     > > > > >     > > >     > Limited (a company registered in
        >> England
        >> > > and
        >> > > > Wales,
        >> > > >     > > > > company
        >> > > >     > > > > >     > number
        >> > > >     > > > > >     > > >     > 01190902). Registered address at
        >> Cannon
        >> > > > Bridge
        >> > > >     > House,
        >> > > >     > > > 25
        >> > > >     > > > > > Dowgate
        >> > > >     > > > > >     > > > Hill,
        >> > > >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets
        >> Limited
        >> > > > (register
        >> > > >     > > > number
        >> > > >     > > > > > 195355)
        >> > > >     > > > > >     > > > and IG
        >> > > >     > > > > >     > > >     > Index Limited (register number 114059)
        >> > are
        >> > > >     > authorised
        >> > > >     > > > and
        >> > > >     > > > > >     > regulated
        >> > > >     > > > > >     > > > by the
        >> > > >     > > > > >     > > >     > Financial Conduct Authority.
        >> > > >     > > > > >     > > >     >
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > > The information contained in this email is
        >> > > strictly
        >> > > >     > > > > confidential
        >> > > >     > > > > > and
        >> > > >     > > > > >     > for
        >> > > >     > > > > >     > > > the use of the addressee only, unless
        >> otherwise
        >> > > >     > indicated.
        >> > > >     > > If
        >> > > >     > > > > > you are
        >> > > >     > > > > >     > not
        >> > > >     > > > > >     > > > the intended recipient, please do not read,
        >> > copy,
        >> > > > use or
        >> > > >     > > > > > disclose to
        >> > > >     > > > > >     > > others
        >> > > >     > > > > >     > > > this message or any attachment. Please also
        >> > > notify
        >> > > > the
        >> > > >     > > sender
        >> > > >     > > > > by
        >> > > >     > > > > >     > replying
        >> > > >     > > > > >     > > > to this email or by telephone (+44(020 7896
        >> > 0011
        >> > > >     > <020%207896%200011>) and then
        >> > > >     > > > > > delete the
        >> > > >     > > > > >     > > email
        >> > > >     > > > > >     > > > and any copies of it. Opinions, conclusion
        >> > (etc)
        >> > > > that do
        >> > > >     > > not
        >> > > >     > > > > > relate to
        >> > > >     > > > > >     > > the
        >> > > >     > > > > >     > > > official business of this company shall be
        >> > > > understood as
        >> > > >     > > > > neither
        >> > > >     > > > > > given
        >> > > >     > > > > >     > > nor
        >> > > >     > > > > >     > > > endorsed by it. IG is a trading name of IG
        >> > > Markets
        >> > > >     > Limited
        >> > > >     > > (a
        >> > > >     > > > > > company
        >> > > >     > > > > >     > > > registered in England and Wales, company
        >> number
        >> > > > 04008957)
        >> > > >     > > and
        >> > > >     > > > > IG
        >> > > >     > > > > > Index
        >> > > >     > > > > >     > > > Limited (a company registered in England and
        >> > > Wales,
        >> > > >     > company
        >> > > >     > > > > > number
        >> > > >     > > > > >     > > > 01190902). Registered address at Cannon
        >> Bridge
        >> > > > House, 25
        >> > > >     > > > > Dowgate
        >> > > >     > > > > > Hill,
        >> > > >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
        >> > > (register
        >> > > > number
        >> > > >     > > > > > 195355) and
        >> > > >     > > > > >     > IG
        >> > > >     > > > > >     > > > Index Limited (register number 114059) are
        >> > > > authorised and
        >> > > >     > > > > > regulated by
        >> > > >     > > > > >     > > the
        >> > > >     > > > > >     > > > Financial Conduct Authority.
        >> > > >     > > > > >     > > >
        >> > > >     > > > > >     > > The information contained in this email is
        >> > strictly
        >> > > >     > > > confidential
        >> > > >     > > > > > and for
        >> > > >     > > > > >     > > the use of the addressee only, unless
        >> otherwise
        >> > > > indicated.
        >> > > >     > If
        >> > > >     > > > you
        >> > > >     > > > > > are not
        >> > > >     > > > > >     > > the intended recipient, please do not read,
        >> copy,
        >> > > > use or
        >> > > >     > > > disclose
        >> > > >     > > > > > to
        >> > > >     > > > > >     > others
        >> > > >     > > > > >     > > this message or any attachment. Please also
        >> > notify
        >> > > > the
        >> > > >     > sender
        >> > > >     > > > by
        >> > > >     > > > > > replying
        >> > > >     > > > > >     > > to this email or by telephone (+44(020 7896
        >> 0011
        >> > > >     > <020%207896%200011>) and then
        >> > > >     > > > delete
        >> > > >     > > > > > the
        >> > > >     > > > > >     > email
        >> > > >     > > > > >     > > and any copies of it. Opinions, conclusion
        >> (etc)
        >> > > > that do
        >> > > >     > not
        >> > > >     > > > > > relate to
        >> > > >     > > > > >     > the
        >> > > >     > > > > >     > > official business of this company shall be
        >> > > > understood as
        >> > > >     > > > neither
        >> > > >     > > > > > given
        >> > > >     > > > > >     > nor
        >> > > >     > > > > >     > > endorsed by it. IG is a trading name of IG
        >> > Markets
        >> > > > Limited
        >> > > >     > (a
        >> > > >     > > > > > company
        >> > > >     > > > > >     > > registered in England and Wales, company
        >> number
        >> > > > 04008957)
        >> > > >     > and
        >> > > >     > > > IG
        >> > > >     > > > > > Index
        >> > > >     > > > > >     > > Limited (a company registered in England and
        >> > Wales,
        >> > > > company
        >> > > >     > > > > number
        >> > > >     > > > > >     > > 01190902). Registered address at Cannon Bridge
        >> > > > House, 25
        >> > > >     > > > Dowgate
        >> > > >     > > > > > Hill,
        >> > > >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited
        >> > (register
        >> > > > number
        >> > > >     > > > 195355)
        >> > > >     > > > > > and IG
        >> > > >     > > > > >     > > Index Limited (register number 114059) are
        >> > > > authorised and
        >> > > >     > > > > > regulated by
        >> > > >     > > > > >     > the
        >> > > >     > > > > >     > > Financial Conduct Authority.
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > >
        >> > > >     > > > > >     > The information contained in this email is
        >> strictly
        >> > > >     > > confidential
        >> > > >     > > > > and
        >> > > >     > > > > > for
        >> > > >     > > > > >     > the use of the addressee only, unless otherwise
        >> > > > indicated. If
        >> > > >     > > you
        >> > > >     > > > > > are not
        >> > > >     > > > > >     > the intended recipient, please do not read,
        >> copy,
        >> > use
        >> > > > or
        >> > > >     > > disclose
        >> > > >     > > > > to
        >> > > >     > > > > > others
        >> > > >     > > > > >     > this message or any attachment. Please also
        >> notify
        >> > > the
        >> > > > sender
        >> > > >     > > by
        >> > > >     > > > > > replying
        >> > > >     > > > > >     > to this email or by telephone (+44(020 7896 0011
        >> > > >     > <020%207896%200011>) and then
        >> > > >     > > delete
        >> > > >     > > > > > the email
        >> > > >     > > > > >     > and any copies of it. Opinions, conclusion (etc)
        >> > that
        >> > > > do not
        >> > > >     > > > relate
        >> > > >     > > > > > to the
        >> > > >     > > > > >     > official business of this company shall be
        >> > understood
        >> > > > as
        >> > > >     > > neither
        >> > > >     > > > > > given nor
        >> > > >     > > > > >     > endorsed by it. IG is a trading name of IG
        >> Markets
        >> > > > Limited (a
        >> > > >     > > > > company
        >> > > >     > > > > >     > registered in England and Wales, company number
        >> > > > 04008957) and
        >> > > >     > > IG
        >> > > >     > > > > > Index
        >> > > >     > > > > >     > Limited (a company registered in England and
        >> Wales,
        >> > > > company
        >> > > >     > > > number
        >> > > >     > > > > >     > 01190902). Registered address at Cannon Bridge
        >> > House,
        >> > > > 25
        >> > > >     > > Dowgate
        >> > > >     > > > > > Hill,
        >> > > >     > > > > >     > London EC4R 2YA. Both IG Markets Limited
        >> (register
        >> > > > number
        >> > > >     > > 195355)
        >> > > >     > > > > > and IG
        >> > > >     > > > > >     > Index Limited (register number 114059) are
        >> > authorised
        >> > > > and
        >> > > >     > > > regulated
        >> > > >     > > > > > by the
        >> > > >     > > > > >     > Financial Conduct Authority.
        >> > > >     > > > > >     >
        >> > > >     > > > > >     >
        >> > > >     > > > > >
        >> > > >     > > > > >
        >> > > >     > > > > > The information contained in this email is strictly
        >> > > > confidential
        >> > > >     > and
        >> > > >     > > > for
        >> > > >     > > > > > the use of the addressee only, unless otherwise
        >> > indicated.
        >> > > > If you
        >> > > >     > are
        >> > > >     > > > not
        >> > > >     > > > > > the intended recipient, please do not read, copy, use
        >> or
        >> > > > disclose
        >> > > >     > to
        >> > > >     > > > > others
        >> > > >     > > > > > this message or any attachment. Please also notify the
        >> > > > sender by
        >> > > >     > > > replying
        >> > > >     > > > > > to this email or by telephone (+44(020 7896 0011
        >> > > >     > <020%207896%200011>) and then delete the
        >> > > >     > > > > email
        >> > > >     > > > > > and any copies of it. Opinions, conclusion (etc) that
        >> do
        >> > > not
        >> > > > relate
        >> > > >     > > to
        >> > > >     > > > > the
        >> > > >     > > > > > official business of this company shall be understood
        >> as
        >> > > > neither
        >> > > >     > > given
        >> > > >     > > > > nor
        >> > > >     > > > > > endorsed by it. IG is a trading name of IG Markets
        >> > Limited
        >> > > (a
        >> > > >     > company
        >> > > >     > > > > > registered in England and Wales, company number
        >> 04008957)
        >> > > > and IG
        >> > > >     > > Index
        >> > > >     > > > > > Limited (a company registered in England and Wales,
        >> > company
        >> > > > number
        >> > > >     > > > > > 01190902). Registered address at Cannon Bridge House,
        >> 25
        >> > > > Dowgate
        >> > > >     > > Hill,
        >> > > >     > > > > > London EC4R 2YA. Both IG Markets Limited (register
        >> number
        >> > > > 195355)
        >> > > >     > and
        >> > > >     > > > IG
        >> > > >     > > > > > Index Limited (register number 114059) are authorised
        >> and
        >> > > > regulated
        >> > > >     > > by
        >> > > >     > > > > the
        >> > > >     > > > > > Financial Conduct Authority.
        >> > > >     > > > > >
        >> > > >     > > > > The information contained in this email is strictly
        >> > > > confidential and
        >> > > >     > > for
        >> > > >     > > > > the use of the addressee only, unless otherwise
        >> indicated.
        >> > If
        >> > > > you are
        >> > > >     > > not
        >> > > >     > > > > the intended recipient, please do not read, copy, use or
        >> > > > disclose to
        >> > > >     > > > others
        >> > > >     > > > > this message or any attachment. Please also notify the
        >> > sender
        >> > > > by
        >> > > >     > > replying
        >> > > >     > > > > to this email or by telephone (+44(020 7896 0011
        >> > > > <020%207896%200011>)
        >> > > >     > and then delete the
        >> > > >     > > > email
        >> > > >     > > > > and any copies of it. Opinions, conclusion (etc) that do
        >> > not
        >> > > > relate
        >> > > >     > to
        >> > > >     > > > the
        >> > > >     > > > > official business of this company shall be understood as
        >> > > > neither
        >> > > >     > given
        >> > > >     > > > nor
        >> > > >     > > > > endorsed by it. IG is a trading name of IG Markets
        >> Limited
        >> > (a
        >> > > > company
        >> > > >     > > > > registered in England and Wales, company number
        >> 04008957)
        >> > and
        >> > > > IG
        >> > > >     > Index
        >> > > >     > > > > Limited (a company registered in England and Wales,
        >> company
        >> > > > number
        >> > > >     > > > > 01190902). Registered address at Cannon Bridge House, 25
        >> > > > Dowgate
        >> > > >     > Hill,
        >> > > >     > > > > London EC4R 2YA. Both IG Markets Limited (register
        >> number
        >> > > > 195355) and
        >> > > >     > > IG
        >> > > >     > > > > Index Limited (register number 114059) are authorised
        >> and
        >> > > > regulated
        >> > > >     > by
        >> > > >     > > > the
        >> > > >     > > > > Financial Conduct Authority.
        >> > > >     > > > >
        >> > > >     > > > The information contained in this email is strictly
        >> > > confidential
        >> > > > and
        >> > > >     > for
        >> > > >     > > > the use of the addressee only, unless otherwise
        >> indicated. If
        >> > > > you are
        >> > > >     > not
        >> > > >     > > > the intended recipient, please do not read, copy, use or
        >> > > > disclose to
        >> > > >     > > others
        >> > > >     > > > this message or any attachment. Please also notify the
        >> sender
        >> > > by
        >> > > >     > replying
        >> > > >     > > > to this email or by telephone (+44(020 7896 0011
        >> > > > <020%207896%200011>)
        >> > > >     > and then delete the
        >> > > >     > > email
        >> > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
        >> not
        >> > > > relate to
        >> > > >     > > the
        >> > > >     > > > official business of this company shall be understood as
        >> > > neither
        >> > > > given
        >> > > >     > > nor
        >> > > >     > > > endorsed by it. IG is a trading name of IG Markets
        >> Limited (a
        >> > > > company
        >> > > >     > > > registered in England and Wales, company number 04008957)
        >> and
        >> > > IG
        >> > > > Index
        >> > > >     > > > Limited (a company registered in England and Wales,
        >> company
        >> > > > number
        >> > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
        >> > > Dowgate
        >> > > > Hill,
        >> > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
        >> > > > 195355) and
        >> > > >     > IG
        >> > > >     > > > Index Limited (register number 114059) are authorised and
        >> > > > regulated by
        >> > > >     > > the
        >> > > >     > > > Financial Conduct Authority.
        >> > > >     > > >
        >> > > >     > > The information contained in this email is strictly
        >> > confidential
        >> > > > and for
        >> > > >     > > the use of the addressee only, unless otherwise indicated.
        >> If
        >> > you
        >> > > > are not
        >> > > >     > > the intended recipient, please do not read, copy, use or
        >> > disclose
        >> > > > to
        >> > > >     > others
        >> > > >     > > this message or any attachment. Please also notify the
        >> sender
        >> > by
        >> > > > replying
        >> > > >     > > to this email or by telephone (+44(020 7896 0011
        >> > > > <020%207896%200011>)
        >> > > >     > and then delete the email
        >> > > >     > > and any copies of it. Opinions, conclusion (etc) that do not
        >> > > > relate to
        >> > > >     > the
        >> > > >     > > official business of this company shall be understood as
        >> > neither
        >> > > > given
        >> > > >     > nor
        >> > > >     > > endorsed by it. IG is a trading name of IG Markets Limited
        >> (a
        >> > > > company
        >> > > >     > > registered in England and Wales, company number 04008957)
        >> and
        >> > IG
        >> > > > Index
        >> > > >     > > Limited (a company registered in England and Wales, company
        >> > > number
        >> > > >     > > 01190902). Registered address at Cannon Bridge House, 25
        >> > Dowgate
        >> > > > Hill,
        >> > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
        >> > 195355)
        >> > > > and IG
        >> > > >     > > Index Limited (register number 114059) are authorised and
        >> > > > regulated by
        >> > > >     > the
        >> > > >     > > Financial Conduct Authority.
        >> > > >     > >
        >> > > >     >
        >> > > >
        >> > > >
        >> > > > The information contained in this email is strictly confidential and
        >> > for
        >> > > > the use of the addressee only, unless otherwise indicated. If you
        >> are
        >> > not
        >> > > > the intended recipient, please do not read, copy, use or disclose to
        >> > > others
        >> > > > this message or any attachment. Please also notify the sender by
        >> > replying
        >> > > > to this email or by telephone (+44(020 7896 0011) and then delete
        >> the
        >> > > email
        >> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
        >> to
        >> > > the
        >> > > > official business of this company shall be understood as neither
        >> given
        >> > > nor
        >> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
        >> company
        >> > > > registered in England and Wales, company number 04008957) and IG
        >> Index
        >> > > > Limited (a company registered in England and Wales, company number
        >> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
        >> Hill,
        >> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
        >> and
        >> > IG
        >> > > > Index Limited (register number 114059) are authorised and regulated
        >> by
        >> > > the
        >> > > > Financial Conduct Authority.
        >> > > >
        >> > > The information contained in this email is strictly confidential and
        >> for
        >> > > the use of the addressee only, unless otherwise indicated. If you are
        >> not
        >> > > the intended recipient, please do not read, copy, use or disclose to
        >> > others
        >> > > this message or any attachment. Please also notify the sender by
        >> replying
        >> > > to this email or by telephone (+44(020 7896 0011) and then delete the
        >> > email
        >> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
        >> > the
        >> > > official business of this company shall be understood as neither given
        >> > nor
        >> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
        >> > > registered in England and Wales, company number 04008957) and IG Index
        >> > > Limited (a company registered in England and Wales, company number
        >> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
        >> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
        >> IG
        >> > > Index Limited (register number 114059) are authorised and regulated by
        >> > the
        >> > > Financial Conduct Authority.
        >> > >
        >> > The information contained in this email is strictly confidential and for
        >> > the use of the addressee only, unless otherwise indicated. If you are
        >> not
        >> > the intended recipient, please do not read, copy, use or disclose to
        >> others
        >> > this message or any attachment. Please also notify the sender by
        >> replying
        >> > to this email or by telephone (+44(020 7896 0011) and then delete the
        >> email
        >> > and any copies of it. Opinions, conclusion (etc) that do not relate to
        >> the
        >> > official business of this company shall be understood as neither given
        >> nor
        >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
        >> > registered in England and Wales, company number 04008957) and IG Index
        >> > Limited (a company registered in England and Wales, company number
        >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
        >> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
        >> > Index Limited (register number 114059) are authorised and regulated by
        >> the
        >> > Financial Conduct Authority.
        >> >
        >>
        >
        >
        The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.




The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

@Jay.

On KIP-82 are you and Jun now on board that Headers are the better of the two options/kips? 

If so could you reply on that thread this is the case.

We really would like to get KIP-82 going/moving on an active discussion around the last few implementation details, rather than where we’re at that seems were held up on getting consensus that the concept of KIP-82 is worth it. 

If we don’t push on this one. 

Either way we really need a solution of 82 or 87 Apr/May time release (or when ever the next one is after the imminent on in Feb)

Cheers
Mike

On 20/12/2016, 05:55, "Michael Pearce" <Mi...@ig.com> wrote:

    Agreed. As I said there is still a use case just I wouldn't be pushing for it, the. Need for it reduces for me.
    
    Sent using OWA for iPhone
    ________________________________________
    From: radai <ra...@gmail.com>
    Sent: Tuesday, December 20, 2016 3:51:33 AM
    To: dev@kafka.apache.org
    Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    
    i didnt mean to sound as insisting. what i actually mean is it would still
    be a valid issue but of much less concern.
    
    On Mon, Dec 19, 2016 at 7:50 PM, radai <ra...@gmail.com> wrote:
    
    > this kip fixes a "bug" (quirk?) that arises when people implement headers
    > "in V" (in the payload part of a message).
    > if you have proper headers you obviously dont need to to stick them in V
    > and so wont run into this, but its still a valid issue.
    >
    > On Mon, Dec 19, 2016 at 3:06 PM, Jay Kreps <ja...@confluent.io> wrote:
    >
    >> Makes sense!
    >>
    >> -Jay
    >>
    >> On Mon, Dec 19, 2016 at 2:40 PM, Michael Pearce <Mi...@ig.com>
    >> wrote:
    >>
    >> > Wow just read that def over tired. Hopefully it makes sense. Or you get
    >> > the gist at least.
    >> >
    >> > ________________________________________
    >> > From: Michael Pearce <Mi...@ig.com>
    >> > Sent: Monday, December 19, 2016 9:19:02 PM
    >> > To: dev@kafka.apache.org
    >> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >> >
    >> > Hi Jay,
    >> >
    >> > Agreed this stemmed as offshoot from KIP-82.
    >> >
    >> > Which our main driver for was to be able to have some headers for a null
    >> > value as such for our routing, audit, tracing and a few other bits which
    >> > currently we are forced to do with a message wrapper, if we all agreed
    >> on
    >> > KIP-82 that we need native headers and look to implement that the push
    >> for
    >> > this would dissipate.
    >> >
    >> > This KIP would allow for though one use case that comes to mind we could
    >> > see which is to have business data with a delete. Though as said this
    >> isn't
    >> > something we are pushing for think really we would have.
    >> >
    >> > As such in summary yes, if you want to fully support KIP-82 and we can
    >> get
    >> > that agreed in principle and a target release version, I think quite a
    >> few
    >> > guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.
    >> >
    >> > Cheers
    >> > Mike
    >> >
    >> > Sent using OWA for iPhone
    >> > ________________________________________
    >> > From: Jay Kreps <ja...@confluent.io>
    >> > Sent: Monday, December 19, 2016 8:51:23 PM
    >> > To: dev@kafka.apache.org
    >> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >> >
    >> > Hey Michael,
    >> >
    >> > Here is the compatibility concern I have:
    >> >
    >> >    1. You have a consumer app that relies on value == null to indicate a
    >> >    delete (current semantics).
    >> >    2. You upgrade Kafka and your clients.
    >> >    3. Some producer starts using the tombstone field in combination with
    >> >    non-null.
    >> >
    >> > I share Ismael's dislike of setting tombstones on records with null
    >> values.
    >> > This makes sense as a transitional state, but as an end state its a bit
    >> > weird. You'd expect to be able to mix null values and tombstones, and
    >> have
    >> > the null values remain and the tombstones get compacted. However what
    >> will
    >> > happen is both will be compacted and upon debugging this you'll learn
    >> that
    >> > we sometimes use null in the value to indicate tombstone. Ismael's
    >> solution
    >> > is a bigger compatibility break, though, so not sure if that is better.
    >> >
    >> > My other question is the relationship to KIP-82. My read is that this
    >> KIP
    >> > solves some but not all of the problems KIP-82 is intended for. KIP-82,
    >> on
    >> > the other hand, seems to address most of the motivating uses for this
    >> KIP.
    >> > The exception is maybe item (5) on the list where you want to
    >> simultaneous
    >> > delete and convey some information to subscribers, but I couldn't
    >> construct
    >> > a concrete examples for that one. Do we need to rationalize these two
    >> KIPs?
    >> > That is, do you still advocate this proposal if we do KIP-82 and vice
    >> > versa? As you may have noticed I'm somewhat emotionally invested in the
    >> > simplicity of the core data model, so my default position is let's try
    >> to
    >> > avoid stuffing more stuff in, but if we have to add stuff I like each of
    >> > these individually more than doing both. :-)
    >> >
    >> > -Jay
    >> >
    >> >
    >> >
    >> >
    >> > On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Michael.Pearce@ig.com
    >> >
    >> > wrote:
    >> >
    >> > > Hi Jay
    >> > >
    >> > > I disagree here that we are breaking any compatibility, we went
    >> through
    >> > > this on the discussion thread in fact with the help of that thread is
    >> how
    >> > > the kip came to the solution.
    >> > >
    >> > > Also on the supported combinations front you mention, we are not
    >> taking
    >> > > anything away afaik.
    >> > >
    >> > > Currently supported are only are:
    >> > > Null value = delete
    >> > > Non-null value = non delete
    >> > >
    >> > > With this kip we would support
    >> > > Null value + tombstone = delete
    >> > > Non null value + tombstone = delete
    >> > > Non null value + no tombstone = non delete
    >> > >
    >> > > As for the alternative idea, this is simply a new policy, how can
    >> there
    >> > be
    >> > > confusion here? For this policy it would be explicit that tombstone
    >> > marker
    >> > > would need to be set for a delete.
    >> > >
    >> > > I'm going to vent a little now as starting to get quite frustrated.
    >> > >
    >> > > We are going round in circles on kip-82 as per use cases there is now
    >> > many
    >> > > use cases, how many more are needed? just because confluent don't see
    >> > these
    >> > > doesn't mean they aren't real use cases other have, this is the point
    >> of
    >> > > the Apache foundation, it shouldn't be the view of just one
    >> organisation.
    >> > > It really is getting a feeling of the NIH syndrome. Rather than it
    >> being
    >> > > constructive on discussion of the implementation detail.
    >> > >
    >> > > kip-87 spawned from as on the kip call we all agreed this was needed.
    >> And
    >> > > would at least allow a custom wrapper be supported in a compacted
    >> topic,
    >> > > allowing meta data. Which again now I feel we are spinning wheels, and
    >> > > simply finding reasons not support it.
    >> > >
    >> > > Cheers
    >> > > Mike
    >> > >
    >> > >
    >> > >
    >> > > Sent using OWA for iPad
    >> > > ________________________________________
    >> > > From: Jay Kreps <ja...@confluent.io>
    >> > > Sent: Friday, December 16, 2016 7:09:23 PM
    >> > > To: dev@kafka.apache.org
    >> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >> > >
    >> > > Hey Michael,
    >> > >
    >> > > I do think it might have been better had we started with a separate
    >> > concept
    >> > > of null vs delete. Given that we are where we are, I'm not sure that
    >> the
    >> > > possible cures we explored so far are better than the disease.
    >> > >
    >> > > I apologize for coming to this late, but I didn't really understand
    >> the
    >> > > implications of the KIP and that we'd be breaking compatibility with
    >> > > existing apps until the vote had begun, and in my defense I'm not sure
    >> > the
    >> > > other folks voting did either.
    >> > >
    >> > > I think we all agree there are many existing apps that are built with
    >> the
    >> > > assumption of "null value non-tombstone" and it isn't possible to
    >> > > disambiguate these from tombstones on the producer. It isn't that
    >> anyone
    >> > is
    >> > > saying we have to support all four possibilities at once, it's that we
    >> > > simply can't orphan one of the existing combinations or our users will
    >> > eat
    >> > > us!
    >> > >
    >> > > If I've understood your alternate solution of adding another setting
    >> for
    >> > > compaction, I think this does fix the compatibility problem, but it
    >> adds
    >> > an
    >> > > odd mode the user has to add on all their topics. While the current
    >> state
    >> > > is easily explainable, the resulting state where the meaning of
    >> tombstone
    >> > > and null are overlapping and ambiguous and dependent on a compaction
    >> > > setting that could change out of band or not be in sync with your
    >> code in
    >> > > some environment seems worse to me then where we currently are. I
    >> think
    >> > the
    >> > > question is how would this combination be explained to users and does
    >> it
    >> > > make sense?
    >> > >
    >> > > -Jay
    >> > >
    >> > >
    >> > >
    >> > > On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <
    >> Michael.Pearce@ig.com>
    >> > > wrote:
    >> > >
    >> > > > Hi Chaps,
    >> > > >
    >> > > > Can we either get one more +1 binding (we have 2 already) on the
    >> > > existing?
    >> > > >
    >> > > > Or have some response on the below possible alternative. We are
    >> keen to
    >> > > > get working on this, so we make next feature release.
    >> > > >
    >> > > > Cheers
    >> > > > Mike
    >> > > >
    >> > > >
    >> > > > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com>
    >> wrote:
    >> > > >
    >> > > >     Hi Ismael
    >> > > >
    >> > > >     Did you see our email this morning, what's your thoughts on this
    >> > > > approach to instead we simply have a brand new policy?
    >> > > >
    >> > > >     Cheers
    >> > > >     Mike
    >> > > >
    >> > > >
    >> > > >     Sent using OWA for iPhone
    >> > > >     ________________________________________
    >> > > >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
    >> > > Juma <
    >> > > > ismael@juma.me.uk>
    >> > > >     Sent: Tuesday, December 13, 2016 11:30:05 AM
    >> > > >     To: dev@kafka.apache.org
    >> > > >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >> > > >
    >> > > >     Yes, this is actually tricky to do in a way where we both have
    >> the
    >> > > > desired
    >> > > >     semantics and maintain compatibility. When someone creates a
    >> > > >     `ProducerRecord` with a `null` value today, the producer doesn't
    >> > know
    >> > > > if
    >> > > >     it's meant to be a tombstone or not. For V3 messages, it's easy
    >> > when
    >> > > > the
    >> > > >     constructor that takes a tombstone is used. However, if any
    >> other
    >> > > >     constructor is used, it's not clear. A couple of options I can
    >> > think
    >> > > > of,
    >> > > >     none of them particularly nice:
    >> > > >
    >> > > >     1. Have a third state where tombstone = unknown and the broker
    >> > would
    >> > > > set
    >> > > >     the tombstone bit if the value was null and the topic was
    >> > compacted.
    >> > > > People
    >> > > >     that wanted to pass a non-null value for the tombstone would
    >> have
    >> > to
    >> > > > use
    >> > > >     the constructor that takes a tombstone. The drawbacks: third
    >> state
    >> > > for
    >> > > >     tombstone in message format, message conversion at the broker
    >> for a
    >> > > > common
    >> > > >     case.
    >> > > >
    >> > > >     2. Extend MetadataResponse to optionally include topic configs,
    >> > which
    >> > > > would
    >> > > >     make it possible for the producer to be smarter about setting
    >> the
    >> > > >     tombstone. It would only do it if a tombstone was not passed
    >> > > > explicitly,
    >> > > >     the value was null and the topic was compacted. The main
    >> drawback
    >> > is
    >> > > > that
    >> > > >     the producer would be getting a bit more data for each topic
    >> even
    >> > > > though it
    >> > > >     probably won't use it most of the time. Extending
    >> MetadataResponse
    >> > to
    >> > > >     return topic configs would be useful for other reasons as well,
    >> so
    >> > > that
    >> > > >     part seems OK.
    >> > > >
    >> > > >     In addition, for both proposals, we could consider adding
    >> warnings
    >> > to
    >> > > > the
    >> > > >     documentation that the behaviour of the constructors that don't
    >> > take
    >> > > a
    >> > > >     tombstone would change in the next major release so that
    >> tombstone
    >> > =
    >> > > > false.
    >> > > >     Not sure if this would be worth it though.
    >> > > >
    >> > > >     Ismael
    >> > > >
    >> > > >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
    >> > > > ewen@confluent.io>
    >> > > >     wrote:
    >> > > >
    >> > > >     > Michael,
    >> > > >     >
    >> > > >     > It kind of depends on how you want to interpret the tombstone
    >> > flag.
    >> > > > If it's
    >> > > >     > purely a producer-facing Kafka-level thing that we treat as
    >> > > internal
    >> > > > to the
    >> > > >     > broker and log cleaner once the record is sent, then your
    >> > approach
    >> > > > makes
    >> > > >     > sense. You're just moving copying the null-indicates-delete
    >> > > behavior
    >> > > > of the
    >> > > >     > old constructor into the tombstone flag.
    >> > > >     >
    >> > > >     > However, if you want this change to more generally decouple
    >> the
    >> > > idea
    >> > > > of
    >> > > >     > deletion and null values, then you are sometimes converting
    >> what
    >> > > > might be a
    >> > > >     > completely valid null value that doesn't indicate deletion
    >> into a
    >> > > >     > tombstone. Downstream applications could potentially handle
    >> these
    >> > > > cases
    >> > > >     > differently given the separation of deletion from value.
    >> > > >     >
    >> > > >     > I guess the question is if we want to try to support the
    >> latter
    >> > > even
    >> > > > for
    >> > > >     > topics where we have older produce requests. An example where
    >> > this
    >> > > > could
    >> > > >     > come up is in something like a CDC Connector. If we try to
    >> > support
    >> > > > the
    >> > > >     > semantic difference, a connector might write changes to Kafka
    >> > using
    >> > > > the
    >> > > >     > tombstone flag to indicate when a row was truly deleted (vs an
    >> > > > update that
    >> > > >     > sets it to null but still present; this probably makes more
    >> sense
    >> > > > for CDC
    >> > > >     > from document stores or extracting single columns). There are
    >> > > various
    >> > > >     > reasons we might want to maintain the full log and not turn
    >> > > > compaction on
    >> > > >     > (or just use a time-based retention policy), but downstream
    >> > > > applications
    >> > > >     > might care to know the difference between a delete and a null
    >> > > value.
    >> > > > In
    >> > > >     > fact both versions of the same log (compacted and
    >> time-retention)
    >> > > > could be
    >> > > >     > useful and I don't think it'll be uncommon to maintain both or
    >> > use
    >> > > > KIP-71
    >> > > >     > to maintain a hybrid compacted/retention topic.
    >> > > >     >
    >> > > >     > -Ewen
    >> > > >     >
    >> > > >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
    >> > > > Michael.Pearce@ig.com>
    >> > > >     > wrote:
    >> > > >     >
    >> > > >     > > Hi Jay,
    >> > > >     > >
    >> > > >     > > Why wouldn't that work, the tombstone value is only looked
    >> at
    >> > by
    >> > > > the
    >> > > >     > > broker, on a topic configured for compaction as such is
    >> benign
    >> > on
    >> > > > non
    >> > > >     > > compacted topics. This is as much as sending a null value
    >> > > currently
    >> > > >     > >
    >> > > >     > >
    >> > > >     > > Regards
    >> > > >     > > Mike
    >> > > >     > >
    >> > > >     > >
    >> > > >     > >
    >> > > >     > > Sent using OWA for iPhone
    >> > > >     > > ________________________________________
    >> > > >     > > From: Jay Kreps <ja...@confluent.io>
    >> > > >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
    >> > > >     > > To: dev@kafka.apache.org
    >> > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >> > > >     > >
    >> > > >     > > Hey Michael,
    >> > > >     > >
    >> > > >     > > I'm not quite sure that works as that would translate ALL
    >> null
    >> > > > values to
    >> > > >     > > tombstones, even for non-compacted topics that use null as
    >> an
    >> > > > acceptable
    >> > > >     > > value sent by the producer and expected by the consumer.
    >> > > >     > >
    >> > > >     > > -Jay
    >> > > >     > >
    >> > > >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
    >> > > > Michael.Pearce@ig.com>
    >> > > >     > > wrote:
    >> > > >     > >
    >> > > >     > > > Hi Ewen,
    >> > > >     > > >
    >> > > >     > > > I think the easiest way to show this is with code.
    >> > > >     > > >
    >> > > >     > > > As you can see we keep the existing behaviour for
    >> > code/binaries
    >> > > > calling
    >> > > >     > > > the pre-existing constructors, whereby if the value is
    >> null
    >> > the
    >> > > >     > tombstone
    >> > > >     > > > is set to true.
    >> > > >     > > >
    >> > > >     > > > Regards
    >> > > >     > > > Mike
    >> > > >     > > >
    >> > > >     > > >
    >> > > >     > > >
    >> > > >     > > >     /**
    >> > > >     > > >      * Creates a record with a specified timestamp to be
    >> sent
    >> > > to
    >> > > > a
    >> > > >     > > > specified topic and partition
    >> > > >     > > >      *
    >> > > >     > > >      * @param topic The topic the record will be appended
    >> to
    >> > > >     > > >      * @param partition The partition to which the record
    >> > > should
    >> > > > be
    >> > > >     > sent
    >> > > >     > > >      * @param timestamp The timestamp of the record
    >> > > >     > > >      * @param tombstone if the record should be treated
    >> as a
    >> > > > tombstone
    >> > > >     > if
    >> > > >     > > > the topic is compacted
    >> > > >     > > >      * @param key The key that will be included in the
    >> record
    >> > > >     > > >      * @param value The record contents
    >> > > >     > > >      */
    >> > > >     > > >     public ProducerRecord(String topic, Integer partition,
    >> > > > Boolean
    >> > > >     > > > tombstone, Long timestamp, K key, V value) {
    >> > > >     > > >         if (topic == null)
    >> > > >     > > >             throw new IllegalArgumentException("Topic
    >> cannot
    >> > > be
    >> > > >     > null.");
    >> > > >     > > >         if (timestamp != null && timestamp < 0)
    >> > > >     > > >             throw new IllegalArgumentException(
    >> > > >     > > >                     String.format("Invalid timestamp: %d.
    >> > > > Timestamp
    >> > > >     > > should
    >> > > >     > > > always be non-negative or null.", timestamp));
    >> > > >     > > >         if (partition != null && partition < 0)
    >> > > >     > > >             throw new IllegalArgumentException(
    >> > > >     > > >                     String.format("Invalid partition: %d.
    >> > > > Partition
    >> > > >     > > number
    >> > > >     > > > should always be non-negative or null.", partition));
    >> > > >     > > >         if (!tombstone && value == null){
    >> > > >     > > >             throw new IllegalArgumentException(
    >> > > >     > > >                     String.format("Tombstone must be true
    >> if
    >> > > null
    >> > > >     > > value"));
    >> > > >     > > >         }
    >> > > >     > > >         this.topic = topic;
    >> > > >     > > >         this.partition = partition;
    >> > > >     > > >         this.tombstone = tombstone;
    >> > > >     > > >         this.key = key;
    >> > > >     > > >         this.value = value;
    >> > > >     > > >         this.timestamp = timestamp;
    >> > > >     > > >     }
    >> > > >     > > >
    >> > > >     > > >     public ProducerRecord(String topic, Integer partition,
    >> > Long
    >> > > >     > > timestamp,
    >> > > >     > > > K key, V value) {
    >> > > >     > > >         this(topic, partition, value == null, timestamp,
    >> key,
    >> > > > value);
    >> > > >     > > >     }
    >> > > >     > > > ________________________________________
    >> > > >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
    >> > > >     > > > Sent: Sunday, December 11, 2016 5:45 AM
    >> > > >     > > > To: dev@kafka.apache.org
    >> > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >> > > >     > > >
    >> > > >     > > > It seemed like this was addressed in the migration
    >> section,
    >> > > > wasn't it?
    >> > > >     > > The
    >> > > >     > > > V2 vs V3 requests and the fact that the broker will
    >> downgrade
    >> > > the
    >> > > >     > message
    >> > > >     > > > format (losing zero copy) if you issues a V2 request to a
    >> > > broker
    >> > > > using
    >> > > >     > V3
    >> > > >     > > > format handles compatibility, does it not? The existing
    >> > > > consumers won't
    >> > > >     > > see
    >> > > >     > > > the extra metadata in the value, but they will get a null
    >> > > > instead and
    >> > > >     > > treat
    >> > > >     > > > it as a tombstone. Certainly there is a performance
    >> impact,
    >> > but
    >> > > > it
    >> > > >     > seemed
    >> > > >     > > > compatible.
    >> > > >     > > >
    >> > > >     > > > I'm worried about this though:
    >> > > >     > > >
    >> > > >     > > >
    >> > > >     > > >    - *NOTE *: With the new version of producer client
    >> using
    >> > > >     > > ProduceRequest
    >> > > >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not
    >> > set)
    >> > > > and
    >> > > >     > null
    >> > > >     > > > value
    >> > > >     > > >    should not be allowed as the older version of consumer
    >> > using
    >> > > >     > > > FetchRequest
    >> > > >     > > >    V2 will think of this as a tombstone when its actually
    >> > not.
    >> > > >     > > >
    >> > > >     > > >
    >> > > >     > > > Unless I'm misunderstanding, this ends up breaking binary
    >> > > > compatibility
    >> > > >     > > for
    >> > > >     > > > the Java API. It sounds like this suggests that passing a
    >> > null
    >> > > > value to
    >> > > >     > > the
    >> > > >     > > > existing ProducerRecord constructors would generate an
    >> > > exception
    >> > > > since
    >> > > >     > > you
    >> > > >     > > > didn't explicitly enable the tombstone (via whatever new
    >> > > > constructor is
    >> > > >     > > > provided). But that means you can't swap in a newer client
    >> > jar
    >> > > > without
    >> > > >     > > > recompiling and get the same behavior if your app deletes
    >> > keys
    >> > > > using
    >> > > >     > the
    >> > > >     > > > current approach because your app will start throwing
    >> > > > exceptions. Maybe
    >> > > >     > > > this is a tradeoff we're ok with, but we've tried pretty
    >> hard
    >> > > to
    >> > > > avoid
    >> > > >     > > > breaking compatibility like this so far -- it makes
    >> picking
    >> > up
    >> > > > bug
    >> > > >     > fixes
    >> > > >     > > in
    >> > > >     > > > the clients harder for users of frameworks that have to
    >> pin
    >> > to
    >> > > > earlier
    >> > > >     > > > default versions for compatibility.
    >> > > >     > > >
    >> > > >     > > > But then later the KIP says:
    >> > > >     > > >
    >> > > >     > > >
    >> > > >     > > >    - The old constructors for ProducerRecord without this
    >> > > > parameter
    >> > > >     > will
    >> > > >     > > be
    >> > > >     > > >    kept but updated so that their default behaviour if
    >> > setting
    >> > > > null
    >> > > >     > value
    >> > > >     > > > will
    >> > > >     > > >    be the tombstone will be set to true to keep existing
    >> > > > behaviour.
    >> > > >     > > >
    >> > > >     > > >
    >> > > >     > > > So maybe I am misinterpreting something.
    >> > > >     > > >
    >> > > >     > > > And just a nit re: motivation section, item 6 would be
    >> > clearer
    >> > > > for a
    >> > > >     > > union
    >> > > >     > > > schema with null (or optional schemas in other formats),
    >> e.g.
    >> > > > [null,
    >> > > >     > > > string], in which case losing the schema truly is losing
    >> > > > information
    >> > > >     > > > (whereas null is already the only valid value for a pure
    >> null
    >> > > > schema).
    >> > > >     > > >
    >> > > >     > > > -Ewen
    >> > > >     > > >
    >> > > >     > > >
    >> > > >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
    >> > > > Michael.Pearce@ig.com
    >> > > >     > >
    >> > > >     > > > wrote:
    >> > > >     > > >
    >> > > >     > > > > Hi Jay,
    >> > > >     > > > >
    >> > > >     > > > > Good point this detail is missing in the KIP write up.
    >> Ive
    >> > > > added this
    >> > > >     > > > now.
    >> > > >     > > > >
    >> > > >     > > > > Essentially simply just upgrading the clients we do not
    >> > > expect
    >> > > > any
    >> > > >     > > client
    >> > > >     > > > > app code change needed.
    >> > > >     > > > >
    >> > > >     > > > > Cheers
    >> > > >     > > > > Mike
    >> > > >     > > > > ________________________________________
    >> > > >     > > > > From: Jay Kreps <ja...@confluent.io>
    >> > > >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
    >> > > >     > > > > To: dev@kafka.apache.org
    >> > > >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
    >> Flag
    >> > > >     > > > >
    >> > > >     > > > > Michael,
    >> > > >     > > > >
    >> > > >     > > > > The compatibility section goes through the migration
    >> path,
    >> > > but
    >> > > > isn't
    >> > > >     > > the
    >> > > >     > > > > bigger compatibility issue with existing apps? There are
    >> > many
    >> > > >     > (probably
    >> > > >     > > > > thousands) of apps in production that use this feature
    >> and
    >> > > > send null
    >> > > >     > to
    >> > > >     > > > > mean delete. It seems like this would break
    >> compatibility
    >> > > with
    >> > > > them,
    >> > > >     > > and
    >> > > >     > > > > they would have to be rewritten to right?
    >> > > >     > > > >
    >> > > >     > > > > -Jay
    >> > > >     > > > >
    >> > > >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
    >> > > >     > Michael.Pearce@ig.com
    >> > > >     > > >
    >> > > >     > > > > wrote:
    >> > > >     > > > >
    >> > > >     > > > > > Hi Jun,
    >> > > >     > > > > >
    >> > > >     > > > > > 4) On v3 we honour the tombstone. As such we expect
    >> it to
    >> > > be
    >> > > > set
    >> > > >     > > > > correctly
    >> > > >     > > > > > as per the KIP.
    >> > > >     > > > > >
    >> > > >     > > > > > 4.1) why would we want to produce an error when its
    >> v3?
    >> > > This
    >> > > > is the
    >> > > >     > > > exact
    >> > > >     > > > > > purpose to support non-null tombstone's
    >> > > >     > > > > > 4.2) again here im unclear on the question on the v3,
    >> > > produce
    >> > > >     > request
    >> > > >     > > > we
    >> > > >     > > > > > expect the tombstone flag to be set correctly.
    >> > > >     > > > > >
    >> > > >     > > > > > 4.4) compaction only occurs on compacted topics, the
    >> bit
    >> > > > makes no
    >> > > >     > > > > > difference and not looked at on un-compacted
    >> (time/size
    >> > > based
    >> > > >     > > > eviction).
    >> > > >     > > > > >
    >> > > >     > > > > >
    >> > > >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io>
    >> > wrote:
    >> > > >     > > > > >
    >> > > >     > > > > >     Hi, Michael,
    >> > > >     > > > > >
    >> > > >     > > > > >     4. Then, I think I misunderstood this point. Could
    >> > you
    >> > > > document
    >> > > >     > > the
    >> > > >     > > > > >     following points in the wiki?
    >> > > >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
    >> > > > non-null
    >> > > >     > value,
    >> > > >     > > > > does
    >> > > >     > > > > > the
    >> > > >     > > > > >     send() get an error or does the producer
    >> > automatically
    >> > > > set the
    >> > > >     > > > value
    >> > > >     > > > > to
    >> > > >     > > > > >     null?
    >> > > >     > > > > >     4.2 If producer V3 doesn't set tombstone, but
    >> > provides
    >> > > a
    >> > > > null
    >> > > >     > > > value,
    >> > > >     > > > > > does
    >> > > >     > > > > >     the send() get an error or does the producer
    >> > > > automatically sets
    >> > > >     > > the
    >> > > >     > > > > >     tombstone?
    >> > > >     > > > > >     4.3 Does the broker only expect messages that (a)
    >> > have
    >> > > no
    >> > > >     > > tombstone
    >> > > >     > > > > and
    >> > > >     > > > > >     non-null value; (b) have tombstone and null value
    >> and
    >> > > > reject
    >> > > >     > the
    >> > > >     > > > > > messages
    >> > > >     > > > > >     with an error code otherwise?
    >> > > >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic
    >> is
    >> > > > compacted
    >> > > >     > on
    >> > > >     > > > > not?
    >> > > >     > > > > >
    >> > > >     > > > > >     Thanks,
    >> > > >     > > > > >
    >> > > >     > > > > >     Jun
    >> > > >     > > > > >
    >> > > >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
    >> > > >     > > > > Michael.Pearce@ig.com
    >> > > >     > > > > > >
    >> > > >     > > > > >     wrote:
    >> > > >     > > > > >
    >> > > >     > > > > >     > Not at all.  This only acts on compacted topics
    >> > just
    >> > > > as what
    >> > > >     > > > occurs
    >> > > >     > > > > > today
    >> > > >     > > > > >     >
    >> > > >     > > > > >     > Sent using OWA for iPhone
    >> > > >     > > > > >     > ________________________________________
    >> > > >     > > > > >     > From: Jun Rao <ju...@confluent.io>
    >> > > >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
    >> > > >     > > > > >     > To: dev@kafka.apache.org
    >> > > >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction
    >> > Tombstone
    >> > > > Flag
    >> > > >     > > > > >     >
    >> > > >     > > > > >     > Hi, Michael,
    >> > > >     > > > > >     >
    >> > > >     > > > > >     > 4. Hmm, does that mean the new client library
    >> can
    >> > > > never send
    >> > > >     > a
    >> > > >     > > > null
    >> > > >     > > > > > message
    >> > > >     > > > > >     > even to a regular topic? This seems like a
    >> change
    >> > of
    >> > > > the
    >> > > >     > > existing
    >> > > >     > > > > > behavior.
    >> > > >     > > > > >     >
    >> > > >     > > > > >     > Thanks,
    >> > > >     > > > > >     >
    >> > > >     > > > > >     > Jun
    >> > > >     > > > > >     >
    >> > > >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
    >> > > >     > > > > > Michael.Pearce@ig.com>
    >> > > >     > > > > >     > wrote:
    >> > > >     > > > > >     >
    >> > > >     > > > > >     > > Hi Jun,
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > Re 4) That's because we expect the tombstone
    >> > value
    >> > > > to be
    >> > > >     > set
    >> > > >     > > > > > correctly if
    >> > > >     > > > > >     > > message bit is 2, as such if an older client
    >> > sends
    >> > > > in on
    >> > > >     > old
    >> > > >     > > > > > message the
    >> > > >     > > > > >     > > message is upcast and the bit is set
    >> correctly.
    >> > And
    >> > > > such no
    >> > > >     > > > > longer
    >> > > >     > > > > > need
    >> > > >     > > > > >     > to
    >> > > >     > > > > >     > > check the value. Mayuresh can you confirm my
    >> > > > thinking and
    >> > > >     > > > > > understanding
    >> > > >     > > > > >     > of
    >> > > >     > > > > >     > > what we've discussed?
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > The second point I understand what you're
    >> getting
    >> > > at
    >> > > > now my
    >> > > >     > > > > > apologies.
    >> > > >     > > > > >     > Yes
    >> > > >     > > > > >     > > this makes sense to save on touching the
    >> message,
    >> > > if
    >> > > > we're
    >> > > >     > > the
    >> > > >     > > > > > only kip
    >> > > >     > > > > >     > > going in, in this release.
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > Cheers
    >> > > >     > > > > >     > > Mike
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > Sent using OWA for iPhone
    >> > > >     > > > > >     > > ________________________________________
    >> > > >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
    >> > > >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
    >> > > >     > > > > >     > > To: dev@kafka.apache.org
    >> > > >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
    >> > > > Tombstone Flag
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > Hi, Michael,
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > 4. Is this updated in the wiki? The text "If
    >> the
    >> > > > magic byte
    >> > > >     > > on
    >> > > >     > > > > > message is
    >> > > >     > > > > >     > > 2, the broker should use the tombstone bit for
    >> > log
    >> > > >     > > compaction."
    >> > > >     > > > > > doesn't
    >> > > >     > > > > >     > > seem to have changed.
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > 2. My point is that if we change the message
    >> > format
    >> > > > just
    >> > > >     > for
    >> > > >     > > > this
    >> > > >     > > > > > KIP, we
    >> > > >     > > > > >     > > should consider whether it's worth optimizing
    >> the
    >> > > > down
    >> > > >     > > > conversion
    >> > > >     > > > > > path
    >> > > >     > > > > >     > > (i.e., decide whether a conversion is needed
    >> by
    >> > > just
    >> > > >     > looking
    >> > > >     > > at
    >> > > >     > > > > the
    >> > > >     > > > > >     > > tombstone bit in the wrapper message) since
    >> > > > tombstone will
    >> > > >     > be
    >> > > >     > > > > used
    >> > > >     > > > > >     > rarely.
    >> > > >     > > > > >     > > However, if the message format change here is
    >> > > > combined with
    >> > > >     > > > other
    >> > > >     > > > > > KIPs,
    >> > > >     > > > > >     > > then this optimization likely won't be needed.
    >> > The
    >> > > > latter
    >> > > >     > > > > probably
    >> > > >     > > > > > makes
    >> > > >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do
    >> you
    >> > > > think?
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > Other than those, +1 from me,
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > Thanks,
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > Jun
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael
    >> Pearce <
    >> > > >     > > > > > Michael.Pearce@ig.com>
    >> > > >     > > > > >     > > wrote:
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > > > Hi Jun
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > do we have your vote on this now?
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > Any other concerns?
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > Cheers
    >> > > >     > > > > >     > > > Mike
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > Sent using OWA for iPhone
    >> > > >     > > > > >     > > > ________________________________________
    >> > > >     > > > > >     > > > From: Michael Pearce <Michael.Pearce@ig.com
    >> >
    >> > > >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
    >> > > >     > > > > >     > > > To: dev@kafka.apache.org
    >> > > >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
    >> > > > Tombstone
    >> > > >     > Flag
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > Hi Jun,
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > Have updated. Thanks again for the feedback.
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > Agree yes we should align up when it gets to
    >> > > that,
    >> > > > I
    >> > > >     > assume
    >> > > >     > > > > > you've
    >> > > >     > > > > >     > > flagged
    >> > > >     > > > > >     > > > the same in the other KIP?
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > I think for now let's get this KIP approved,
    >> > then
    >> > > > we can
    >> > > >     > > > start
    >> > > >     > > > > > the work
    >> > > >     > > > > >     > > to
    >> > > >     > > > > >     > > > get an initial PR, then we can discuss how
    >> to
    >> > > > align the
    >> > > >     > two
    >> > > >     > > > if
    >> > > >     > > > > > needed.
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > Cheers,
    >> > > >     > > > > >     > > > Mike
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
    >> > > jun@confluent.io>
    >> > > >     > wrote:
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >     Hi, Michael,
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >     For 2), this is fine. Could you update
    >> the
    >> > > KIP
    >> > > > wiki
    >> > > >     > to
    >> > > >     > > > make
    >> > > >     > > > > > this
    >> > > >     > > > > >     > and
    >> > > >     > > > > >     > > > other
    >> > > >     > > > > >     > > >     points clearer? Other than that, the KIP
    >> > > looks
    >> > > > good
    >> > > >     > to
    >> > > >     > > > me.
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >     An orthogonal thing is that there are
    >> other
    >> > > > KIPs such
    >> > > >     > > as
    >> > > >     > > > > > KIP-98
    >> > > >     > > > > >     > that
    >> > > >     > > > > >     > > > also
    >> > > >     > > > > >     > > >     intend to change the message format. If
    >> > they
    >> > > > all get
    >> > > >     > > > > > approved, we
    >> > > >     > > > > >     > > > should
    >> > > >     > > > > >     > > >     think about whether it's better to just
    >> > bump
    >> > > > up the
    >> > > >     > > magic
    >> > > >     > > > > > byte once
    >> > > >     > > > > >     > > to
    >> > > >     > > > > >     > > >     incorporate multiple format changes
    >> like we
    >> > > > did in
    >> > > >     > > > > > KIP-31/KIP-32.
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >     Thanks,
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >     Jun
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
    >> > > Pearce
    >> > > > <
    >> > > >     > > > > >     > > Michael.Pearce@ig.com>
    >> > > >     > > > > >     > > >     wrote:
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >     > Hi Jao
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     > Thanks for the response. Sorry for
    >> slow
    >> > > > reply, both
    >> > > >     > > > with
    >> > > >     > > > > > personal
    >> > > >     > > > > >     > > > sickness
    >> > > >     > > > > >     > > >     > and also battling some critical issues
    >> > > > encountered
    >> > > >     > > > since
    >> > > >     > > > > >     > upgrading
    >> > > >     > > > > >     > > to
    >> > > >     > > > > >     > > >     > 0.10.1.0
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     > 1) Thans for spotting, Document error
    >> > where
    >> > > > we
    >> > > >     > > branched
    >> > > >     > > > > > this KIP
    >> > > >     > > > > >     > > from
    >> > > >     > > > > >     > > >     > KIP-82, will get that removed.
    >> > > >     > > > > >     > > >     > 2) Intent is to do this just at the
    >> > record
    >> > > > message
    >> > > >     > > > level.
    >> > > >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure
    >> this
    >> > is
    >> > > >     > > corrected.
    >> > > >     > > > > >     > > >     > 4) As per discussion thread we will
    >> > support
    >> > > >     > > tombstone +
    >> > > >     > > > > > null
    >> > > >     > > > > >     > value,
    >> > > >     > > > > >     > > >     > tombstone + non null value, no
    >> tombstone
    >> > +
    >> > > > null
    >> > > >     > > value.
    >> > > >     > > > > >     > > >     > 5) I believe this was in the
    >> discussion
    >> > > > thread,
    >> > > >     > > > @Mayuresh
    >> > > >     > > > > > is this
    >> > > >     > > > > >     > > >     > something we've overlooked? I thought
    >> we
    >> > > > would down
    >> > > >     > > > > > convert and
    >> > > >     > > > > >     > > > remove the
    >> > > >     > > > > >     > > >     > value so the old consumer had existing
    >> > > > behavior, or
    >> > > >     > > is
    >> > > >     > > > > > there
    >> > > >     > > > > >     > > > something we
    >> > > >     > > > > >     > > >     > haven't thought about?
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     > Cheers
    >> > > >     > > > > >     > > >     > Mike
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
    >> > > > jun@confluent.io>
    >> > > >     > > > > wrote:
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     Hi, Michael,
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     Thanks for the KIP. A few comments
    >> > > below.
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     1. The message format change
    >> contains
    >> > > >     > > > "HeadersLength
    >> > > >     > > > > >     > Headers".
    >> > > >     > > > > >     > > > Is that
    >> > > >     > > > > >     > > >     >     intended?
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     2. For compressed messageset, is
    >> the
    >> > > > tombstone
    >> > > >     > > bit
    >> > > >     > > > > > only set
    >> > > >     > > > > >     > at
    >> > > >     > > > > >     > > > the
    >> > > >     > > > > >     > > >     > shallow
    >> > > >     > > > > >     > > >     >     level? Do we always leave that
    >> bit in
    >> > > the
    >> > > >     > wrapper
    >> > > >     > > > > > message
    >> > > >     > > > > >     > > unset?
    >> > > >     > > > > >     > > > An
    >> > > >     > > > > >     > > >     >     alternative is to set the
    >> tombstone
    >> > bit
    >> > > > in the
    >> > > >     > > > > wrapper
    >> > > >     > > > > > if at
    >> > > >     > > > > >     > > > least one
    >> > > >     > > > > >     > > >     >     inner message has the tombstone
    >> bit
    >> > > set.
    >> > > > This
    >> > > >     > > makes
    >> > > >     > > > > > things a
    >> > > >     > > > > >     > > bit
    >> > > >     > > > > >     > > > more
    >> > > >     > > > > >     > > >     >     complicated, but we could
    >> potentially
    >> > > > exploit
    >> > > >     > > that
    >> > > >     > > > > for
    >> > > >     > > > > >     > > > optimizing down
    >> > > >     > > > > >     > > >     >     conversion. For example, we only
    >> need
    >> > > to
    >> > > >     > convert
    >> > > >     > > > > > messages
    >> > > >     > > > > >     > with
    >> > > >     > > > > >     > > > magic 2
    >> > > >     > > > > >     > > >     > to
    >> > > >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone
    >> > bit
    >> > > > is set
    >> > > >     > > > > > (conversion is
    >> > > >     > > > > >     > > > always
    >> > > >     > > > > >     > > >     > needed
    >> > > >     > > > > >     > > >     >     from magic 2 to magic 0). Not
    >> sure if
    >> > > the
    >> > > >     > > > > optimization
    >> > > >     > > > > > is
    >> > > >     > > > > >     > worth
    >> > > >     > > > > >     > > > the
    >> > > >     > > > > >     > > >     >     complexity though.
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     3. The referencing of the new
    >> version
    >> > > of
    >> > > >     > > > > >     > > > ProducerRequest/FetchRequest
    >> > > >     > > > > >     > > >     > is
    >> > > >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
    >> > > > convention
    >> > > >     > > > starts
    >> > > >     > > > > at
    >> > > >     > > > > >     > version
    >> > > >     > > > > >     > > > at 0, I
    >> > > >     > > > > >     > > >     >     think the new version would be 3.
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     4. "If the magic byte on message
    >> is
    >> > 2,
    >> > > > the
    >> > > >     > broker
    >> > > >     > > > > > should use
    >> > > >     > > > > >     > > the
    >> > > >     > > > > >     > > >     > tombstone
    >> > > >     > > > > >     > > >     >     bit for log compaction." What
    >> about
    >> > > null
    >> > > > value?
    >> > > >     > > My
    >> > > >     > > > > >     > > understanding
    >> > > >     > > > > >     > > > is
    >> > > >     > > > > >     > > >     > that
    >> > > >     > > > > >     > > >     >     null value will be treated the
    >> same
    >> > as
    >> > > > setting
    >> > > >     > > the
    >> > > >     > > > > > tombstone
    >> > > >     > > > > >     > > bit.
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     5. For the migration path, it
    >> would
    >> > be
    >> > > > useful
    >> > > >     > to
    >> > > >     > > > > > describe the
    >> > > >     > > > > >     > > > down
    >> > > >     > > > > >     > > >     >     conversion path to consumers
    >> (i.e.,
    >> > > > brokers on
    >> > > >     > > > > message
    >> > > >     > > > > > format
    >> > > >     > > > > >     > > > 0.10.2
    >> > > >     > > > > >     > > >     > and
    >> > > >     > > > > >     > > >     >     consumers on older version).
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     Thanks,
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     Jun
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
    >> > > Michael
    >> > > >     > Pearce <
    >> > > >     > > > > >     > > > Michael.Pearce@ig.com
    >> > > >     > > > > >     > > >     > >
    >> > > >     > > > > >     > > >     >     wrote:
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >     > Hi All,
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     > We have been discussing in the
    >> > below
    >> > > > thread
    >> > > >     > and
    >> > > >     > > > > final
    >> > > >     > > > > >     > changes
    >> > > >     > > > > >     > > > have
    >> > > >     > > > > >     > > >     > been
    >> > > >     > > > > >     > > >     >     > made to the KIP wiki based on
    >> these
    >> > > >     > > discussions.
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     > We would now like to put to the
    >> > vote
    >> > > > the
    >> > > >     > > > following
    >> > > >     > > > > > KIP:
    >> > > >     > > > > >     > > >     >     > https://cwiki.apache.org/
    >> > > >     > > > > > confluence/display/KAFKA/KIP-
    >> > > >     > > > > >     > 87+-+
    >> > > >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     > This kip is for having a
    >> distinct
    >> > > > compaction
    >> > > >     > > > > > attribute
    >> > > >     > > > > >     > > > "tombstone"
    >> > > >     > > > > >     > > >     > flag
    >> > > >     > > > > >     > > >     >     > instead of relying on null
    >> value,
    >> > > > allowing
    >> > > >     > > > non-null
    >> > > >     > > > > > value
    >> > > >     > > > > >     > > > delete
    >> > > >     > > > > >     > > >     > messages.
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     > Many thanks,
    >> > > >     > > > > >     > > >     >     > Michael
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
    >> > > Pearce"
    >> > > > <
    >> > > >     > > > > >     > > Michael.Pearce@ig.com>
    >> > > >     > > > > >     > > >     > wrote:
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >     Hi Mayuresh,
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >     LGTM. Ive just made one
    >> small
    >> > > > adjustment
    >> > > >     > > > > > updating the
    >> > > >     > > > > >     > > wire
    >> > > >     > > > > >     > > >     > protocol to
    >> > > >     > > > > >     > > >     >     > show the magic byte bump.
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >     Do we think we're good to
    >> put
    >> > to
    >> > > a
    >> > > > vote?
    >> > > >     > Is
    >> > > >     > > > > > there any
    >> > > >     > > > > >     > > > other bits
    >> > > >     > > > > >     > > >     >     > needing discussion?
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >     Cheers
    >> > > >     > > > > >     > > >     >     >     Mike
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >     On 21/11/2016, 18:26,
    >> "Mayuresh
    >> > > > Gharat" <
    >> > > >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
    >> > > >     > > > > >     > > >     >     > wrote:
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >         Hi Michael,
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >         I have updated the
    >> > migration
    >> > > > section
    >> > > >     > of
    >> > > >     > > > the
    >> > > >     > > > > > KIP.
    >> > > >     > > > > >     > Can
    >> > > >     > > > > >     > > > you
    >> > > >     > > > > >     > > >     > please
    >> > > >     > > > > >     > > >     >     > take a look?
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >         Thanks,
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >         Mayuresh
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at
    >> > 9:07
    >> > > > AM,
    >> > > >     > > Mayuresh
    >> > > >     > > > > > Gharat <
    >> > > >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
    >> > > >     > > > > >     > > >     >     >         > wrote:
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >         > Hi Michael,
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         > That whilst sending
    >> > > > tombstone and
    >> > > >     > non
    >> > > >     > > > > null
    >> > > >     > > > > > value,
    >> > > >     > > > > >     > > the
    >> > > >     > > > > >     > > >     > consumer
    >> > > >     > > > > >     > > >     >     > can expect
    >> > > >     > > > > >     > > >     >     >         > only to receive the
    >> > > non-null
    >> > > >     > message
    >> > > >     > > > only
    >> > > >     > > > > > in step
    >> > > >     > > > > >     > > > (3) is
    >> > > >     > > > > >     > > >     > this
    >> > > >     > > > > >     > > >     >     > correct?
    >> > > >     > > > > >     > > >     >     >         > ---> I do agree with
    >> you
    >> > > > here.
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         > Becket, Ismael : can
    >> you
    >> > > guys
    >> > > >     > review
    >> > > >     > > > the
    >> > > >     > > > > >     > migration
    >> > > >     > > > > >     > > > plan
    >> > > >     > > > > >     > > >     > listed
    >> > > >     > > > > >     > > >     >     > above using
    >> > > >     > > > > >     > > >     >     >         > magic byte?
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         > Thanks,
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         > Mayuresh
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016
    >> at
    >> > > 8:58
    >> > > > AM,
    >> > > >     > > > Michael
    >> > > >     > > > > > Pearce <
    >> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
    >> > > >     > > > > >     > > >     >     >         > wrote:
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         >> Many thanks for this
    >> > > > Mayuresh. I
    >> > > >     > > don't
    >> > > >     > > > > > have any
    >> > > >     > > > > >     > > >     > objections.
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> I assume we should
    >> > state:
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> That whilst sending
    >> > > > tombstone and
    >> > > >     > > non
    >> > > >     > > > > null
    >> > > >     > > > > >     > value,
    >> > > >     > > > > >     > > > the
    >> > > >     > > > > >     > > >     > consumer
    >> > > >     > > > > >     > > >     >     > can expect
    >> > > >     > > > > >     > > >     >     >         >> only to receive the
    >> > > non-null
    >> > > >     > message
    >> > > >     > > > > only
    >> > > >     > > > > > in
    >> > > >     > > > > >     > step
    >> > > >     > > > > >     > > > (3) is
    >> > > >     > > > > >     > > >     > this
    >> > > >     > > > > >     > > >     >     > correct?
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Cheers
    >> > > >     > > > > >     > > >     >     >         >> Mike
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Sent using OWA for
    >> > iPhone
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > > ______________________________
    >> > > >     > > > > __________
    >> > > >     > > > > >     > > >     >     >         >> From: Mayuresh
    >> Gharat <
    >> > > >     > > > > >     > gharatmayuresh15@gmail.com
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >     >     >         >> Sent: Thursday,
    >> November
    >> > > > 17, 2016
    >> > > >     > > > > 5:18:41
    >> > > >     > > > > > PM
    >> > > >     > > > > >     > > >     >     >         >> To:
    >> > dev@kafka.apache.org
    >> > > >     > > > > >     > > >     >     >         >> Subject: Re:
    >> [DISCUSS]
    >> > > > KIP-87 -
    >> > > >     > Add
    >> > > >     > > > > > Compaction
    >> > > >     > > > > >     > > > Tombstone
    >> > > >     > > > > >     > > >     > Flag
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Hi Ismael,
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Thanks for the
    >> > > explanation.
    >> > > >     > > > > >     > > >     >     >         >> Specially I like this
    >> > part
    >> > > > where
    >> > > >     > in
    >> > > >     > > > you
    >> > > >     > > > > >     > mentioned
    >> > > >     > > > > >     > > > we can
    >> > > >     > > > > >     > > >     > get
    >> > > >     > > > > >     > > >     >     > rid of the
    >> > > >     > > > > >     > > >     >     >         >> older null value
    >> support
    >> > > > for log
    >> > > >     > > > > > compaction
    >> > > >     > > > > >     > later
    >> > > >     > > > > >     > > > on,
    >> > > >     > > > > >     > > >     > here :
    >> > > >     > > > > >     > > >     >     >         >> We can't change
    >> > semantics
    >> > > > of the
    >> > > >     > > > message
    >> > > >     > > > > > format
    >> > > >     > > > > >     > > > without
    >> > > >     > > > > >     > > >     > having
    >> > > >     > > > > >     > > >     >     > a long
    >> > > >     > > > > >     > > >     >     >         >> transition period.
    >> And
    >> > we
    >> > > > can't
    >> > > >     > rely
    >> > > >     > > > > >     > > >     >     >         >> on people reading
    >> > > > documentation or
    >> > > >     > > > > acting
    >> > > >     > > > > > on a
    >> > > >     > > > > >     > > > warning for
    >> > > >     > > > > >     > > >     >     > something so
    >> > > >     > > > > >     > > >     >     >         >> fundamental. As
    >> such, my
    >> > > > take is
    >> > > >     > > that
    >> > > >     > > > we
    >> > > >     > > > > > need to
    >> > > >     > > > > >     > > > bump the
    >> > > >     > > > > >     > > >     > magic
    >> > > >     > > > > >     > > >     >     > byte. The
    >> > > >     > > > > >     > > >     >     >         >> good news is
    >> > > >     > > > > >     > > >     >     >         >> that we don't have to
    >> > > > support all
    >> > > >     > > > > versions
    >> > > >     > > > > >     > > forever.
    >> > > >     > > > > >     > > > We
    >> > > >     > > > > >     > > >     > have
    >> > > >     > > > > >     > > >     >     > said that we
    >> > > >     > > > > >     > > >     >     >         >> will support direct
    >> > > > upgrades for 2
    >> > > >     > > > > years.
    >> > > >     > > > > > That
    >> > > >     > > > > >     > > > means that
    >> > > >     > > > > >     > > >     >     > message format
    >> > > >     > > > > >     > > >     >     >         >> version n could, in
    >> > > theory,
    >> > > > be
    >> > > >     > > > removed 2
    >> > > >     > > > > > years
    >> > > >     > > > > >     > > > after the
    >> > > >     > > > > >     > > >     > it's
    >> > > >     > > > > >     > > >     >     > introduced.
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Just a heads up, I
    >> would
    >> > > > like to
    >> > > >     > > > mention
    >> > > >     > > > > > that
    >> > > >     > > > > >     > even
    >> > > >     > > > > >     > > > without
    >> > > >     > > > > >     > > >     >     > bumping magic
    >> > > >     > > > > >     > > >     >     >         >> byte, we will *NOT*
    >> > loose
    >> > > > zero
    >> > > >     > copy
    >> > > >     > > as
    >> > > >     > > > > in
    >> > > >     > > > > > the
    >> > > >     > > > > >     > > > client(x+1)
    >> > > >     > > > > >     > > >     > in my
    >> > > >     > > > > >     > > >     >     >         >> explanation
    >> > > >     > > > > >     > > >     >     >         >> above will convert
    >> > > > internally a
    >> > > >     > null
    >> > > >     > > > > > value to
    >> > > >     > > > > >     > > have a
    >> > > >     > > > > >     > > >     > tombstone
    >> > > >     > > > > >     > > >     >     > bit set and
    >> > > >     > > > > >     > > >     >     >         >> a tombstone bit set
    >> to
    >> > > have
    >> > > > a null
    >> > > >     > > > value
    >> > > >     > > > > >     > > > automatically
    >> > > >     > > > > >     > > >     >     > internally and by
    >> > > >     > > > > >     > > >     >     >         >> the time we move to
    >> > > version
    >> > > > (x+2),
    >> > > >     > > the
    >> > > >     > > > > > clients
    >> > > >     > > > > >     > > > would have
    >> > > >     > > > > >     > > >     >     > upgraded.
    >> > > >     > > > > >     > > >     >     >         >> Obviously if we
    >> support
    >> > a
    >> > > > request
    >> > > >     > > from
    >> > > >     > > > > >     > > consumer(x),
    >> > > >     > > > > >     > > > we
    >> > > >     > > > > >     > > >     > will
    >> > > >     > > > > >     > > >     >     > loose zero
    >> > > >     > > > > >     > > >     >     >         >> copy
    >> > > >     > > > > >     > > >     >     >         >> but that is the same
    >> > case
    >> > > > with
    >> > > >     > magic
    >> > > >     > > > > byte.
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> But if magic byte
    >> bump
    >> > > > makes life
    >> > > >     > > > easier
    >> > > >     > > > > > for
    >> > > >     > > > > >     > > > transition
    >> > > >     > > > > >     > > >     > for the
    >> > > >     > > > > >     > > >     >     > above
    >> > > >     > > > > >     > > >     >     >         >> reasons that you
    >> > > explained,
    >> > > > I am
    >> > > >     > OK
    >> > > >     > > > with
    >> > > >     > > > > > it
    >> > > >     > > > > >     > since
    >> > > >     > > > > >     > > > we are
    >> > > >     > > > > >     > > >     > going
    >> > > >     > > > > >     > > >     >     > to meet the
    >> > > >     > > > > >     > > >     >     >         >> end goal down the
    >> road
    >> > :)
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> On a side note can we
    >> > > > update the
    >> > > >     > doc
    >> > > >     > > > > here
    >> > > >     > > > > > on
    >> > > >     > > > > >     > magic
    >> > > >     > > > > >     > > > byte
    >> > > >     > > > > >     > > >     > to say
    >> > > >     > > > > >     > > >     >     > that "*it
    >> > > >     > > > > >     > > >     >     >         >> should be bumped
    >> > whenever
    >> > > > the
    >> > > >     > > message
    >> > > >     > > > > > format is
    >> > > >     > > > > >     > > > changed
    >> > > >     > > > > >     > > >     > or the
    >> > > >     > > > > >     > > >     >     >         >> interpretation of
    >> > message
    >> > > > format
    >> > > >     > > > (usage
    >> > > >     > > > > > of the
    >> > > >     > > > > >     > > > reserved
    >> > > >     > > > > >     > > >     > bits as
    >> > > >     > > > > >     > > >     >     > well) is
    >> > > >     > > > > >     > > >     >     >         >> changed*".
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Hi Michael,
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Here is the update
    >> plan
    >> > > > that we
    >> > > >     > > > > discussed
    >> > > >     > > > > >     > offline
    >> > > >     > > > > >     > > >     > yesterday :
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Currently the
    >> magic-byte
    >> > > > which
    >> > > >     > > > > > corresponds to
    >> > > >     > > > > >     > the
    >> > > >     > > > > >     > > >     >     > "message.format.version"
    >> > > >     > > > > >     > > >     >     >         >> is set to 1.
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> 1) On broker it will
    >> be
    >> > > set
    >> > > > to 1
    >> > > >     > > > > > initially.
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> 2) When a producer
    >> > client
    >> > > > sends a
    >> > > >     > > > > message
    >> > > >     > > > > > with
    >> > > >     > > > > >     > > > magic-byte
    >> > > >     > > > > >     > > >     > = 2,
    >> > > >     > > > > >     > > >     >     > since the
    >> > > >     > > > > >     > > >     >     >         >> broker is on
    >> magic-byte
    >> > =
    >> > > > 1, we
    >> > > >     > will
    >> > > >     > > > > down
    >> > > >     > > > > >     > convert
    >> > > >     > > > > >     > > > it,
    >> > > >     > > > > >     > > >     > which
    >> > > >     > > > > >     > > >     >     > means if the
    >> > > >     > > > > >     > > >     >     >         >> tombstone bit is set,
    >> > the
    >> > > > value
    >> > > >     > will
    >> > > >     > > > be
    >> > > >     > > > > > set to
    >> > > >     > > > > >     > > > null. A
    >> > > >     > > > > >     > > >     > consumer
    >> > > >     > > > > >     > > >     >     >         >> understanding
    >> > magic-byte =
    >> > > > 1, will
    >> > > >     > > > still
    >> > > >     > > > > > work
    >> > > >     > > > > >     > with
    >> > > >     > > > > >     > > > this. A
    >> > > >     > > > > >     > > >     >     > consumer
    >> > > >     > > > > >     > > >     >     >         >> working
    >> > > >     > > > > >     > > >     >     >         >> with magic-byte =2
    >> will
    >> > > > also be
    >> > > >     > able
    >> > > >     > > > to
    >> > > >     > > > > >     > understand
    >> > > >     > > > > >     > > > this,
    >> > > >     > > > > >     > > >     > since
    >> > > >     > > > > >     > > >     >     > it
    >> > > >     > > > > >     > > >     >     >         >> understands the
    >> > tombstone.
    >> > > >     > > > > >     > > >     >     >         >> Now there is still
    >> the
    >> > > > question of
    >> > > >     > > > > > supporting a
    >> > > >     > > > > >     > > >     > non-tombstone
    >> > > >     > > > > >     > > >     >     > and null
    >> > > >     > > > > >     > > >     >     >         >> value from producer
    >> > client
    >> > > > with
    >> > > >     > > > > > magic-byte = 2.*
    >> > > >     > > > > >     > > (I
    >> > > >     > > > > >     > > > am
    >> > > >     > > > > >     > > >     > not sure
    >> > > >     > > > > >     > > >     >     > if we
    >> > > >     > > > > >     > > >     >     >         >> should support this.
    >> > > > Ismael/Becket
    >> > > >     > > can
    >> > > >     > > > > > comment
    >> > > >     > > > > >     > > > here)*
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> 3) When almost all
    >> the
    >> > > > clients
    >> > > >     > have
    >> > > >     > > > > > upgraded,
    >> > > >     > > > > >     > the
    >> > > >     > > > > >     > > >     >     > message.format.version
    >> > > >     > > > > >     > > >     >     >         >> on
    >> > > >     > > > > >     > > >     >     >         >> the broker can be
    >> > changed
    >> > > > to 2,
    >> > > >     > > where
    >> > > >     > > > in
    >> > > >     > > > > > the
    >> > > >     > > > > >     > down
    >> > > >     > > > > >     > > >     > conversion in
    >> > > >     > > > > >     > > >     >     > the above
    >> > > >     > > > > >     > > >     >     >         >> step will not
    >> happen. If
    >> > > at
    >> > > > this
    >> > > >     > > point
    >> > > >     > > > > we
    >> > > >     > > > > > get a
    >> > > >     > > > > >     > > > consumer
    >> > > >     > > > > >     > > >     >     > request from a
    >> > > >     > > > > >     > > >     >     >         >> older consumer, we
    >> might
    >> > > > have to
    >> > > >     > > down
    >> > > >     > > > > > convert
    >> > > >     > > > > >     > > where
    >> > > >     > > > > >     > > > in we
    >> > > >     > > > > >     > > >     > loose
    >> > > >     > > > > >     > > >     >     > zero copy,
    >> > > >     > > > > >     > > >     >     >         >> but these cases
    >> should
    >> > be
    >> > > > rare.
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Becket can you review
    >> > this
    >> > > > plan
    >> > > >     > and
    >> > > >     > > > add
    >> > > >     > > > > > more
    >> > > >     > > > > >     > > > details if I
    >> > > >     > > > > >     > > >     > have
    >> > > >     > > > > >     > > >     >     >         >> missed/wronged
    >> > something,
    >> > > > before
    >> > > >     > we
    >> > > >     > > > put
    >> > > >     > > > > > it on
    >> > > >     > > > > >     > KIP.
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Thanks,
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> Mayuresh
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016
    >> at
    >> > > > 11:07 PM,
    >> > > >     > > > > Michael
    >> > > >     > > > > >     > Pearce <
    >> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
    >> > > >     > > > > >     > > >     >     >         >> wrote:
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> > Thanks guys, for
    >> > > > discussing this
    >> > > >     > > > > > offline and
    >> > > >     > > > > >     > > > getting
    >> > > >     > > > > >     > > >     > some
    >> > > >     > > > > >     > > >     >     > consensus.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > So its clear for
    >> > myself
    >> > > > and
    >> > > >     > others
    >> > > >     > > > > what
    >> > > >     > > > > > is
    >> > > >     > > > > >     > > > proposed now
    >> > > >     > > > > >     > > >     > (i
    >> > > >     > > > > >     > > >     >     > think i
    >> > > >     > > > > >     > > >     >     >         >> > understand, but
    >> want
    >> > to
    >> > > > make
    >> > > >     > sure)
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > Could i ask either
    >> > > > directly
    >> > > >     > update
    >> > > >     > > > the
    >> > > >     > > > > > kip to
    >> > > >     > > > > >     > > > detail the
    >> > > >     > > > > >     > > >     >     > migration
    >> > > >     > > > > >     > > >     >     >         >> > strategy, or
    >> > (re-)state
    >> > > > your
    >> > > >     > > offline
    >> > > >     > > > > > discussed
    >> > > >     > > > > >     > > and
    >> > > >     > > > > >     > > >     > agreed
    >> > > >     > > > > >     > > >     >     > migration
    >> > > >     > > > > >     > > >     >     >         >> > strategy based on a
    >> > > magic
    >> > > > byte
    >> > > >     > is
    >> > > >     > > in
    >> > > >     > > > > > this
    >> > > >     > > > > >     > > thread.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > The main original
    >> > driver
    >> > > > for the
    >> > > >     > > KIP
    >> > > >     > > > > > was to
    >> > > >     > > > > >     > > > support
    >> > > >     > > > > >     > > >     >     > compaction where
    >> > > >     > > > > >     > > >     >     >         >> value
    >> > > >     > > > > >     > > >     >     >         >> > isn't null, based
    >> off
    >> > > the
    >> > > >     > > > discussions
    >> > > >     > > > > on
    >> > > >     > > > > >     > KIP-82
    >> > > >     > > > > >     > > > thread.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > We should be able
    >> to
    >> > > > support
    >> > > >     > > > > > non-tombstone +
    >> > > >     > > > > >     > > null
    >> > > >     > > > > >     > > > value
    >> > > >     > > > > >     > > >     > by the
    >> > > >     > > > > >     > > >     >     >         >> completion
    >> > > >     > > > > >     > > >     >     >         >> > of the KIP, as we
    >> > noted
    >> > > > when
    >> > > >     > > > > discussing
    >> > > >     > > > > > this
    >> > > >     > > > > >     > > kip,
    >> > > >     > > > > >     > > > having
    >> > > >     > > > > >     > > >     >     > logic based on
    >> > > >     > > > > >     > > >     >     >         >> a
    >> > > >     > > > > >     > > >     >     >         >> > null value isn't
    >> very
    >> > > > clean and
    >> > > >     > > also
    >> > > >     > > > > > separates
    >> > > >     > > > > >     > > the
    >> > > >     > > > > >     > > >     > concerns.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > As discussed
    >> already
    >> > > > though we
    >> > > >     > can
    >> > > >     > > > > > split this
    >> > > >     > > > > >     > > into
    >> > > >     > > > > >     > > >     > KIP-87a
    >> > > >     > > > > >     > > >     >     > and KIP-87b
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > Where we look to
    >> > deliver
    >> > > > KIP-87a
    >> > > >     > > on
    >> > > >     > > > a
    >> > > >     > > > > >     > compacted
    >> > > >     > > > > >     > > > topic
    >> > > >     > > > > >     > > >     > (to
    >> > > >     > > > > >     > > >     >     > address the
    >> > > >     > > > > >     > > >     >     >         >> > immediate issues)
    >> > > >     > > > > >     > > >     >     >         >> > * tombstone + null
    >> > value
    >> > > >     > > > > >     > > >     >     >         >> > * tombstone +
    >> non-null
    >> > > > value
    >> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
    >> > > > non-null value
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > Then we can discuss
    >> > once
    >> > > > KIP-87a
    >> > > >     > > is
    >> > > >     > > > > > completed
    >> > > >     > > > > >     > > > options
    >> > > >     > > > > >     > > >     > later
    >> > > >     > > > > >     > > >     >     > and how we
    >> > > >     > > > > >     > > >     >     >         >> > support the second
    >> > part
    >> > > > KIP-87b
    >> > > >     > to
    >> > > >     > > > > > deliver:
    >> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
    >> null
    >> > > > value
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > Cheers
    >> > > >     > > > > >     > > >     >     >         >> > Mike
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > > ______________________________
    >> > > >     > > > > > __________
    >> > > >     > > > > >     > > >     >     >         >> > From: Becket Qin <
    >> > > >     > > > > becket.qin@gmail.com>
    >> > > >     > > > > >     > > >     >     >         >> > Sent: Thursday,
    >> > November
    >> > > > 17,
    >> > > >     > 2016
    >> > > >     > > > 1:43
    >> > > >     > > > > > AM
    >> > > >     > > > > >     > > >     >     >         >> > To:
    >> > > dev@kafka.apache.org
    >> > > >     > > > > >     > > >     >     >         >> > Subject: Re:
    >> [DISCUSS]
    >> > > > KIP-87 -
    >> > > >     > > Add
    >> > > >     > > > > > Compaction
    >> > > >     > > > > >     > > >     > Tombstone Flag
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and
    >> I
    >> > had
    >> > > > an
    >> > > >     > > offline
    >> > > >     > > > > >     > discussion,
    >> > > >     > > > > >     > > > and
    >> > > >     > > > > >     > > >     > following
    >> > > >     > > > > >     > > >     >     > is a brief
    >> > > >     > > > > >     > > >     >     >         >> > summary.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > 1. We agreed that
    >> not
    >> > > > bumping up
    >> > > >     > > > magic
    >> > > >     > > > > > value
    >> > > >     > > > > >     > may
    >> > > >     > > > > >     > > > result
    >> > > >     > > > > >     > > >     > in
    >> > > >     > > > > >     > > >     >     > losing zero
    >> > > >     > > > > >     > > >     >     >         >> copy
    >> > > >     > > > > >     > > >     >     >         >> > during migration.
    >> > > >     > > > > >     > > >     >     >         >> > 2. Given that
    >> bumping
    >> > up
    >> > > > magic
    >> > > >     > > value
    >> > > >     > > > > is
    >> > > >     > > > > > almost
    >> > > >     > > > > >     > > > free and
    >> > > >     > > > > >     > > >     > has
    >> > > >     > > > > >     > > >     >     > benefit of
    >> > > >     > > > > >     > > >     >     >         >> > avoiding potential
    >> > > > performance
    >> > > >     > > > issue.
    >> > > >     > > > > > It is
    >> > > >     > > > > >     > > > probably
    >> > > >     > > > > >     > > >     > worth
    >> > > >     > > > > >     > > >     >     > doing.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > One issue we still
    >> > need
    >> > > > to think
    >> > > >     > > > about
    >> > > >     > > > > > is
    >> > > >     > > > > >     > > whether
    >> > > >     > > > > >     > > > we
    >> > > >     > > > > >     > > >     > want to
    >> > > >     > > > > >     > > >     >     > support a
    >> > > >     > > > > >     > > >     >     >         >> > non-tombstone
    >> message
    >> > > > with null
    >> > > >     > > > value.
    >> > > >     > > > > >     > > >     >     >         >> > Currently it is not
    >> > > > supported by
    >> > > >     > > > > Kafka.
    >> > > >     > > > > > If we
    >> > > >     > > > > >     > > > allow a
    >> > > >     > > > > >     > > >     >     > non-tombstone null
    >> > > >     > > > > >     > > >     >     >         >> > value message to
    >> exist
    >> > > > after
    >> > > >     > > KIP-87.
    >> > > >     > > > > The
    >> > > >     > > > > >     > problem
    >> > > >     > > > > >     > > > is
    >> > > >     > > > > >     > > >     > that such
    >> > > >     > > > > >     > > >     >     > message
    >> > > >     > > > > >     > > >     >     >         >> will
    >> > > >     > > > > >     > > >     >     >         >> > not be supported by
    >> > the
    >> > > >     > consumers
    >> > > >     > > > > prior
    >> > > >     > > > > > to
    >> > > >     > > > > >     > > KIP-87.
    >> > > >     > > > > >     > > >     > Because a
    >> > > >     > > > > >     > > >     >     > null value
    >> > > >     > > > > >     > > >     >     >         >> > will always be
    >> > > > interpreted to a
    >> > > >     > > > > > tombstone.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > One option is that
    >> we
    >> > > > keep the
    >> > > >     > > > current
    >> > > >     > > > > > way,
    >> > > >     > > > > >     > i.e.
    >> > > >     > > > > >     > > > do not
    >> > > >     > > > > >     > > >     >     > support such
    >> > > >     > > > > >     > > >     >     >         >> > message. It would
    >> be
    >> > > good
    >> > > > to
    >> > > >     > know
    >> > > >     > > if
    >> > > >     > > > > > there is
    >> > > >     > > > > >     > a
    >> > > >     > > > > >     > > >     > concrete use
    >> > > >     > > > > >     > > >     >     > case for
    >> > > >     > > > > >     > > >     >     >         >> such
    >> > > >     > > > > >     > > >     >     >         >> > message. If there
    >> is
    >> > > not,
    >> > > > we can
    >> > > >     > > > > > probably just
    >> > > >     > > > > >     > > not
    >> > > >     > > > > >     > > >     > support it.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > Thanks,
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > JIangjie (Becket)
    >> Qin
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > On Wed, Nov 16,
    >> 2016
    >> > at
    >> > > > 1:28 PM,
    >> > > >     > > > > > Mayuresh
    >> > > >     > > > > >     > > Gharat <
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > > gharatmayuresh15@gmail.com
    >> > > >     > > > > >     > > >     >     >         >> > > wrote:
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >> > > Hi Ismael,
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > > This is
    >> something I
    >> > > can
    >> > > > think
    >> > > >     > of
    >> > > >     > > > for
    >> > > >     > > > > >     > migration
    >> > > >     > > > > >     > > > plan:
    >> > > >     > > > > >     > > >     >     >         >> > > So the migration
    >> > plan
    >> > > > can look
    >> > > >     > > > > > something
    >> > > >     > > > > >     > like
    >> > > >     > > > > >     > > > this,
    >> > > >     > > > > >     > > >     > with up
    >> > > >     > > > > >     > > >     >     >         >> conversion :
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > > 1) Currently lets
    >> > say
    >> > > > we have
    >> > > >     > > > Broker
    >> > > >     > > > > > at
    >> > > >     > > > > >     > > version
    >> > > >     > > > > >     > > > x.
    >> > > >     > > > > >     > > >     >     >         >> > > 2) Currently we
    >> have
    >> > > > clients
    >> > > >     > at
    >> > > >     > > > > > version x.
    >> > > >     > > > > >     > > >     >     >         >> > > 3) a) We move the
    >> > > > version to
    >> > > >     > > > > > Broker(x+1) :
    >> > > >     > > > > >     > > > supports
    >> > > >     > > > > >     > > >     > both
    >> > > >     > > > > >     > > >     >     > tombstone and
    >> > > >     > > > > >     > > >     >     >         >> > null
    >> > > >     > > > > >     > > >     >     >         >> > > for log
    >> compaction.
    >> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
    >> > the
    >> > > > client
    >> > > >     > to
    >> > > >     > > > > > version
    >> > > >     > > > > >     > > > client(x+1) :
    >> > > >     > > > > >     > > >     > if in
    >> > > >     > > > > >     > > >     >     > the
    >> > > >     > > > > >     > > >     >     >         >> producer
    >> > > >     > > > > >     > > >     >     >         >> > > client(x+1) the
    >> > value
    >> > > > is set
    >> > > >     > to
    >> > > >     > > > > null,
    >> > > >     > > > > > we
    >> > > >     > > > > >     > will
    >> > > >     > > > > >     > > >     > automatically
    >> > > >     > > > > >     > > >     >     > set the
    >> > > >     > > > > >     > > >     >     >         >> > > Tombstone bit
    >> > > > internally. If
    >> > > >     > the
    >> > > >     > > > > > producer
    >> > > >     > > > > >     > > > client(x+1)
    >> > > >     > > > > >     > > >     > sets
    >> > > >     > > > > >     > > >     >     > the
    >> > > >     > > > > >     > > >     >     >         >> tombstone
    >> > > >     > > > > >     > > >     >     >         >> > > itself, well and
    >> > good.
    >> > > > For
    >> > > >     > > > producer
    >> > > >     > > > > >     > client(x),
    >> > > >     > > > > >     > > > the
    >> > > >     > > > > >     > > >     > broker
    >> > > >     > > > > >     > > >     >     > will up
    >> > > >     > > > > >     > > >     >     >         >> convert
    >> > > >     > > > > >     > > >     >     >         >> > > to have the
    >> > tombstone
    >> > > > bit.
    >> > > >     > > > > > Broker(x+1) is
    >> > > >     > > > > >     > > > supporting
    >> > > >     > > > > >     > > >     > both.
    >> > > >     > > > > >     > > >     >     > Consumer
    >> > > >     > > > > >     > > >     >     >         >> > > client(x+1) will
    >> be
    >> > > > aware of
    >> > > >     > > this
    >> > > >     > > > > and
    >> > > >     > > > > > should
    >> > > >     > > > > >     > > be
    >> > > >     > > > > >     > > > able
    >> > > >     > > > > >     > > >     > to
    >> > > >     > > > > >     > > >     >     > handle this.
    >> > > >     > > > > >     > > >     >     >         >> For
    >> > > >     > > > > >     > > >     >     >         >> > > consumer
    >> client(x)
    >> > we
    >> > > > will
    >> > > >     > down
    >> > > >     > > > > > convert the
    >> > > >     > > > > >     > > > message
    >> > > >     > > > > >     > > >     > on the
    >> > > >     > > > > >     > > >     >     > broker
    >> > > >     > > > > >     > > >     >     >         >> side.
    >> > > >     > > > > >     > > >     >     >         >> > >     c) At this
    >> point
    >> > > we
    >> > > > will
    >> > > >     > > have
    >> > > >     > > > to
    >> > > >     > > > > >     > specify a
    >> > > >     > > > > >     > > >     > warning or
    >> > > >     > > > > >     > > >     >     > clearly
    >> > > >     > > > > >     > > >     >     >         >> specify
    >> > > >     > > > > >     > > >     >     >         >> > > in docs that this
    >> > > > behavior is
    >> > > >     > > > about
    >> > > >     > > > > > to be
    >> > > >     > > > > >     > > > changed for
    >> > > >     > > > > >     > > >     > log
    >> > > >     > > > > >     > > >     >     > compaction.
    >> > > >     > > > > >     > > >     >     >         >> > > 4) a) In next
    >> > release
    >> > > > of the
    >> > > >     > > > > > Broker(x+2), we
    >> > > >     > > > > >     > > > say that
    >> > > >     > > > > >     > > >     > only
    >> > > >     > > > > >     > > >     >     > Tombstone
    >> > > >     > > > > >     > > >     >     >         >> is
    >> > > >     > > > > >     > > >     >     >         >> > > used for log
    >> > > compaction
    >> > > > on the
    >> > > >     > > > > Broker
    >> > > >     > > > > > side.
    >> > > >     > > > > >     > > >     > Clients(x+1)
    >> > > >     > > > > >     > > >     >     > still is
    >> > > >     > > > > >     > > >     >     >         >> > > supported.
    >> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
    >> > the
    >> > > > client
    >> > > >     > to
    >> > > >     > > > > > version
    >> > > >     > > > > >     > > > client(x+2) :
    >> > > >     > > > > >     > > >     > if
    >> > > >     > > > > >     > > >     >     > value is set
    >> > > >     > > > > >     > > >     >     >         >> to
    >> > > >     > > > > >     > > >     >     >         >> > > null, tombstone
    >> will
    >> > > > not be
    >> > > >     > set
    >> > > >     > > > > >     > automatically.
    >> > > >     > > > > >     > > > The
    >> > > >     > > > > >     > > >     > client
    >> > > >     > > > > >     > > >     >     > will have to
    >> > > >     > > > > >     > > >     >     >         >> > call
    >> > > >     > > > > >     > > >     >     >         >> > > setTombstone() to
    >> > > > actually set
    >> > > >     > > the
    >> > > >     > > > > >     > tombstone.
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > > We should compare
    >> > this
    >> > > >     > migration
    >> > > >     > > > > plan
    >> > > >     > > > > > with
    >> > > >     > > > > >     > the
    >> > > >     > > > > >     > > >     > migration
    >> > > >     > > > > >     > > >     >     > plan for
    >> > > >     > > > > >     > > >     >     >         >> magic
    >> > > >     > > > > >     > > >     >     >         >> > > byte bump and do
    >> > > > whatever
    >> > > >     > looks
    >> > > >     > > > > good.
    >> > > >     > > > > >     > > >     >     >         >> > > I am just worried
    >> > that
    >> > > > if we
    >> > > >     > go
    >> > > >     > > > down
    >> > > >     > > > > > magic
    >> > > >     > > > > >     > > byte
    >> > > >     > > > > >     > > > route,
    >> > > >     > > > > >     > > >     >     > unless I am
    >> > > >     > > > > >     > > >     >     >         >> > missing
    >> > > >     > > > > >     > > >     >     >         >> > > something, it
    >> sounds
    >> > > > like
    >> > > >     > kafka
    >> > > >     > > > will
    >> > > >     > > > > > be
    >> > > >     > > > > >     > stuck
    >> > > >     > > > > >     > > > with
    >> > > >     > > > > >     > > >     >     > supporting both
    >> > > >     > > > > >     > > >     >     >         >> null
    >> > > >     > > > > >     > > >     >     >         >> > > value and
    >> tombstone
    >> > > bit
    >> > > > for
    >> > > >     > log
    >> > > >     > > > > > compaction
    >> > > >     > > > > >     > for
    >> > > >     > > > > >     > > > life
    >> > > >     > > > > >     > > >     > long,
    >> > > >     > > > > >     > > >     >     > which does
    >> > > >     > > > > >     > > >     >     >         >> not
    >> > > >     > > > > >     > > >     >     >         >> > > look like a good
    >> end
    >> > > > state.
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > > Thanks,
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > > Mayuresh
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16,
    >> 2016
    >> > > at
    >> > > > 9:32
    >> > > >     > AM,
    >> > > >     > > > > > Mayuresh
    >> > > >     > > > > >     > > > Gharat <
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > > gharatmayuresh15@gmail.com
    >> > > >     > > > > >     > > >     >     >         >> > > > wrote:
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > That's a very
    >> good
    >> > > > point
    >> > > >     > > which I
    >> > > >     > > > > > might
    >> > > >     > > > > >     > have
    >> > > >     > > > > >     > > > not
    >> > > >     > > > > >     > > >     >     > considered earlier.
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > Here is a plan
    >> > that
    >> > > I
    >> > > > can
    >> > > >     > > think
    >> > > >     > > > > of:
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > Stage 1) The
    >> > broker
    >> > > > from now
    >> > > >     > > on,
    >> > > >     > > > > up
    >> > > >     > > > > >     > converts
    >> > > >     > > > > >     > > > the
    >> > > >     > > > > >     > > >     > message
    >> > > >     > > > > >     > > >     >     > to have the
    >> > > >     > > > > >     > > >     >     >         >> > > > tombstone
    >> marker.
    >> > > The
    >> > > > log
    >> > > >     > > > > compaction
    >> > > >     > > > > >     > thread
    >> > > >     > > > > >     > > > does log
    >> > > >     > > > > >     > > >     >     > compaction
    >> > > >     > > > > >     > > >     >     >         >> based
    >> > > >     > > > > >     > > >     >     >         >> > on
    >> > > >     > > > > >     > > >     >     >         >> > > > both null and
    >> > > > tombstone
    >> > > >     > > marker.
    >> > > >     > > > > > This is
    >> > > >     > > > > >     > our
    >> > > >     > > > > >     > > >     > transition
    >> > > >     > > > > >     > > >     >     > period.
    >> > > >     > > > > >     > > >     >     >         >> > > > Stage 2) The
    >> next
    >> > > > release we
    >> > > >     > > > only
    >> > > >     > > > > > say that
    >> > > >     > > > > >     > > log
    >> > > >     > > > > >     > > >     > compaction
    >> > > >     > > > > >     > > >     >     > is based
    >> > > >     > > > > >     > > >     >     >         >> on
    >> > > >     > > > > >     > > >     >     >         >> > > > tombstone
    >> marker.
    >> > > > (Open
    >> > > >     > source
    >> > > >     > > > > > kafka makes
    >> > > >     > > > > >     > > > this as a
    >> > > >     > > > > >     > > >     >     > policy). By
    >> > > >     > > > > >     > > >     >     >         >> this
    >> > > >     > > > > >     > > >     >     >         >> > > time,
    >> > > >     > > > > >     > > >     >     >         >> > > > the
    >> organization
    >> > > > which is
    >> > > >     > > moving
    >> > > >     > > > > to
    >> > > >     > > > > > this
    >> > > >     > > > > >     > > > release
    >> > > >     > > > > >     > > >     > will be
    >> > > >     > > > > >     > > >     >     > sure that
    >> > > >     > > > > >     > > >     >     >         >> they
    >> > > >     > > > > >     > > >     >     >         >> > > > have gone
    >> through
    >> > > the
    >> > > > entire
    >> > > >     > > > > > transition
    >> > > >     > > > > >     > > > period.
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > My only goal of
    >> > > doing
    >> > > > this
    >> > > >     > is
    >> > > >     > > > that
    >> > > >     > > > > > Kafka
    >> > > >     > > > > >     > > > clearly
    >> > > >     > > > > >     > > >     >     > specifies the end
    >> > > >     > > > > >     > > >     >     >         >> > state
    >> > > >     > > > > >     > > >     >     >         >> > > > about what log
    >> > > > compaction
    >> > > >     > > means
    >> > > >     > > > > (is
    >> > > >     > > > > > it
    >> > > >     > > > > >     > null
    >> > > >     > > > > >     > > > value
    >> > > >     > > > > >     > > >     > or a
    >> > > >     > > > > >     > > >     >     > tombstone
    >> > > >     > > > > >     > > >     >     >         >> > marker,
    >> > > >     > > > > >     > > >     >     >         >> > > > but not both).
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > What do you
    >> think?
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > Thanks,
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh
    >> > > >     > > > > >     > > >     >     >         >> > > > .
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16,
    >> > 2016
    >> > > > at 9:17
    >> > > >     > > AM,
    >> > > >     > > > > > Ismael
    >> > > >     > > > > >     > > Juma <
    >> > > >     > > > > >     > > >     >     > ismael@juma.me.uk>
    >> > > >     > > > > >     > > >     >     >         >> > wrote:
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > >> One comment
    >> > below.
    >> > > >     > > > > >     > > >     >     >         >> > > >>
    >> > > >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov
    >> 16,
    >> > > 2016
    >> > > > at
    >> > > >     > 5:08
    >> > > >     > > > PM,
    >> > > >     > > > > > Mayuresh
    >> > > >     > > > > >     > > > Gharat <
    >> > > >     > > > > >     > > >     >     >         >> > > >>
    >> > > > gharatmayuresh15@gmail.com
    >> > > >     > > > > >     > > >     >     >         >> > > >> > wrote:
    >> > > >     > > > > >     > > >     >     >         >> > > >>
    >> > > >     > > > > >     > > >     >     >         >> > > >> >    - If we
    >> > don't
    >> > > > bump up
    >> > > >     > > the
    >> > > >     > > > > > magic
    >> > > >     > > > > >     > byte,
    >> > > >     > > > > >     > > > on the
    >> > > >     > > > > >     > > >     > broker
    >> > > >     > > > > >     > > >     >     > side, the
    >> > > >     > > > > >     > > >     >     >         >> > > broker
    >> > > >     > > > > >     > > >     >     >         >> > > >> >    will
    >> always
    >> > > > have to
    >> > > >     > look
    >> > > >     > > > at
    >> > > >     > > > > > both
    >> > > >     > > > > >     > > > tombstone
    >> > > >     > > > > >     > > >     > bit and
    >> > > >     > > > > >     > > >     >     > the value
    >> > > >     > > > > >     > > >     >     >         >> when
    >> > > >     > > > > >     > > >     >     >         >> > > do
    >> > > >     > > > > >     > > >     >     >         >> > > >> the
    >> > > >     > > > > >     > > >     >     >         >> > > >> >
    >> compaction.
    >> > > > Assuming
    >> > > >     > we
    >> > > >     > > do
    >> > > >     > > > > > not bump
    >> > > >     > > > > >     > up
    >> > > >     > > > > >     > > > the
    >> > > >     > > > > >     > > >     > magic
    >> > > >     > > > > >     > > >     >     > byte,
    >> > > >     > > > > >     > > >     >     >         >> > > >> >    imagine
    >> the
    >> > > > broker
    >> > > >     > sees
    >> > > >     > > a
    >> > > >     > > > > > message
    >> > > >     > > > > >     > > which
    >> > > >     > > > > >     > > > does
    >> > > >     > > > > >     > > >     > not
    >> > > >     > > > > >     > > >     >     > have a
    >> > > >     > > > > >     > > >     >     >         >> tombstone
    >> > > >     > > > > >     > > >     >     >         >> > > bit
    >> > > >     > > > > >     > > >     >     >         >> > > >> >    set. The
    >> > > broker
    >> > > > does
    >> > > >     > not
    >> > > >     > > > > know
    >> > > >     > > > > > when
    >> > > >     > > > > >     > the
    >> > > >     > > > > >     > > >     > message was
    >> > > >     > > > > >     > > >     >     > produced
    >> > > >     > > > > >     > > >     >     >         >> (i.e.
    >> > > >     > > > > >     > > >     >     >         >> > > >> > whether
    >> > > >     > > > > >     > > >     >     >         >> > > >> >    the
    >> message
    >> > > has
    >> > > > been
    >> > > >     > up
    >> > > >     > > > > > converted or
    >> > > >     > > > > >     > > > not), it
    >> > > >     > > > > >     > > >     > has
    >> > > >     > > > > >     > > >     >     > to take a
    >> > > >     > > > > >     > > >     >     >         >> > further
    >> > > >     > > > > >     > > >     >     >         >> > > >> > look at
    >> > > >     > > > > >     > > >     >     >         >> > > >> >    the
    >> value to
    >> > > > see if it
    >> > > >     > > is
    >> > > >     > > > > > null or
    >> > > >     > > > > >     > not
    >> > > >     > > > > >     > > in
    >> > > >     > > > > >     > > >     > order to
    >> > > >     > > > > >     > > >     >     > determine
    >> > > >     > > > > >     > > >     >     >         >> if it
    >> > > >     > > > > >     > > >     >     >         >> > > is
    >> > > >     > > > > >     > > >     >     >         >> > > >> a
    >> > > >     > > > > >     > > >     >     >         >> > > >> >
    >> tombstone.
    >> > The
    >> > > > same
    >> > > >     > > logic
    >> > > >     > > > > has
    >> > > >     > > > > > to be
    >> > > >     > > > > >     > > put
    >> > > >     > > > > >     > > > on the
    >> > > >     > > > > >     > > >     >     > consumer as
    >> > > >     > > > > >     > > >     >     >         >> well
    >> > > >     > > > > >     > > >     >     >         >> > > >> because
    >> > > >     > > > > >     > > >     >     >         >> > > >> > the
    >> > > >     > > > > >     > > >     >     >         >> > > >> >    consumer
    >> > does
    >> > > > not know
    >> > > >     > > if
    >> > > >     > > > > the
    >> > > >     > > > > >     > message
    >> > > >     > > > > >     > > > has
    >> > > >     > > > > >     > > >     > been up
    >> > > >     > > > > >     > > >     >     > converted or
    >> > > >     > > > > >     > > >     >     >         >> > not.
    >> > > >     > > > > >     > > >     >     >         >> > > >> >       - If
    >> we
    >> > > > upconvert
    >> > > >     > > while
    >> > > >     > > > > >     > appending,
    >> > > >     > > > > >     > > > this is
    >> > > >     > > > > >     > > >     > not
    >> > > >     > > > > >     > > >     >     > the case,
    >> > > >     > > > > >     > > >     >     >         >> > right?
    >> > > >     > > > > >     > > >     >     >         >> > > >>
    >> > > >     > > > > >     > > >     >     >         >> > > >>
    >> > > >     > > > > >     > > >     >     >         >> > > >> If I
    >> understand
    >> > you
    >> > > >     > > correctly,
    >> > > >     > > > > > this is
    >> > > >     > > > > >     > not
    >> > > >     > > > > >     > > >     > sufficient
    >> > > >     > > > > >     > > >     >     > because the
    >> > > >     > > > > >     > > >     >     >         >> log
    >> > > >     > > > > >     > > >     >     >         >> > > may
    >> > > >     > > > > >     > > >     >     >         >> > > >> have messages
    >> > > > appended
    >> > > >     > before
    >> > > >     > > > it
    >> > > >     > > > > > was
    >> > > >     > > > > >     > > > upgraded to
    >> > > >     > > > > >     > > >     > include
    >> > > >     > > > > >     > > >     >     > KIP-87.
    >> > > >     > > > > >     > > >     >     >         >> > > >>
    >> > > >     > > > > >     > > >     >     >         >> > > >> Ismael
    >> > > >     > > > > >     > > >     >     >         >> > > >>
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > > > --
    >> > > >     > > > > >     > > >     >     >         >> > > > -Regards,
    >> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh R.
    >> Gharat
    >> > > >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
    >> > > >     > > > > >     > > >     >     >         >> > > >
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > > --
    >> > > >     > > > > >     > > >     >     >         >> > > -Regards,
    >> > > >     > > > > >     > > >     >     >         >> > > Mayuresh R.
    >> Gharat
    >> > > >     > > > > >     > > >     >     >         >> > > (862) 250-7125
    >> > > >     > > > > >     > > >     >     >         >> > >
    >> > > >     > > > > >     > > >     >     >         >> > The information
    >> > > contained
    >> > > > in
    >> > > >     > this
    >> > > >     > > > > email
    >> > > >     > > > > > is
    >> > > >     > > > > >     > > > strictly
    >> > > >     > > > > >     > > >     >     > confidential and for
    >> > > >     > > > > >     > > >     >     >         >> > the use of the
    >> > addressee
    >> > > > only,
    >> > > >     > > > unless
    >> > > >     > > > > >     > otherwise
    >> > > >     > > > > >     > > >     > indicated. If
    >> > > >     > > > > >     > > >     >     > you are
    >> > > >     > > > > >     > > >     >     >         >> not
    >> > > >     > > > > >     > > >     >     >         >> > the intended
    >> > recipient,
    >> > > > please
    >> > > >     > do
    >> > > >     > > > not
    >> > > >     > > > > > read,
    >> > > >     > > > > >     > > copy,
    >> > > >     > > > > >     > > > use or
    >> > > >     > > > > >     > > >     >     > disclose to
    >> > > >     > > > > >     > > >     >     >         >> others
    >> > > >     > > > > >     > > >     >     >         >> > this message or any
    >> > > > attachment.
    >> > > >     > > > Please
    >> > > >     > > > > > also
    >> > > >     > > > > >     > > > notify the
    >> > > >     > > > > >     > > >     > sender
    >> > > >     > > > > >     > > >     >     > by
    >> > > >     > > > > >     > > >     >     >         >> replying
    >> > > >     > > > > >     > > >     >     >         >> > to this email or by
    >> > > > telephone
    >> > > >     > > > (+44(020
    >> > > >     > > > > > 7896
    >> > > >     > > > > >     > > 0011)
    >> > > >     > > > > >     > > > and
    >> > > >     > > > > >     > > >     > then
    >> > > >     > > > > >     > > >     >     > delete the
    >> > > >     > > > > >     > > >     >     >         >> email
    >> > > >     > > > > >     > > >     >     >         >> > and any copies of
    >> it.
    >> > > > Opinions,
    >> > > >     > > > > > conclusion
    >> > > >     > > > > >     > (etc)
    >> > > >     > > > > >     > > > that
    >> > > >     > > > > >     > > >     > do not
    >> > > >     > > > > >     > > >     >     > relate to
    >> > > >     > > > > >     > > >     >     >         >> the
    >> > > >     > > > > >     > > >     >     >         >> > official business
    >> of
    >> > > this
    >> > > >     > company
    >> > > >     > > > > shall
    >> > > >     > > > > > be
    >> > > >     > > > > >     > > > understood as
    >> > > >     > > > > >     > > >     >     > neither given
    >> > > >     > > > > >     > > >     >     >         >> nor
    >> > > >     > > > > >     > > >     >     >         >> > endorsed by it. IG
    >> is
    >> > a
    >> > > > trading
    >> > > >     > > name
    >> > > >     > > > > of
    >> > > >     > > > > > IG
    >> > > >     > > > > >     > > Markets
    >> > > >     > > > > >     > > >     > Limited (a
    >> > > >     > > > > >     > > >     >     > company
    >> > > >     > > > > >     > > >     >     >         >> > registered in
    >> England
    >> > > and
    >> > > > Wales,
    >> > > >     > > > > company
    >> > > >     > > > > >     > number
    >> > > >     > > > > >     > > >     > 04008957) and
    >> > > >     > > > > >     > > >     >     > IG Index
    >> > > >     > > > > >     > > >     >     >         >> > Limited (a company
    >> > > > registered in
    >> > > >     > > > > > England and
    >> > > >     > > > > >     > > > Wales,
    >> > > >     > > > > >     > > >     > company
    >> > > >     > > > > >     > > >     >     > number
    >> > > >     > > > > >     > > >     >     >         >> > 01190902).
    >> Registered
    >> > > > address at
    >> > > >     > > > > Cannon
    >> > > >     > > > > > Bridge
    >> > > >     > > > > >     > > > House, 25
    >> > > >     > > > > >     > > >     >     > Dowgate Hill,
    >> > > >     > > > > >     > > >     >     >         >> > London EC4R 2YA.
    >> Both
    >> > IG
    >> > > > Markets
    >> > > >     > > > > Limited
    >> > > >     > > > > >     > > (register
    >> > > >     > > > > >     > > >     > number
    >> > > >     > > > > >     > > >     >     > 195355) and IG
    >> > > >     > > > > >     > > >     >     >         >> > Index Limited
    >> > (register
    >> > > > number
    >> > > >     > > > 114059)
    >> > > >     > > > > > are
    >> > > >     > > > > >     > > > authorised
    >> > > >     > > > > >     > > >     > and
    >> > > >     > > > > >     > > >     >     > regulated by
    >> > > >     > > > > >     > > >     >     >         >> the
    >> > > >     > > > > >     > > >     >     >         >> > Financial Conduct
    >> > > > Authority.
    >> > > >     > > > > >     > > >     >     >         >> >
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >> --
    >> > > >     > > > > >     > > >     >     >         >> -Regards,
    >> > > >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
    >> > > >     > > > > >     > > >     >     >         >> (862) 250-7125
    >> > > >     > > > > >     > > >     >     >         >> The information
    >> > contained
    >> > > > in this
    >> > > >     > > > email
    >> > > >     > > > > is
    >> > > >     > > > > >     > > strictly
    >> > > >     > > > > >     > > >     >     > confidential and for
    >> > > >     > > > > >     > > >     >     >         >> the use of the
    >> addressee
    >> > > > only,
    >> > > >     > > unless
    >> > > >     > > > > > otherwise
    >> > > >     > > > > >     > > >     > indicated. If
    >> > > >     > > > > >     > > >     >     > you are not
    >> > > >     > > > > >     > > >     >     >         >> the intended
    >> recipient,
    >> > > > please do
    >> > > >     > > not
    >> > > >     > > > > > read,
    >> > > >     > > > > >     > copy,
    >> > > >     > > > > >     > > > use or
    >> > > >     > > > > >     > > >     >     > disclose to others
    >> > > >     > > > > >     > > >     >     >         >> this message or any
    >> > > > attachment.
    >> > > >     > > Please
    >> > > >     > > > > > also
    >> > > >     > > > > >     > notify
    >> > > >     > > > > >     > > > the
    >> > > >     > > > > >     > > >     > sender
    >> > > >     > > > > >     > > >     >     > by replying
    >> > > >     > > > > >     > > >     >     >         >> to this email or by
    >> > > > telephone
    >> > > >     > > (+44(020
    >> > > >     > > > > > 7896
    >> > > >     > > > > >     > 0011)
    >> > > >     > > > > >     > > > and then
    >> > > >     > > > > >     > > >     >     > delete the email
    >> > > >     > > > > >     > > >     >     >         >> and any copies of it.
    >> > > > Opinions,
    >> > > >     > > > > > conclusion (etc)
    >> > > >     > > > > >     > > > that do
    >> > > >     > > > > >     > > >     > not
    >> > > >     > > > > >     > > >     >     > relate to the
    >> > > >     > > > > >     > > >     >     >         >> official business of
    >> > this
    >> > > > company
    >> > > >     > > > shall
    >> > > >     > > > > be
    >> > > >     > > > > >     > > > understood as
    >> > > >     > > > > >     > > >     >     > neither given nor
    >> > > >     > > > > >     > > >     >     >         >> endorsed by it. IG
    >> is a
    >> > > > trading
    >> > > >     > name
    >> > > >     > > > of
    >> > > >     > > > > IG
    >> > > >     > > > > >     > Markets
    >> > > >     > > > > >     > > >     > Limited (a
    >> > > >     > > > > >     > > >     >     > company
    >> > > >     > > > > >     > > >     >     >         >> registered in England
    >> > and
    >> > > > Wales,
    >> > > >     > > > company
    >> > > >     > > > > > number
    >> > > >     > > > > >     > > > 04008957)
    >> > > >     > > > > >     > > >     > and
    >> > > >     > > > > >     > > >     >     > IG Index
    >> > > >     > > > > >     > > >     >     >         >> Limited (a company
    >> > > > registered in
    >> > > >     > > > England
    >> > > >     > > > > > and
    >> > > >     > > > > >     > > Wales,
    >> > > >     > > > > >     > > >     > company
    >> > > >     > > > > >     > > >     >     > number
    >> > > >     > > > > >     > > >     >     >         >> 01190902). Registered
    >> > > > address at
    >> > > >     > > > Cannon
    >> > > >     > > > > > Bridge
    >> > > >     > > > > >     > > > House, 25
    >> > > >     > > > > >     > > >     >     > Dowgate Hill,
    >> > > >     > > > > >     > > >     >     >         >> London EC4R 2YA.
    >> Both IG
    >> > > > Markets
    >> > > >     > > > Limited
    >> > > >     > > > > >     > (register
    >> > > >     > > > > >     > > > number
    >> > > >     > > > > >     > > >     >     > 195355) and IG
    >> > > >     > > > > >     > > >     >     >         >> Index Limited
    >> (register
    >> > > > number
    >> > > >     > > 114059)
    >> > > >     > > > > are
    >> > > >     > > > > >     > > > authorised and
    >> > > >     > > > > >     > > >     >     > regulated by the
    >> > > >     > > > > >     > > >     >     >         >> Financial Conduct
    >> > > Authority.
    >> > > >     > > > > >     > > >     >     >         >>
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >         > --
    >> > > >     > > > > >     > > >     >     >         > -Regards,
    >> > > >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
    >> > > >     > > > > >     > > >     >     >         > (862) 250-7125
    >> > > >     > > > > >     > > >     >     >         >
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >         --
    >> > > >     > > > > >     > > >     >     >         -Regards,
    >> > > >     > > > > >     > > >     >     >         Mayuresh R. Gharat
    >> > > >     > > > > >     > > >     >     >         (862) 250-7125
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >     The information contained in
    >> > this
    >> > > > email
    >> > > >     > is
    >> > > >     > > > > > strictly
    >> > > >     > > > > >     > > > confidential
    >> > > >     > > > > >     > > >     > and
    >> > > >     > > > > >     > > >     >     > for the use of the addressee
    >> only,
    >> > > > unless
    >> > > >     > > > otherwise
    >> > > >     > > > > >     > > indicated.
    >> > > >     > > > > >     > > > If
    >> > > >     > > > > >     > > >     > you are
    >> > > >     > > > > >     > > >     >     > not the intended recipient,
    >> please
    >> > do
    >> > > > not
    >> > > >     > read,
    >> > > >     > > > > > copy, use
    >> > > >     > > > > >     > or
    >> > > >     > > > > >     > > >     > disclose to
    >> > > >     > > > > >     > > >     >     > others this message or any
    >> > > attachment.
    >> > > > Please
    >> > > >     > > > also
    >> > > >     > > > > > notify
    >> > > >     > > > > >     > the
    >> > > >     > > > > >     > > > sender
    >> > > >     > > > > >     > > >     > by
    >> > > >     > > > > >     > > >     >     > replying to this email or by
    >> > > telephone
    >> > > >     > (+44(020
    >> > > >     > > > > 7896
    >> > > >     > > > > > 0011)
    >> > > >     > > > > >     > > and
    >> > > >     > > > > >     > > > then
    >> > > >     > > > > >     > > >     > delete
    >> > > >     > > > > >     > > >     >     > the email and any copies of it.
    >> > > > Opinions,
    >> > > >     > > > > conclusion
    >> > > >     > > > > > (etc)
    >> > > >     > > > > >     > > > that do
    >> > > >     > > > > >     > > >     > not
    >> > > >     > > > > >     > > >     >     > relate to the official business
    >> of
    >> > > this
    >> > > >     > company
    >> > > >     > > > > > shall be
    >> > > >     > > > > >     > > > understood
    >> > > >     > > > > >     > > >     > as
    >> > > >     > > > > >     > > >     >     > neither given nor endorsed by
    >> it.
    >> > IG
    >> > > > is a
    >> > > >     > > trading
    >> > > >     > > > > > name of
    >> > > >     > > > > >     > IG
    >> > > >     > > > > >     > > > Markets
    >> > > >     > > > > >     > > >     >     > Limited (a company registered in
    >> > > > England and
    >> > > >     > > > Wales,
    >> > > >     > > > > > company
    >> > > >     > > > > >     > > > number
    >> > > >     > > > > >     > > >     >     > 04008957) and IG Index Limited
    >> (a
    >> > > > company
    >> > > >     > > > > registered
    >> > > >     > > > > > in
    >> > > >     > > > > >     > > > England and
    >> > > >     > > > > >     > > >     > Wales,
    >> > > >     > > > > >     > > >     >     > company number 01190902).
    >> > Registered
    >> > > > address
    >> > > >     > at
    >> > > >     > > > > > Cannon
    >> > > >     > > > > >     > Bridge
    >> > > >     > > > > >     > > > House,
    >> > > >     > > > > >     > > >     > 25
    >> > > >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA.
    >> Both
    >> > > IG
    >> > > >     > Markets
    >> > > >     > > > > > Limited
    >> > > >     > > > > >     > > > (register
    >> > > >     > > > > >     > > >     > number
    >> > > >     > > > > >     > > >     >     > 195355) and IG Index Limited
    >> > > (register
    >> > > > number
    >> > > >     > > > > > 114059) are
    >> > > >     > > > > >     > > > authorised
    >> > > >     > > > > >     > > >     > and
    >> > > >     > > > > >     > > >     >     > regulated by the Financial
    >> Conduct
    >> > > > Authority.
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >     >
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >     > The information contained in this
    >> email
    >> > is
    >> > > > strictly
    >> > > >     > > > > > confidential
    >> > > >     > > > > >     > > and
    >> > > >     > > > > >     > > > for
    >> > > >     > > > > >     > > >     > the use of the addressee only, unless
    >> > > > otherwise
    >> > > >     > > > > indicated.
    >> > > >     > > > > > If you
    >> > > >     > > > > >     > > > are not
    >> > > >     > > > > >     > > >     > the intended recipient, please do not
    >> > read,
    >> > > > copy,
    >> > > >     > use
    >> > > >     > > > or
    >> > > >     > > > > > disclose
    >> > > >     > > > > >     > > to
    >> > > >     > > > > >     > > > others
    >> > > >     > > > > >     > > >     > this message or any attachment. Please
    >> > also
    >> > > > notify
    >> > > >     > > the
    >> > > >     > > > > > sender by
    >> > > >     > > > > >     > > > replying
    >> > > >     > > > > >     > > >     > to this email or by telephone (+44(020
    >> > 7896
    >> > > > 0011
    >> > > >     > <020%207896%200011>) and
    >> > > >     > > > > then
    >> > > >     > > > > > delete
    >> > > >     > > > > >     > > > the email
    >> > > >     > > > > >     > > >     > and any copies of it. Opinions,
    >> > conclusion
    >> > > > (etc)
    >> > > >     > that
    >> > > >     > > > do
    >> > > >     > > > > > not
    >> > > >     > > > > >     > relate
    >> > > >     > > > > >     > > > to the
    >> > > >     > > > > >     > > >     > official business of this company
    >> shall
    >> > be
    >> > > >     > understood
    >> > > >     > > > as
    >> > > >     > > > > > neither
    >> > > >     > > > > >     > > > given nor
    >> > > >     > > > > >     > > >     > endorsed by it. IG is a trading name
    >> of
    >> > IG
    >> > > > Markets
    >> > > >     > > > > Limited
    >> > > >     > > > > > (a
    >> > > >     > > > > >     > > company
    >> > > >     > > > > >     > > >     > registered in England and Wales,
    >> company
    >> > > > number
    >> > > >     > > > 04008957)
    >> > > >     > > > > > and IG
    >> > > >     > > > > >     > > > Index
    >> > > >     > > > > >     > > >     > Limited (a company registered in
    >> England
    >> > > and
    >> > > > Wales,
    >> > > >     > > > > company
    >> > > >     > > > > >     > number
    >> > > >     > > > > >     > > >     > 01190902). Registered address at
    >> Cannon
    >> > > > Bridge
    >> > > >     > House,
    >> > > >     > > > 25
    >> > > >     > > > > > Dowgate
    >> > > >     > > > > >     > > > Hill,
    >> > > >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets
    >> Limited
    >> > > > (register
    >> > > >     > > > number
    >> > > >     > > > > > 195355)
    >> > > >     > > > > >     > > > and IG
    >> > > >     > > > > >     > > >     > Index Limited (register number 114059)
    >> > are
    >> > > >     > authorised
    >> > > >     > > > and
    >> > > >     > > > > >     > regulated
    >> > > >     > > > > >     > > > by the
    >> > > >     > > > > >     > > >     > Financial Conduct Authority.
    >> > > >     > > > > >     > > >     >
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > > The information contained in this email is
    >> > > strictly
    >> > > >     > > > > confidential
    >> > > >     > > > > > and
    >> > > >     > > > > >     > for
    >> > > >     > > > > >     > > > the use of the addressee only, unless
    >> otherwise
    >> > > >     > indicated.
    >> > > >     > > If
    >> > > >     > > > > > you are
    >> > > >     > > > > >     > not
    >> > > >     > > > > >     > > > the intended recipient, please do not read,
    >> > copy,
    >> > > > use or
    >> > > >     > > > > > disclose to
    >> > > >     > > > > >     > > others
    >> > > >     > > > > >     > > > this message or any attachment. Please also
    >> > > notify
    >> > > > the
    >> > > >     > > sender
    >> > > >     > > > > by
    >> > > >     > > > > >     > replying
    >> > > >     > > > > >     > > > to this email or by telephone (+44(020 7896
    >> > 0011
    >> > > >     > <020%207896%200011>) and then
    >> > > >     > > > > > delete the
    >> > > >     > > > > >     > > email
    >> > > >     > > > > >     > > > and any copies of it. Opinions, conclusion
    >> > (etc)
    >> > > > that do
    >> > > >     > > not
    >> > > >     > > > > > relate to
    >> > > >     > > > > >     > > the
    >> > > >     > > > > >     > > > official business of this company shall be
    >> > > > understood as
    >> > > >     > > > > neither
    >> > > >     > > > > > given
    >> > > >     > > > > >     > > nor
    >> > > >     > > > > >     > > > endorsed by it. IG is a trading name of IG
    >> > > Markets
    >> > > >     > Limited
    >> > > >     > > (a
    >> > > >     > > > > > company
    >> > > >     > > > > >     > > > registered in England and Wales, company
    >> number
    >> > > > 04008957)
    >> > > >     > > and
    >> > > >     > > > > IG
    >> > > >     > > > > > Index
    >> > > >     > > > > >     > > > Limited (a company registered in England and
    >> > > Wales,
    >> > > >     > company
    >> > > >     > > > > > number
    >> > > >     > > > > >     > > > 01190902). Registered address at Cannon
    >> Bridge
    >> > > > House, 25
    >> > > >     > > > > Dowgate
    >> > > >     > > > > > Hill,
    >> > > >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
    >> > > (register
    >> > > > number
    >> > > >     > > > > > 195355) and
    >> > > >     > > > > >     > IG
    >> > > >     > > > > >     > > > Index Limited (register number 114059) are
    >> > > > authorised and
    >> > > >     > > > > > regulated by
    >> > > >     > > > > >     > > the
    >> > > >     > > > > >     > > > Financial Conduct Authority.
    >> > > >     > > > > >     > > >
    >> > > >     > > > > >     > > The information contained in this email is
    >> > strictly
    >> > > >     > > > confidential
    >> > > >     > > > > > and for
    >> > > >     > > > > >     > > the use of the addressee only, unless
    >> otherwise
    >> > > > indicated.
    >> > > >     > If
    >> > > >     > > > you
    >> > > >     > > > > > are not
    >> > > >     > > > > >     > > the intended recipient, please do not read,
    >> copy,
    >> > > > use or
    >> > > >     > > > disclose
    >> > > >     > > > > > to
    >> > > >     > > > > >     > others
    >> > > >     > > > > >     > > this message or any attachment. Please also
    >> > notify
    >> > > > the
    >> > > >     > sender
    >> > > >     > > > by
    >> > > >     > > > > > replying
    >> > > >     > > > > >     > > to this email or by telephone (+44(020 7896
    >> 0011
    >> > > >     > <020%207896%200011>) and then
    >> > > >     > > > delete
    >> > > >     > > > > > the
    >> > > >     > > > > >     > email
    >> > > >     > > > > >     > > and any copies of it. Opinions, conclusion
    >> (etc)
    >> > > > that do
    >> > > >     > not
    >> > > >     > > > > > relate to
    >> > > >     > > > > >     > the
    >> > > >     > > > > >     > > official business of this company shall be
    >> > > > understood as
    >> > > >     > > > neither
    >> > > >     > > > > > given
    >> > > >     > > > > >     > nor
    >> > > >     > > > > >     > > endorsed by it. IG is a trading name of IG
    >> > Markets
    >> > > > Limited
    >> > > >     > (a
    >> > > >     > > > > > company
    >> > > >     > > > > >     > > registered in England and Wales, company
    >> number
    >> > > > 04008957)
    >> > > >     > and
    >> > > >     > > > IG
    >> > > >     > > > > > Index
    >> > > >     > > > > >     > > Limited (a company registered in England and
    >> > Wales,
    >> > > > company
    >> > > >     > > > > number
    >> > > >     > > > > >     > > 01190902). Registered address at Cannon Bridge
    >> > > > House, 25
    >> > > >     > > > Dowgate
    >> > > >     > > > > > Hill,
    >> > > >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited
    >> > (register
    >> > > > number
    >> > > >     > > > 195355)
    >> > > >     > > > > > and IG
    >> > > >     > > > > >     > > Index Limited (register number 114059) are
    >> > > > authorised and
    >> > > >     > > > > > regulated by
    >> > > >     > > > > >     > the
    >> > > >     > > > > >     > > Financial Conduct Authority.
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > >
    >> > > >     > > > > >     > The information contained in this email is
    >> strictly
    >> > > >     > > confidential
    >> > > >     > > > > and
    >> > > >     > > > > > for
    >> > > >     > > > > >     > the use of the addressee only, unless otherwise
    >> > > > indicated. If
    >> > > >     > > you
    >> > > >     > > > > > are not
    >> > > >     > > > > >     > the intended recipient, please do not read,
    >> copy,
    >> > use
    >> > > > or
    >> > > >     > > disclose
    >> > > >     > > > > to
    >> > > >     > > > > > others
    >> > > >     > > > > >     > this message or any attachment. Please also
    >> notify
    >> > > the
    >> > > > sender
    >> > > >     > > by
    >> > > >     > > > > > replying
    >> > > >     > > > > >     > to this email or by telephone (+44(020 7896 0011
    >> > > >     > <020%207896%200011>) and then
    >> > > >     > > delete
    >> > > >     > > > > > the email
    >> > > >     > > > > >     > and any copies of it. Opinions, conclusion (etc)
    >> > that
    >> > > > do not
    >> > > >     > > > relate
    >> > > >     > > > > > to the
    >> > > >     > > > > >     > official business of this company shall be
    >> > understood
    >> > > > as
    >> > > >     > > neither
    >> > > >     > > > > > given nor
    >> > > >     > > > > >     > endorsed by it. IG is a trading name of IG
    >> Markets
    >> > > > Limited (a
    >> > > >     > > > > company
    >> > > >     > > > > >     > registered in England and Wales, company number
    >> > > > 04008957) and
    >> > > >     > > IG
    >> > > >     > > > > > Index
    >> > > >     > > > > >     > Limited (a company registered in England and
    >> Wales,
    >> > > > company
    >> > > >     > > > number
    >> > > >     > > > > >     > 01190902). Registered address at Cannon Bridge
    >> > House,
    >> > > > 25
    >> > > >     > > Dowgate
    >> > > >     > > > > > Hill,
    >> > > >     > > > > >     > London EC4R 2YA. Both IG Markets Limited
    >> (register
    >> > > > number
    >> > > >     > > 195355)
    >> > > >     > > > > > and IG
    >> > > >     > > > > >     > Index Limited (register number 114059) are
    >> > authorised
    >> > > > and
    >> > > >     > > > regulated
    >> > > >     > > > > > by the
    >> > > >     > > > > >     > Financial Conduct Authority.
    >> > > >     > > > > >     >
    >> > > >     > > > > >     >
    >> > > >     > > > > >
    >> > > >     > > > > >
    >> > > >     > > > > > The information contained in this email is strictly
    >> > > > confidential
    >> > > >     > and
    >> > > >     > > > for
    >> > > >     > > > > > the use of the addressee only, unless otherwise
    >> > indicated.
    >> > > > If you
    >> > > >     > are
    >> > > >     > > > not
    >> > > >     > > > > > the intended recipient, please do not read, copy, use
    >> or
    >> > > > disclose
    >> > > >     > to
    >> > > >     > > > > others
    >> > > >     > > > > > this message or any attachment. Please also notify the
    >> > > > sender by
    >> > > >     > > > replying
    >> > > >     > > > > > to this email or by telephone (+44(020 7896 0011
    >> > > >     > <020%207896%200011>) and then delete the
    >> > > >     > > > > email
    >> > > >     > > > > > and any copies of it. Opinions, conclusion (etc) that
    >> do
    >> > > not
    >> > > > relate
    >> > > >     > > to
    >> > > >     > > > > the
    >> > > >     > > > > > official business of this company shall be understood
    >> as
    >> > > > neither
    >> > > >     > > given
    >> > > >     > > > > nor
    >> > > >     > > > > > endorsed by it. IG is a trading name of IG Markets
    >> > Limited
    >> > > (a
    >> > > >     > company
    >> > > >     > > > > > registered in England and Wales, company number
    >> 04008957)
    >> > > > and IG
    >> > > >     > > Index
    >> > > >     > > > > > Limited (a company registered in England and Wales,
    >> > company
    >> > > > number
    >> > > >     > > > > > 01190902). Registered address at Cannon Bridge House,
    >> 25
    >> > > > Dowgate
    >> > > >     > > Hill,
    >> > > >     > > > > > London EC4R 2YA. Both IG Markets Limited (register
    >> number
    >> > > > 195355)
    >> > > >     > and
    >> > > >     > > > IG
    >> > > >     > > > > > Index Limited (register number 114059) are authorised
    >> and
    >> > > > regulated
    >> > > >     > > by
    >> > > >     > > > > the
    >> > > >     > > > > > Financial Conduct Authority.
    >> > > >     > > > > >
    >> > > >     > > > > The information contained in this email is strictly
    >> > > > confidential and
    >> > > >     > > for
    >> > > >     > > > > the use of the addressee only, unless otherwise
    >> indicated.
    >> > If
    >> > > > you are
    >> > > >     > > not
    >> > > >     > > > > the intended recipient, please do not read, copy, use or
    >> > > > disclose to
    >> > > >     > > > others
    >> > > >     > > > > this message or any attachment. Please also notify the
    >> > sender
    >> > > > by
    >> > > >     > > replying
    >> > > >     > > > > to this email or by telephone (+44(020 7896 0011
    >> > > > <020%207896%200011>)
    >> > > >     > and then delete the
    >> > > >     > > > email
    >> > > >     > > > > and any copies of it. Opinions, conclusion (etc) that do
    >> > not
    >> > > > relate
    >> > > >     > to
    >> > > >     > > > the
    >> > > >     > > > > official business of this company shall be understood as
    >> > > > neither
    >> > > >     > given
    >> > > >     > > > nor
    >> > > >     > > > > endorsed by it. IG is a trading name of IG Markets
    >> Limited
    >> > (a
    >> > > > company
    >> > > >     > > > > registered in England and Wales, company number
    >> 04008957)
    >> > and
    >> > > > IG
    >> > > >     > Index
    >> > > >     > > > > Limited (a company registered in England and Wales,
    >> company
    >> > > > number
    >> > > >     > > > > 01190902). Registered address at Cannon Bridge House, 25
    >> > > > Dowgate
    >> > > >     > Hill,
    >> > > >     > > > > London EC4R 2YA. Both IG Markets Limited (register
    >> number
    >> > > > 195355) and
    >> > > >     > > IG
    >> > > >     > > > > Index Limited (register number 114059) are authorised
    >> and
    >> > > > regulated
    >> > > >     > by
    >> > > >     > > > the
    >> > > >     > > > > Financial Conduct Authority.
    >> > > >     > > > >
    >> > > >     > > > The information contained in this email is strictly
    >> > > confidential
    >> > > > and
    >> > > >     > for
    >> > > >     > > > the use of the addressee only, unless otherwise
    >> indicated. If
    >> > > > you are
    >> > > >     > not
    >> > > >     > > > the intended recipient, please do not read, copy, use or
    >> > > > disclose to
    >> > > >     > > others
    >> > > >     > > > this message or any attachment. Please also notify the
    >> sender
    >> > > by
    >> > > >     > replying
    >> > > >     > > > to this email or by telephone (+44(020 7896 0011
    >> > > > <020%207896%200011>)
    >> > > >     > and then delete the
    >> > > >     > > email
    >> > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
    >> not
    >> > > > relate to
    >> > > >     > > the
    >> > > >     > > > official business of this company shall be understood as
    >> > > neither
    >> > > > given
    >> > > >     > > nor
    >> > > >     > > > endorsed by it. IG is a trading name of IG Markets
    >> Limited (a
    >> > > > company
    >> > > >     > > > registered in England and Wales, company number 04008957)
    >> and
    >> > > IG
    >> > > > Index
    >> > > >     > > > Limited (a company registered in England and Wales,
    >> company
    >> > > > number
    >> > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
    >> > > Dowgate
    >> > > > Hill,
    >> > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
    >> > > > 195355) and
    >> > > >     > IG
    >> > > >     > > > Index Limited (register number 114059) are authorised and
    >> > > > regulated by
    >> > > >     > > the
    >> > > >     > > > Financial Conduct Authority.
    >> > > >     > > >
    >> > > >     > > The information contained in this email is strictly
    >> > confidential
    >> > > > and for
    >> > > >     > > the use of the addressee only, unless otherwise indicated.
    >> If
    >> > you
    >> > > > are not
    >> > > >     > > the intended recipient, please do not read, copy, use or
    >> > disclose
    >> > > > to
    >> > > >     > others
    >> > > >     > > this message or any attachment. Please also notify the
    >> sender
    >> > by
    >> > > > replying
    >> > > >     > > to this email or by telephone (+44(020 7896 0011
    >> > > > <020%207896%200011>)
    >> > > >     > and then delete the email
    >> > > >     > > and any copies of it. Opinions, conclusion (etc) that do not
    >> > > > relate to
    >> > > >     > the
    >> > > >     > > official business of this company shall be understood as
    >> > neither
    >> > > > given
    >> > > >     > nor
    >> > > >     > > endorsed by it. IG is a trading name of IG Markets Limited
    >> (a
    >> > > > company
    >> > > >     > > registered in England and Wales, company number 04008957)
    >> and
    >> > IG
    >> > > > Index
    >> > > >     > > Limited (a company registered in England and Wales, company
    >> > > number
    >> > > >     > > 01190902). Registered address at Cannon Bridge House, 25
    >> > Dowgate
    >> > > > Hill,
    >> > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
    >> > 195355)
    >> > > > and IG
    >> > > >     > > Index Limited (register number 114059) are authorised and
    >> > > > regulated by
    >> > > >     > the
    >> > > >     > > Financial Conduct Authority.
    >> > > >     > >
    >> > > >     >
    >> > > >
    >> > > >
    >> > > > The information contained in this email is strictly confidential and
    >> > for
    >> > > > the use of the addressee only, unless otherwise indicated. If you
    >> are
    >> > not
    >> > > > the intended recipient, please do not read, copy, use or disclose to
    >> > > others
    >> > > > this message or any attachment. Please also notify the sender by
    >> > replying
    >> > > > to this email or by telephone (+44(020 7896 0011) and then delete
    >> the
    >> > > email
    >> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
    >> to
    >> > > the
    >> > > > official business of this company shall be understood as neither
    >> given
    >> > > nor
    >> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
    >> company
    >> > > > registered in England and Wales, company number 04008957) and IG
    >> Index
    >> > > > Limited (a company registered in England and Wales, company number
    >> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    >> Hill,
    >> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
    >> and
    >> > IG
    >> > > > Index Limited (register number 114059) are authorised and regulated
    >> by
    >> > > the
    >> > > > Financial Conduct Authority.
    >> > > >
    >> > > The information contained in this email is strictly confidential and
    >> for
    >> > > the use of the addressee only, unless otherwise indicated. If you are
    >> not
    >> > > the intended recipient, please do not read, copy, use or disclose to
    >> > others
    >> > > this message or any attachment. Please also notify the sender by
    >> replying
    >> > > to this email or by telephone (+44(020 7896 0011) and then delete the
    >> > email
    >> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
    >> > the
    >> > > official business of this company shall be understood as neither given
    >> > nor
    >> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    >> > > registered in England and Wales, company number 04008957) and IG Index
    >> > > Limited (a company registered in England and Wales, company number
    >> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    >> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    >> IG
    >> > > Index Limited (register number 114059) are authorised and regulated by
    >> > the
    >> > > Financial Conduct Authority.
    >> > >
    >> > The information contained in this email is strictly confidential and for
    >> > the use of the addressee only, unless otherwise indicated. If you are
    >> not
    >> > the intended recipient, please do not read, copy, use or disclose to
    >> others
    >> > this message or any attachment. Please also notify the sender by
    >> replying
    >> > to this email or by telephone (+44(020 7896 0011) and then delete the
    >> email
    >> > and any copies of it. Opinions, conclusion (etc) that do not relate to
    >> the
    >> > official business of this company shall be understood as neither given
    >> nor
    >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
    >> > registered in England and Wales, company number 04008957) and IG Index
    >> > Limited (a company registered in England and Wales, company number
    >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    >> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    >> > Index Limited (register number 114059) are authorised and regulated by
    >> the
    >> > Financial Conduct Authority.
    >> >
    >>
    >
    >
    The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
    

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Agreed. As I said there is still a use case just I wouldn't be pushing for it, the. Need for it reduces for me.

Sent using OWA for iPhone
________________________________________
From: radai <ra...@gmail.com>
Sent: Tuesday, December 20, 2016 3:51:33 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

i didnt mean to sound as insisting. what i actually mean is it would still
be a valid issue but of much less concern.

On Mon, Dec 19, 2016 at 7:50 PM, radai <ra...@gmail.com> wrote:

> this kip fixes a "bug" (quirk?) that arises when people implement headers
> "in V" (in the payload part of a message).
> if you have proper headers you obviously dont need to to stick them in V
> and so wont run into this, but its still a valid issue.
>
> On Mon, Dec 19, 2016 at 3:06 PM, Jay Kreps <ja...@confluent.io> wrote:
>
>> Makes sense!
>>
>> -Jay
>>
>> On Mon, Dec 19, 2016 at 2:40 PM, Michael Pearce <Mi...@ig.com>
>> wrote:
>>
>> > Wow just read that def over tired. Hopefully it makes sense. Or you get
>> > the gist at least.
>> >
>> > ________________________________________
>> > From: Michael Pearce <Mi...@ig.com>
>> > Sent: Monday, December 19, 2016 9:19:02 PM
>> > To: dev@kafka.apache.org
>> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> >
>> > Hi Jay,
>> >
>> > Agreed this stemmed as offshoot from KIP-82.
>> >
>> > Which our main driver for was to be able to have some headers for a null
>> > value as such for our routing, audit, tracing and a few other bits which
>> > currently we are forced to do with a message wrapper, if we all agreed
>> on
>> > KIP-82 that we need native headers and look to implement that the push
>> for
>> > this would dissipate.
>> >
>> > This KIP would allow for though one use case that comes to mind we could
>> > see which is to have business data with a delete. Though as said this
>> isn't
>> > something we are pushing for think really we would have.
>> >
>> > As such in summary yes, if you want to fully support KIP-82 and we can
>> get
>> > that agreed in principle and a target release version, I think quite a
>> few
>> > guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.
>> >
>> > Cheers
>> > Mike
>> >
>> > Sent using OWA for iPhone
>> > ________________________________________
>> > From: Jay Kreps <ja...@confluent.io>
>> > Sent: Monday, December 19, 2016 8:51:23 PM
>> > To: dev@kafka.apache.org
>> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> >
>> > Hey Michael,
>> >
>> > Here is the compatibility concern I have:
>> >
>> >    1. You have a consumer app that relies on value == null to indicate a
>> >    delete (current semantics).
>> >    2. You upgrade Kafka and your clients.
>> >    3. Some producer starts using the tombstone field in combination with
>> >    non-null.
>> >
>> > I share Ismael's dislike of setting tombstones on records with null
>> values.
>> > This makes sense as a transitional state, but as an end state its a bit
>> > weird. You'd expect to be able to mix null values and tombstones, and
>> have
>> > the null values remain and the tombstones get compacted. However what
>> will
>> > happen is both will be compacted and upon debugging this you'll learn
>> that
>> > we sometimes use null in the value to indicate tombstone. Ismael's
>> solution
>> > is a bigger compatibility break, though, so not sure if that is better.
>> >
>> > My other question is the relationship to KIP-82. My read is that this
>> KIP
>> > solves some but not all of the problems KIP-82 is intended for. KIP-82,
>> on
>> > the other hand, seems to address most of the motivating uses for this
>> KIP.
>> > The exception is maybe item (5) on the list where you want to
>> simultaneous
>> > delete and convey some information to subscribers, but I couldn't
>> construct
>> > a concrete examples for that one. Do we need to rationalize these two
>> KIPs?
>> > That is, do you still advocate this proposal if we do KIP-82 and vice
>> > versa? As you may have noticed I'm somewhat emotionally invested in the
>> > simplicity of the core data model, so my default position is let's try
>> to
>> > avoid stuffing more stuff in, but if we have to add stuff I like each of
>> > these individually more than doing both. :-)
>> >
>> > -Jay
>> >
>> >
>> >
>> >
>> > On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Michael.Pearce@ig.com
>> >
>> > wrote:
>> >
>> > > Hi Jay
>> > >
>> > > I disagree here that we are breaking any compatibility, we went
>> through
>> > > this on the discussion thread in fact with the help of that thread is
>> how
>> > > the kip came to the solution.
>> > >
>> > > Also on the supported combinations front you mention, we are not
>> taking
>> > > anything away afaik.
>> > >
>> > > Currently supported are only are:
>> > > Null value = delete
>> > > Non-null value = non delete
>> > >
>> > > With this kip we would support
>> > > Null value + tombstone = delete
>> > > Non null value + tombstone = delete
>> > > Non null value + no tombstone = non delete
>> > >
>> > > As for the alternative idea, this is simply a new policy, how can
>> there
>> > be
>> > > confusion here? For this policy it would be explicit that tombstone
>> > marker
>> > > would need to be set for a delete.
>> > >
>> > > I'm going to vent a little now as starting to get quite frustrated.
>> > >
>> > > We are going round in circles on kip-82 as per use cases there is now
>> > many
>> > > use cases, how many more are needed? just because confluent don't see
>> > these
>> > > doesn't mean they aren't real use cases other have, this is the point
>> of
>> > > the Apache foundation, it shouldn't be the view of just one
>> organisation.
>> > > It really is getting a feeling of the NIH syndrome. Rather than it
>> being
>> > > constructive on discussion of the implementation detail.
>> > >
>> > > kip-87 spawned from as on the kip call we all agreed this was needed.
>> And
>> > > would at least allow a custom wrapper be supported in a compacted
>> topic,
>> > > allowing meta data. Which again now I feel we are spinning wheels, and
>> > > simply finding reasons not support it.
>> > >
>> > > Cheers
>> > > Mike
>> > >
>> > >
>> > >
>> > > Sent using OWA for iPad
>> > > ________________________________________
>> > > From: Jay Kreps <ja...@confluent.io>
>> > > Sent: Friday, December 16, 2016 7:09:23 PM
>> > > To: dev@kafka.apache.org
>> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > >
>> > > Hey Michael,
>> > >
>> > > I do think it might have been better had we started with a separate
>> > concept
>> > > of null vs delete. Given that we are where we are, I'm not sure that
>> the
>> > > possible cures we explored so far are better than the disease.
>> > >
>> > > I apologize for coming to this late, but I didn't really understand
>> the
>> > > implications of the KIP and that we'd be breaking compatibility with
>> > > existing apps until the vote had begun, and in my defense I'm not sure
>> > the
>> > > other folks voting did either.
>> > >
>> > > I think we all agree there are many existing apps that are built with
>> the
>> > > assumption of "null value non-tombstone" and it isn't possible to
>> > > disambiguate these from tombstones on the producer. It isn't that
>> anyone
>> > is
>> > > saying we have to support all four possibilities at once, it's that we
>> > > simply can't orphan one of the existing combinations or our users will
>> > eat
>> > > us!
>> > >
>> > > If I've understood your alternate solution of adding another setting
>> for
>> > > compaction, I think this does fix the compatibility problem, but it
>> adds
>> > an
>> > > odd mode the user has to add on all their topics. While the current
>> state
>> > > is easily explainable, the resulting state where the meaning of
>> tombstone
>> > > and null are overlapping and ambiguous and dependent on a compaction
>> > > setting that could change out of band or not be in sync with your
>> code in
>> > > some environment seems worse to me then where we currently are. I
>> think
>> > the
>> > > question is how would this combination be explained to users and does
>> it
>> > > make sense?
>> > >
>> > > -Jay
>> > >
>> > >
>> > >
>> > > On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > Hi Chaps,
>> > > >
>> > > > Can we either get one more +1 binding (we have 2 already) on the
>> > > existing?
>> > > >
>> > > > Or have some response on the below possible alternative. We are
>> keen to
>> > > > get working on this, so we make next feature release.
>> > > >
>> > > > Cheers
>> > > > Mike
>> > > >
>> > > >
>> > > > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com>
>> wrote:
>> > > >
>> > > >     Hi Ismael
>> > > >
>> > > >     Did you see our email this morning, what's your thoughts on this
>> > > > approach to instead we simply have a brand new policy?
>> > > >
>> > > >     Cheers
>> > > >     Mike
>> > > >
>> > > >
>> > > >     Sent using OWA for iPhone
>> > > >     ________________________________________
>> > > >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
>> > > Juma <
>> > > > ismael@juma.me.uk>
>> > > >     Sent: Tuesday, December 13, 2016 11:30:05 AM
>> > > >     To: dev@kafka.apache.org
>> > > >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > >
>> > > >     Yes, this is actually tricky to do in a way where we both have
>> the
>> > > > desired
>> > > >     semantics and maintain compatibility. When someone creates a
>> > > >     `ProducerRecord` with a `null` value today, the producer doesn't
>> > know
>> > > > if
>> > > >     it's meant to be a tombstone or not. For V3 messages, it's easy
>> > when
>> > > > the
>> > > >     constructor that takes a tombstone is used. However, if any
>> other
>> > > >     constructor is used, it's not clear. A couple of options I can
>> > think
>> > > > of,
>> > > >     none of them particularly nice:
>> > > >
>> > > >     1. Have a third state where tombstone = unknown and the broker
>> > would
>> > > > set
>> > > >     the tombstone bit if the value was null and the topic was
>> > compacted.
>> > > > People
>> > > >     that wanted to pass a non-null value for the tombstone would
>> have
>> > to
>> > > > use
>> > > >     the constructor that takes a tombstone. The drawbacks: third
>> state
>> > > for
>> > > >     tombstone in message format, message conversion at the broker
>> for a
>> > > > common
>> > > >     case.
>> > > >
>> > > >     2. Extend MetadataResponse to optionally include topic configs,
>> > which
>> > > > would
>> > > >     make it possible for the producer to be smarter about setting
>> the
>> > > >     tombstone. It would only do it if a tombstone was not passed
>> > > > explicitly,
>> > > >     the value was null and the topic was compacted. The main
>> drawback
>> > is
>> > > > that
>> > > >     the producer would be getting a bit more data for each topic
>> even
>> > > > though it
>> > > >     probably won't use it most of the time. Extending
>> MetadataResponse
>> > to
>> > > >     return topic configs would be useful for other reasons as well,
>> so
>> > > that
>> > > >     part seems OK.
>> > > >
>> > > >     In addition, for both proposals, we could consider adding
>> warnings
>> > to
>> > > > the
>> > > >     documentation that the behaviour of the constructors that don't
>> > take
>> > > a
>> > > >     tombstone would change in the next major release so that
>> tombstone
>> > =
>> > > > false.
>> > > >     Not sure if this would be worth it though.
>> > > >
>> > > >     Ismael
>> > > >
>> > > >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
>> > > > ewen@confluent.io>
>> > > >     wrote:
>> > > >
>> > > >     > Michael,
>> > > >     >
>> > > >     > It kind of depends on how you want to interpret the tombstone
>> > flag.
>> > > > If it's
>> > > >     > purely a producer-facing Kafka-level thing that we treat as
>> > > internal
>> > > > to the
>> > > >     > broker and log cleaner once the record is sent, then your
>> > approach
>> > > > makes
>> > > >     > sense. You're just moving copying the null-indicates-delete
>> > > behavior
>> > > > of the
>> > > >     > old constructor into the tombstone flag.
>> > > >     >
>> > > >     > However, if you want this change to more generally decouple
>> the
>> > > idea
>> > > > of
>> > > >     > deletion and null values, then you are sometimes converting
>> what
>> > > > might be a
>> > > >     > completely valid null value that doesn't indicate deletion
>> into a
>> > > >     > tombstone. Downstream applications could potentially handle
>> these
>> > > > cases
>> > > >     > differently given the separation of deletion from value.
>> > > >     >
>> > > >     > I guess the question is if we want to try to support the
>> latter
>> > > even
>> > > > for
>> > > >     > topics where we have older produce requests. An example where
>> > this
>> > > > could
>> > > >     > come up is in something like a CDC Connector. If we try to
>> > support
>> > > > the
>> > > >     > semantic difference, a connector might write changes to Kafka
>> > using
>> > > > the
>> > > >     > tombstone flag to indicate when a row was truly deleted (vs an
>> > > > update that
>> > > >     > sets it to null but still present; this probably makes more
>> sense
>> > > > for CDC
>> > > >     > from document stores or extracting single columns). There are
>> > > various
>> > > >     > reasons we might want to maintain the full log and not turn
>> > > > compaction on
>> > > >     > (or just use a time-based retention policy), but downstream
>> > > > applications
>> > > >     > might care to know the difference between a delete and a null
>> > > value.
>> > > > In
>> > > >     > fact both versions of the same log (compacted and
>> time-retention)
>> > > > could be
>> > > >     > useful and I don't think it'll be uncommon to maintain both or
>> > use
>> > > > KIP-71
>> > > >     > to maintain a hybrid compacted/retention topic.
>> > > >     >
>> > > >     > -Ewen
>> > > >     >
>> > > >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > >     > wrote:
>> > > >     >
>> > > >     > > Hi Jay,
>> > > >     > >
>> > > >     > > Why wouldn't that work, the tombstone value is only looked
>> at
>> > by
>> > > > the
>> > > >     > > broker, on a topic configured for compaction as such is
>> benign
>> > on
>> > > > non
>> > > >     > > compacted topics. This is as much as sending a null value
>> > > currently
>> > > >     > >
>> > > >     > >
>> > > >     > > Regards
>> > > >     > > Mike
>> > > >     > >
>> > > >     > >
>> > > >     > >
>> > > >     > > Sent using OWA for iPhone
>> > > >     > > ________________________________________
>> > > >     > > From: Jay Kreps <ja...@confluent.io>
>> > > >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
>> > > >     > > To: dev@kafka.apache.org
>> > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > >     > >
>> > > >     > > Hey Michael,
>> > > >     > >
>> > > >     > > I'm not quite sure that works as that would translate ALL
>> null
>> > > > values to
>> > > >     > > tombstones, even for non-compacted topics that use null as
>> an
>> > > > acceptable
>> > > >     > > value sent by the producer and expected by the consumer.
>> > > >     > >
>> > > >     > > -Jay
>> > > >     > >
>> > > >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > >     > > wrote:
>> > > >     > >
>> > > >     > > > Hi Ewen,
>> > > >     > > >
>> > > >     > > > I think the easiest way to show this is with code.
>> > > >     > > >
>> > > >     > > > As you can see we keep the existing behaviour for
>> > code/binaries
>> > > > calling
>> > > >     > > > the pre-existing constructors, whereby if the value is
>> null
>> > the
>> > > >     > tombstone
>> > > >     > > > is set to true.
>> > > >     > > >
>> > > >     > > > Regards
>> > > >     > > > Mike
>> > > >     > > >
>> > > >     > > >
>> > > >     > > >
>> > > >     > > >     /**
>> > > >     > > >      * Creates a record with a specified timestamp to be
>> sent
>> > > to
>> > > > a
>> > > >     > > > specified topic and partition
>> > > >     > > >      *
>> > > >     > > >      * @param topic The topic the record will be appended
>> to
>> > > >     > > >      * @param partition The partition to which the record
>> > > should
>> > > > be
>> > > >     > sent
>> > > >     > > >      * @param timestamp The timestamp of the record
>> > > >     > > >      * @param tombstone if the record should be treated
>> as a
>> > > > tombstone
>> > > >     > if
>> > > >     > > > the topic is compacted
>> > > >     > > >      * @param key The key that will be included in the
>> record
>> > > >     > > >      * @param value The record contents
>> > > >     > > >      */
>> > > >     > > >     public ProducerRecord(String topic, Integer partition,
>> > > > Boolean
>> > > >     > > > tombstone, Long timestamp, K key, V value) {
>> > > >     > > >         if (topic == null)
>> > > >     > > >             throw new IllegalArgumentException("Topic
>> cannot
>> > > be
>> > > >     > null.");
>> > > >     > > >         if (timestamp != null && timestamp < 0)
>> > > >     > > >             throw new IllegalArgumentException(
>> > > >     > > >                     String.format("Invalid timestamp: %d.
>> > > > Timestamp
>> > > >     > > should
>> > > >     > > > always be non-negative or null.", timestamp));
>> > > >     > > >         if (partition != null && partition < 0)
>> > > >     > > >             throw new IllegalArgumentException(
>> > > >     > > >                     String.format("Invalid partition: %d.
>> > > > Partition
>> > > >     > > number
>> > > >     > > > should always be non-negative or null.", partition));
>> > > >     > > >         if (!tombstone && value == null){
>> > > >     > > >             throw new IllegalArgumentException(
>> > > >     > > >                     String.format("Tombstone must be true
>> if
>> > > null
>> > > >     > > value"));
>> > > >     > > >         }
>> > > >     > > >         this.topic = topic;
>> > > >     > > >         this.partition = partition;
>> > > >     > > >         this.tombstone = tombstone;
>> > > >     > > >         this.key = key;
>> > > >     > > >         this.value = value;
>> > > >     > > >         this.timestamp = timestamp;
>> > > >     > > >     }
>> > > >     > > >
>> > > >     > > >     public ProducerRecord(String topic, Integer partition,
>> > Long
>> > > >     > > timestamp,
>> > > >     > > > K key, V value) {
>> > > >     > > >         this(topic, partition, value == null, timestamp,
>> key,
>> > > > value);
>> > > >     > > >     }
>> > > >     > > > ________________________________________
>> > > >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
>> > > >     > > > Sent: Sunday, December 11, 2016 5:45 AM
>> > > >     > > > To: dev@kafka.apache.org
>> > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > >     > > >
>> > > >     > > > It seemed like this was addressed in the migration
>> section,
>> > > > wasn't it?
>> > > >     > > The
>> > > >     > > > V2 vs V3 requests and the fact that the broker will
>> downgrade
>> > > the
>> > > >     > message
>> > > >     > > > format (losing zero copy) if you issues a V2 request to a
>> > > broker
>> > > > using
>> > > >     > V3
>> > > >     > > > format handles compatibility, does it not? The existing
>> > > > consumers won't
>> > > >     > > see
>> > > >     > > > the extra metadata in the value, but they will get a null
>> > > > instead and
>> > > >     > > treat
>> > > >     > > > it as a tombstone. Certainly there is a performance
>> impact,
>> > but
>> > > > it
>> > > >     > seemed
>> > > >     > > > compatible.
>> > > >     > > >
>> > > >     > > > I'm worried about this though:
>> > > >     > > >
>> > > >     > > >
>> > > >     > > >    - *NOTE *: With the new version of producer client
>> using
>> > > >     > > ProduceRequest
>> > > >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not
>> > set)
>> > > > and
>> > > >     > null
>> > > >     > > > value
>> > > >     > > >    should not be allowed as the older version of consumer
>> > using
>> > > >     > > > FetchRequest
>> > > >     > > >    V2 will think of this as a tombstone when its actually
>> > not.
>> > > >     > > >
>> > > >     > > >
>> > > >     > > > Unless I'm misunderstanding, this ends up breaking binary
>> > > > compatibility
>> > > >     > > for
>> > > >     > > > the Java API. It sounds like this suggests that passing a
>> > null
>> > > > value to
>> > > >     > > the
>> > > >     > > > existing ProducerRecord constructors would generate an
>> > > exception
>> > > > since
>> > > >     > > you
>> > > >     > > > didn't explicitly enable the tombstone (via whatever new
>> > > > constructor is
>> > > >     > > > provided). But that means you can't swap in a newer client
>> > jar
>> > > > without
>> > > >     > > > recompiling and get the same behavior if your app deletes
>> > keys
>> > > > using
>> > > >     > the
>> > > >     > > > current approach because your app will start throwing
>> > > > exceptions. Maybe
>> > > >     > > > this is a tradeoff we're ok with, but we've tried pretty
>> hard
>> > > to
>> > > > avoid
>> > > >     > > > breaking compatibility like this so far -- it makes
>> picking
>> > up
>> > > > bug
>> > > >     > fixes
>> > > >     > > in
>> > > >     > > > the clients harder for users of frameworks that have to
>> pin
>> > to
>> > > > earlier
>> > > >     > > > default versions for compatibility.
>> > > >     > > >
>> > > >     > > > But then later the KIP says:
>> > > >     > > >
>> > > >     > > >
>> > > >     > > >    - The old constructors for ProducerRecord without this
>> > > > parameter
>> > > >     > will
>> > > >     > > be
>> > > >     > > >    kept but updated so that their default behaviour if
>> > setting
>> > > > null
>> > > >     > value
>> > > >     > > > will
>> > > >     > > >    be the tombstone will be set to true to keep existing
>> > > > behaviour.
>> > > >     > > >
>> > > >     > > >
>> > > >     > > > So maybe I am misinterpreting something.
>> > > >     > > >
>> > > >     > > > And just a nit re: motivation section, item 6 would be
>> > clearer
>> > > > for a
>> > > >     > > union
>> > > >     > > > schema with null (or optional schemas in other formats),
>> e.g.
>> > > > [null,
>> > > >     > > > string], in which case losing the schema truly is losing
>> > > > information
>> > > >     > > > (whereas null is already the only valid value for a pure
>> null
>> > > > schema).
>> > > >     > > >
>> > > >     > > > -Ewen
>> > > >     > > >
>> > > >     > > >
>> > > >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com
>> > > >     > >
>> > > >     > > > wrote:
>> > > >     > > >
>> > > >     > > > > Hi Jay,
>> > > >     > > > >
>> > > >     > > > > Good point this detail is missing in the KIP write up.
>> Ive
>> > > > added this
>> > > >     > > > now.
>> > > >     > > > >
>> > > >     > > > > Essentially simply just upgrading the clients we do not
>> > > expect
>> > > > any
>> > > >     > > client
>> > > >     > > > > app code change needed.
>> > > >     > > > >
>> > > >     > > > > Cheers
>> > > >     > > > > Mike
>> > > >     > > > > ________________________________________
>> > > >     > > > > From: Jay Kreps <ja...@confluent.io>
>> > > >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
>> > > >     > > > > To: dev@kafka.apache.org
>> > > >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
>> Flag
>> > > >     > > > >
>> > > >     > > > > Michael,
>> > > >     > > > >
>> > > >     > > > > The compatibility section goes through the migration
>> path,
>> > > but
>> > > > isn't
>> > > >     > > the
>> > > >     > > > > bigger compatibility issue with existing apps? There are
>> > many
>> > > >     > (probably
>> > > >     > > > > thousands) of apps in production that use this feature
>> and
>> > > > send null
>> > > >     > to
>> > > >     > > > > mean delete. It seems like this would break
>> compatibility
>> > > with
>> > > > them,
>> > > >     > > and
>> > > >     > > > > they would have to be rewritten to right?
>> > > >     > > > >
>> > > >     > > > > -Jay
>> > > >     > > > >
>> > > >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
>> > > >     > Michael.Pearce@ig.com
>> > > >     > > >
>> > > >     > > > > wrote:
>> > > >     > > > >
>> > > >     > > > > > Hi Jun,
>> > > >     > > > > >
>> > > >     > > > > > 4) On v3 we honour the tombstone. As such we expect
>> it to
>> > > be
>> > > > set
>> > > >     > > > > correctly
>> > > >     > > > > > as per the KIP.
>> > > >     > > > > >
>> > > >     > > > > > 4.1) why would we want to produce an error when its
>> v3?
>> > > This
>> > > > is the
>> > > >     > > > exact
>> > > >     > > > > > purpose to support non-null tombstone's
>> > > >     > > > > > 4.2) again here im unclear on the question on the v3,
>> > > produce
>> > > >     > request
>> > > >     > > > we
>> > > >     > > > > > expect the tombstone flag to be set correctly.
>> > > >     > > > > >
>> > > >     > > > > > 4.4) compaction only occurs on compacted topics, the
>> bit
>> > > > makes no
>> > > >     > > > > > difference and not looked at on un-compacted
>> (time/size
>> > > based
>> > > >     > > > eviction).
>> > > >     > > > > >
>> > > >     > > > > >
>> > > >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io>
>> > wrote:
>> > > >     > > > > >
>> > > >     > > > > >     Hi, Michael,
>> > > >     > > > > >
>> > > >     > > > > >     4. Then, I think I misunderstood this point. Could
>> > you
>> > > > document
>> > > >     > > the
>> > > >     > > > > >     following points in the wiki?
>> > > >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
>> > > > non-null
>> > > >     > value,
>> > > >     > > > > does
>> > > >     > > > > > the
>> > > >     > > > > >     send() get an error or does the producer
>> > automatically
>> > > > set the
>> > > >     > > > value
>> > > >     > > > > to
>> > > >     > > > > >     null?
>> > > >     > > > > >     4.2 If producer V3 doesn't set tombstone, but
>> > provides
>> > > a
>> > > > null
>> > > >     > > > value,
>> > > >     > > > > > does
>> > > >     > > > > >     the send() get an error or does the producer
>> > > > automatically sets
>> > > >     > > the
>> > > >     > > > > >     tombstone?
>> > > >     > > > > >     4.3 Does the broker only expect messages that (a)
>> > have
>> > > no
>> > > >     > > tombstone
>> > > >     > > > > and
>> > > >     > > > > >     non-null value; (b) have tombstone and null value
>> and
>> > > > reject
>> > > >     > the
>> > > >     > > > > > messages
>> > > >     > > > > >     with an error code otherwise?
>> > > >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic
>> is
>> > > > compacted
>> > > >     > on
>> > > >     > > > > not?
>> > > >     > > > > >
>> > > >     > > > > >     Thanks,
>> > > >     > > > > >
>> > > >     > > > > >     Jun
>> > > >     > > > > >
>> > > >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
>> > > >     > > > > Michael.Pearce@ig.com
>> > > >     > > > > > >
>> > > >     > > > > >     wrote:
>> > > >     > > > > >
>> > > >     > > > > >     > Not at all.  This only acts on compacted topics
>> > just
>> > > > as what
>> > > >     > > > occurs
>> > > >     > > > > > today
>> > > >     > > > > >     >
>> > > >     > > > > >     > Sent using OWA for iPhone
>> > > >     > > > > >     > ________________________________________
>> > > >     > > > > >     > From: Jun Rao <ju...@confluent.io>
>> > > >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
>> > > >     > > > > >     > To: dev@kafka.apache.org
>> > > >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > Tombstone
>> > > > Flag
>> > > >     > > > > >     >
>> > > >     > > > > >     > Hi, Michael,
>> > > >     > > > > >     >
>> > > >     > > > > >     > 4. Hmm, does that mean the new client library
>> can
>> > > > never send
>> > > >     > a
>> > > >     > > > null
>> > > >     > > > > > message
>> > > >     > > > > >     > even to a regular topic? This seems like a
>> change
>> > of
>> > > > the
>> > > >     > > existing
>> > > >     > > > > > behavior.
>> > > >     > > > > >     >
>> > > >     > > > > >     > Thanks,
>> > > >     > > > > >     >
>> > > >     > > > > >     > Jun
>> > > >     > > > > >     >
>> > > >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
>> > > >     > > > > > Michael.Pearce@ig.com>
>> > > >     > > > > >     > wrote:
>> > > >     > > > > >     >
>> > > >     > > > > >     > > Hi Jun,
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Re 4) That's because we expect the tombstone
>> > value
>> > > > to be
>> > > >     > set
>> > > >     > > > > > correctly if
>> > > >     > > > > >     > > message bit is 2, as such if an older client
>> > sends
>> > > > in on
>> > > >     > old
>> > > >     > > > > > message the
>> > > >     > > > > >     > > message is upcast and the bit is set
>> correctly.
>> > And
>> > > > such no
>> > > >     > > > > longer
>> > > >     > > > > > need
>> > > >     > > > > >     > to
>> > > >     > > > > >     > > check the value. Mayuresh can you confirm my
>> > > > thinking and
>> > > >     > > > > > understanding
>> > > >     > > > > >     > of
>> > > >     > > > > >     > > what we've discussed?
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > The second point I understand what you're
>> getting
>> > > at
>> > > > now my
>> > > >     > > > > > apologies.
>> > > >     > > > > >     > Yes
>> > > >     > > > > >     > > this makes sense to save on touching the
>> message,
>> > > if
>> > > > we're
>> > > >     > > the
>> > > >     > > > > > only kip
>> > > >     > > > > >     > > going in, in this release.
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Cheers
>> > > >     > > > > >     > > Mike
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Sent using OWA for iPhone
>> > > >     > > > > >     > > ________________________________________
>> > > >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
>> > > >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>> > > >     > > > > >     > > To: dev@kafka.apache.org
>> > > >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > > > Tombstone Flag
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Hi, Michael,
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > 4. Is this updated in the wiki? The text "If
>> the
>> > > > magic byte
>> > > >     > > on
>> > > >     > > > > > message is
>> > > >     > > > > >     > > 2, the broker should use the tombstone bit for
>> > log
>> > > >     > > compaction."
>> > > >     > > > > > doesn't
>> > > >     > > > > >     > > seem to have changed.
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > 2. My point is that if we change the message
>> > format
>> > > > just
>> > > >     > for
>> > > >     > > > this
>> > > >     > > > > > KIP, we
>> > > >     > > > > >     > > should consider whether it's worth optimizing
>> the
>> > > > down
>> > > >     > > > conversion
>> > > >     > > > > > path
>> > > >     > > > > >     > > (i.e., decide whether a conversion is needed
>> by
>> > > just
>> > > >     > looking
>> > > >     > > at
>> > > >     > > > > the
>> > > >     > > > > >     > > tombstone bit in the wrapper message) since
>> > > > tombstone will
>> > > >     > be
>> > > >     > > > > used
>> > > >     > > > > >     > rarely.
>> > > >     > > > > >     > > However, if the message format change here is
>> > > > combined with
>> > > >     > > > other
>> > > >     > > > > > KIPs,
>> > > >     > > > > >     > > then this optimization likely won't be needed.
>> > The
>> > > > latter
>> > > >     > > > > probably
>> > > >     > > > > > makes
>> > > >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do
>> you
>> > > > think?
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Other than those, +1 from me,
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Thanks,
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Jun
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael
>> Pearce <
>> > > >     > > > > > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > wrote:
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > > Hi Jun
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > do we have your vote on this now?
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Any other concerns?
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Cheers
>> > > >     > > > > >     > > > Mike
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Sent using OWA for iPhone
>> > > >     > > > > >     > > > ________________________________________
>> > > >     > > > > >     > > > From: Michael Pearce <Michael.Pearce@ig.com
>> >
>> > > >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>> > > >     > > > > >     > > > To: dev@kafka.apache.org
>> > > >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > > > Tombstone
>> > > >     > Flag
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Hi Jun,
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Have updated. Thanks again for the feedback.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Agree yes we should align up when it gets to
>> > > that,
>> > > > I
>> > > >     > assume
>> > > >     > > > > > you've
>> > > >     > > > > >     > > flagged
>> > > >     > > > > >     > > > the same in the other KIP?
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > I think for now let's get this KIP approved,
>> > then
>> > > > we can
>> > > >     > > > start
>> > > >     > > > > > the work
>> > > >     > > > > >     > > to
>> > > >     > > > > >     > > > get an initial PR, then we can discuss how
>> to
>> > > > align the
>> > > >     > two
>> > > >     > > > if
>> > > >     > > > > > needed.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Cheers,
>> > > >     > > > > >     > > > Mike
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
>> > > jun@confluent.io>
>> > > >     > wrote:
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     Hi, Michael,
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     For 2), this is fine. Could you update
>> the
>> > > KIP
>> > > > wiki
>> > > >     > to
>> > > >     > > > make
>> > > >     > > > > > this
>> > > >     > > > > >     > and
>> > > >     > > > > >     > > > other
>> > > >     > > > > >     > > >     points clearer? Other than that, the KIP
>> > > looks
>> > > > good
>> > > >     > to
>> > > >     > > > me.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     An orthogonal thing is that there are
>> other
>> > > > KIPs such
>> > > >     > > as
>> > > >     > > > > > KIP-98
>> > > >     > > > > >     > that
>> > > >     > > > > >     > > > also
>> > > >     > > > > >     > > >     intend to change the message format. If
>> > they
>> > > > all get
>> > > >     > > > > > approved, we
>> > > >     > > > > >     > > > should
>> > > >     > > > > >     > > >     think about whether it's better to just
>> > bump
>> > > > up the
>> > > >     > > magic
>> > > >     > > > > > byte once
>> > > >     > > > > >     > > to
>> > > >     > > > > >     > > >     incorporate multiple format changes
>> like we
>> > > > did in
>> > > >     > > > > > KIP-31/KIP-32.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     Thanks,
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     Jun
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
>> > > Pearce
>> > > > <
>> > > >     > > > > >     > > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > >     wrote:
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     > Hi Jao
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > Thanks for the response. Sorry for
>> slow
>> > > > reply, both
>> > > >     > > > with
>> > > >     > > > > > personal
>> > > >     > > > > >     > > > sickness
>> > > >     > > > > >     > > >     > and also battling some critical issues
>> > > > encountered
>> > > >     > > > since
>> > > >     > > > > >     > upgrading
>> > > >     > > > > >     > > to
>> > > >     > > > > >     > > >     > 0.10.1.0
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > 1) Thans for spotting, Document error
>> > where
>> > > > we
>> > > >     > > branched
>> > > >     > > > > > this KIP
>> > > >     > > > > >     > > from
>> > > >     > > > > >     > > >     > KIP-82, will get that removed.
>> > > >     > > > > >     > > >     > 2) Intent is to do this just at the
>> > record
>> > > > message
>> > > >     > > > level.
>> > > >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure
>> this
>> > is
>> > > >     > > corrected.
>> > > >     > > > > >     > > >     > 4) As per discussion thread we will
>> > support
>> > > >     > > tombstone +
>> > > >     > > > > > null
>> > > >     > > > > >     > value,
>> > > >     > > > > >     > > >     > tombstone + non null value, no
>> tombstone
>> > +
>> > > > null
>> > > >     > > value.
>> > > >     > > > > >     > > >     > 5) I believe this was in the
>> discussion
>> > > > thread,
>> > > >     > > > @Mayuresh
>> > > >     > > > > > is this
>> > > >     > > > > >     > > >     > something we've overlooked? I thought
>> we
>> > > > would down
>> > > >     > > > > > convert and
>> > > >     > > > > >     > > > remove the
>> > > >     > > > > >     > > >     > value so the old consumer had existing
>> > > > behavior, or
>> > > >     > > is
>> > > >     > > > > > there
>> > > >     > > > > >     > > > something we
>> > > >     > > > > >     > > >     > haven't thought about?
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > Cheers
>> > > >     > > > > >     > > >     > Mike
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
>> > > > jun@confluent.io>
>> > > >     > > > > wrote:
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     Hi, Michael,
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     Thanks for the KIP. A few comments
>> > > below.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     1. The message format change
>> contains
>> > > >     > > > "HeadersLength
>> > > >     > > > > >     > Headers".
>> > > >     > > > > >     > > > Is that
>> > > >     > > > > >     > > >     >     intended?
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     2. For compressed messageset, is
>> the
>> > > > tombstone
>> > > >     > > bit
>> > > >     > > > > > only set
>> > > >     > > > > >     > at
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > shallow
>> > > >     > > > > >     > > >     >     level? Do we always leave that
>> bit in
>> > > the
>> > > >     > wrapper
>> > > >     > > > > > message
>> > > >     > > > > >     > > unset?
>> > > >     > > > > >     > > > An
>> > > >     > > > > >     > > >     >     alternative is to set the
>> tombstone
>> > bit
>> > > > in the
>> > > >     > > > > wrapper
>> > > >     > > > > > if at
>> > > >     > > > > >     > > > least one
>> > > >     > > > > >     > > >     >     inner message has the tombstone
>> bit
>> > > set.
>> > > > This
>> > > >     > > makes
>> > > >     > > > > > things a
>> > > >     > > > > >     > > bit
>> > > >     > > > > >     > > > more
>> > > >     > > > > >     > > >     >     complicated, but we could
>> potentially
>> > > > exploit
>> > > >     > > that
>> > > >     > > > > for
>> > > >     > > > > >     > > > optimizing down
>> > > >     > > > > >     > > >     >     conversion. For example, we only
>> need
>> > > to
>> > > >     > convert
>> > > >     > > > > > messages
>> > > >     > > > > >     > with
>> > > >     > > > > >     > > > magic 2
>> > > >     > > > > >     > > >     > to
>> > > >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone
>> > bit
>> > > > is set
>> > > >     > > > > > (conversion is
>> > > >     > > > > >     > > > always
>> > > >     > > > > >     > > >     > needed
>> > > >     > > > > >     > > >     >     from magic 2 to magic 0). Not
>> sure if
>> > > the
>> > > >     > > > > optimization
>> > > >     > > > > > is
>> > > >     > > > > >     > worth
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     >     complexity though.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     3. The referencing of the new
>> version
>> > > of
>> > > >     > > > > >     > > > ProducerRequest/FetchRequest
>> > > >     > > > > >     > > >     > is
>> > > >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
>> > > > convention
>> > > >     > > > starts
>> > > >     > > > > at
>> > > >     > > > > >     > version
>> > > >     > > > > >     > > > at 0, I
>> > > >     > > > > >     > > >     >     think the new version would be 3.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     4. "If the magic byte on message
>> is
>> > 2,
>> > > > the
>> > > >     > broker
>> > > >     > > > > > should use
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > >     > tombstone
>> > > >     > > > > >     > > >     >     bit for log compaction." What
>> about
>> > > null
>> > > > value?
>> > > >     > > My
>> > > >     > > > > >     > > understanding
>> > > >     > > > > >     > > > is
>> > > >     > > > > >     > > >     > that
>> > > >     > > > > >     > > >     >     null value will be treated the
>> same
>> > as
>> > > > setting
>> > > >     > > the
>> > > >     > > > > > tombstone
>> > > >     > > > > >     > > bit.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     5. For the migration path, it
>> would
>> > be
>> > > > useful
>> > > >     > to
>> > > >     > > > > > describe the
>> > > >     > > > > >     > > > down
>> > > >     > > > > >     > > >     >     conversion path to consumers
>> (i.e.,
>> > > > brokers on
>> > > >     > > > > message
>> > > >     > > > > > format
>> > > >     > > > > >     > > > 0.10.2
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     consumers on older version).
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     Thanks,
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     Jun
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
>> > > Michael
>> > > >     > Pearce <
>> > > >     > > > > >     > > > Michael.Pearce@ig.com
>> > > >     > > > > >     > > >     > >
>> > > >     > > > > >     > > >     >     wrote:
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     > Hi All,
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > We have been discussing in the
>> > below
>> > > > thread
>> > > >     > and
>> > > >     > > > > final
>> > > >     > > > > >     > changes
>> > > >     > > > > >     > > > have
>> > > >     > > > > >     > > >     > been
>> > > >     > > > > >     > > >     >     > made to the KIP wiki based on
>> these
>> > > >     > > discussions.
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > We would now like to put to the
>> > vote
>> > > > the
>> > > >     > > > following
>> > > >     > > > > > KIP:
>> > > >     > > > > >     > > >     >     > https://cwiki.apache.org/
>> > > >     > > > > > confluence/display/KAFKA/KIP-
>> > > >     > > > > >     > 87+-+
>> > > >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > This kip is for having a
>> distinct
>> > > > compaction
>> > > >     > > > > > attribute
>> > > >     > > > > >     > > > "tombstone"
>> > > >     > > > > >     > > >     > flag
>> > > >     > > > > >     > > >     >     > instead of relying on null
>> value,
>> > > > allowing
>> > > >     > > > non-null
>> > > >     > > > > > value
>> > > >     > > > > >     > > > delete
>> > > >     > > > > >     > > >     > messages.
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > Many thanks,
>> > > >     > > > > >     > > >     >     > Michael
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
>> > > Pearce"
>> > > > <
>> > > >     > > > > >     > > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > >     > wrote:
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     Hi Mayuresh,
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     LGTM. Ive just made one
>> small
>> > > > adjustment
>> > > >     > > > > > updating the
>> > > >     > > > > >     > > wire
>> > > >     > > > > >     > > >     > protocol to
>> > > >     > > > > >     > > >     >     > show the magic byte bump.
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     Do we think we're good to
>> put
>> > to
>> > > a
>> > > > vote?
>> > > >     > Is
>> > > >     > > > > > there any
>> > > >     > > > > >     > > > other bits
>> > > >     > > > > >     > > >     >     > needing discussion?
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     Cheers
>> > > >     > > > > >     > > >     >     >     Mike
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     On 21/11/2016, 18:26,
>> "Mayuresh
>> > > > Gharat" <
>> > > >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
>> > > >     > > > > >     > > >     >     > wrote:
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         Hi Michael,
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         I have updated the
>> > migration
>> > > > section
>> > > >     > of
>> > > >     > > > the
>> > > >     > > > > > KIP.
>> > > >     > > > > >     > Can
>> > > >     > > > > >     > > > you
>> > > >     > > > > >     > > >     > please
>> > > >     > > > > >     > > >     >     > take a look?
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         Thanks,
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         Mayuresh
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at
>> > 9:07
>> > > > AM,
>> > > >     > > Mayuresh
>> > > >     > > > > > Gharat <
>> > > >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >     >     >         > wrote:
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         > Hi Michael,
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > That whilst sending
>> > > > tombstone and
>> > > >     > non
>> > > >     > > > > null
>> > > >     > > > > > value,
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > >     > consumer
>> > > >     > > > > >     > > >     >     > can expect
>> > > >     > > > > >     > > >     >     >         > only to receive the
>> > > non-null
>> > > >     > message
>> > > >     > > > only
>> > > >     > > > > > in step
>> > > >     > > > > >     > > > (3) is
>> > > >     > > > > >     > > >     > this
>> > > >     > > > > >     > > >     >     > correct?
>> > > >     > > > > >     > > >     >     >         > ---> I do agree with
>> you
>> > > > here.
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > Becket, Ismael : can
>> you
>> > > guys
>> > > >     > review
>> > > >     > > > the
>> > > >     > > > > >     > migration
>> > > >     > > > > >     > > > plan
>> > > >     > > > > >     > > >     > listed
>> > > >     > > > > >     > > >     >     > above using
>> > > >     > > > > >     > > >     >     >         > magic byte?
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > Thanks,
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > Mayuresh
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016
>> at
>> > > 8:58
>> > > > AM,
>> > > >     > > > Michael
>> > > >     > > > > > Pearce <
>> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > >     >     >         > wrote:
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         >> Many thanks for this
>> > > > Mayuresh. I
>> > > >     > > don't
>> > > >     > > > > > have any
>> > > >     > > > > >     > > >     > objections.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> I assume we should
>> > state:
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> That whilst sending
>> > > > tombstone and
>> > > >     > > non
>> > > >     > > > > null
>> > > >     > > > > >     > value,
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > consumer
>> > > >     > > > > >     > > >     >     > can expect
>> > > >     > > > > >     > > >     >     >         >> only to receive the
>> > > non-null
>> > > >     > message
>> > > >     > > > > only
>> > > >     > > > > > in
>> > > >     > > > > >     > step
>> > > >     > > > > >     > > > (3) is
>> > > >     > > > > >     > > >     > this
>> > > >     > > > > >     > > >     >     > correct?
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Cheers
>> > > >     > > > > >     > > >     >     >         >> Mike
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Sent using OWA for
>> > iPhone
>> > > >     > > > > >     > > >     >     >         >>
>> > > > ______________________________
>> > > >     > > > > __________
>> > > >     > > > > >     > > >     >     >         >> From: Mayuresh
>> Gharat <
>> > > >     > > > > >     > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     >     >         >> Sent: Thursday,
>> November
>> > > > 17, 2016
>> > > >     > > > > 5:18:41
>> > > >     > > > > > PM
>> > > >     > > > > >     > > >     >     >         >> To:
>> > dev@kafka.apache.org
>> > > >     > > > > >     > > >     >     >         >> Subject: Re:
>> [DISCUSS]
>> > > > KIP-87 -
>> > > >     > Add
>> > > >     > > > > > Compaction
>> > > >     > > > > >     > > > Tombstone
>> > > >     > > > > >     > > >     > Flag
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Hi Ismael,
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Thanks for the
>> > > explanation.
>> > > >     > > > > >     > > >     >     >         >> Specially I like this
>> > part
>> > > > where
>> > > >     > in
>> > > >     > > > you
>> > > >     > > > > >     > mentioned
>> > > >     > > > > >     > > > we can
>> > > >     > > > > >     > > >     > get
>> > > >     > > > > >     > > >     >     > rid of the
>> > > >     > > > > >     > > >     >     >         >> older null value
>> support
>> > > > for log
>> > > >     > > > > > compaction
>> > > >     > > > > >     > later
>> > > >     > > > > >     > > > on,
>> > > >     > > > > >     > > >     > here :
>> > > >     > > > > >     > > >     >     >         >> We can't change
>> > semantics
>> > > > of the
>> > > >     > > > message
>> > > >     > > > > > format
>> > > >     > > > > >     > > > without
>> > > >     > > > > >     > > >     > having
>> > > >     > > > > >     > > >     >     > a long
>> > > >     > > > > >     > > >     >     >         >> transition period.
>> And
>> > we
>> > > > can't
>> > > >     > rely
>> > > >     > > > > >     > > >     >     >         >> on people reading
>> > > > documentation or
>> > > >     > > > > acting
>> > > >     > > > > > on a
>> > > >     > > > > >     > > > warning for
>> > > >     > > > > >     > > >     >     > something so
>> > > >     > > > > >     > > >     >     >         >> fundamental. As
>> such, my
>> > > > take is
>> > > >     > > that
>> > > >     > > > we
>> > > >     > > > > > need to
>> > > >     > > > > >     > > > bump the
>> > > >     > > > > >     > > >     > magic
>> > > >     > > > > >     > > >     >     > byte. The
>> > > >     > > > > >     > > >     >     >         >> good news is
>> > > >     > > > > >     > > >     >     >         >> that we don't have to
>> > > > support all
>> > > >     > > > > versions
>> > > >     > > > > >     > > forever.
>> > > >     > > > > >     > > > We
>> > > >     > > > > >     > > >     > have
>> > > >     > > > > >     > > >     >     > said that we
>> > > >     > > > > >     > > >     >     >         >> will support direct
>> > > > upgrades for 2
>> > > >     > > > > years.
>> > > >     > > > > > That
>> > > >     > > > > >     > > > means that
>> > > >     > > > > >     > > >     >     > message format
>> > > >     > > > > >     > > >     >     >         >> version n could, in
>> > > theory,
>> > > > be
>> > > >     > > > removed 2
>> > > >     > > > > > years
>> > > >     > > > > >     > > > after the
>> > > >     > > > > >     > > >     > it's
>> > > >     > > > > >     > > >     >     > introduced.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Just a heads up, I
>> would
>> > > > like to
>> > > >     > > > mention
>> > > >     > > > > > that
>> > > >     > > > > >     > even
>> > > >     > > > > >     > > > without
>> > > >     > > > > >     > > >     >     > bumping magic
>> > > >     > > > > >     > > >     >     >         >> byte, we will *NOT*
>> > loose
>> > > > zero
>> > > >     > copy
>> > > >     > > as
>> > > >     > > > > in
>> > > >     > > > > > the
>> > > >     > > > > >     > > > client(x+1)
>> > > >     > > > > >     > > >     > in my
>> > > >     > > > > >     > > >     >     >         >> explanation
>> > > >     > > > > >     > > >     >     >         >> above will convert
>> > > > internally a
>> > > >     > null
>> > > >     > > > > > value to
>> > > >     > > > > >     > > have a
>> > > >     > > > > >     > > >     > tombstone
>> > > >     > > > > >     > > >     >     > bit set and
>> > > >     > > > > >     > > >     >     >         >> a tombstone bit set
>> to
>> > > have
>> > > > a null
>> > > >     > > > value
>> > > >     > > > > >     > > > automatically
>> > > >     > > > > >     > > >     >     > internally and by
>> > > >     > > > > >     > > >     >     >         >> the time we move to
>> > > version
>> > > > (x+2),
>> > > >     > > the
>> > > >     > > > > > clients
>> > > >     > > > > >     > > > would have
>> > > >     > > > > >     > > >     >     > upgraded.
>> > > >     > > > > >     > > >     >     >         >> Obviously if we
>> support
>> > a
>> > > > request
>> > > >     > > from
>> > > >     > > > > >     > > consumer(x),
>> > > >     > > > > >     > > > we
>> > > >     > > > > >     > > >     > will
>> > > >     > > > > >     > > >     >     > loose zero
>> > > >     > > > > >     > > >     >     >         >> copy
>> > > >     > > > > >     > > >     >     >         >> but that is the same
>> > case
>> > > > with
>> > > >     > magic
>> > > >     > > > > byte.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> But if magic byte
>> bump
>> > > > makes life
>> > > >     > > > easier
>> > > >     > > > > > for
>> > > >     > > > > >     > > > transition
>> > > >     > > > > >     > > >     > for the
>> > > >     > > > > >     > > >     >     > above
>> > > >     > > > > >     > > >     >     >         >> reasons that you
>> > > explained,
>> > > > I am
>> > > >     > OK
>> > > >     > > > with
>> > > >     > > > > > it
>> > > >     > > > > >     > since
>> > > >     > > > > >     > > > we are
>> > > >     > > > > >     > > >     > going
>> > > >     > > > > >     > > >     >     > to meet the
>> > > >     > > > > >     > > >     >     >         >> end goal down the
>> road
>> > :)
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> On a side note can we
>> > > > update the
>> > > >     > doc
>> > > >     > > > > here
>> > > >     > > > > > on
>> > > >     > > > > >     > magic
>> > > >     > > > > >     > > > byte
>> > > >     > > > > >     > > >     > to say
>> > > >     > > > > >     > > >     >     > that "*it
>> > > >     > > > > >     > > >     >     >         >> should be bumped
>> > whenever
>> > > > the
>> > > >     > > message
>> > > >     > > > > > format is
>> > > >     > > > > >     > > > changed
>> > > >     > > > > >     > > >     > or the
>> > > >     > > > > >     > > >     >     >         >> interpretation of
>> > message
>> > > > format
>> > > >     > > > (usage
>> > > >     > > > > > of the
>> > > >     > > > > >     > > > reserved
>> > > >     > > > > >     > > >     > bits as
>> > > >     > > > > >     > > >     >     > well) is
>> > > >     > > > > >     > > >     >     >         >> changed*".
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Hi Michael,
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Here is the update
>> plan
>> > > > that we
>> > > >     > > > > discussed
>> > > >     > > > > >     > offline
>> > > >     > > > > >     > > >     > yesterday :
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Currently the
>> magic-byte
>> > > > which
>> > > >     > > > > > corresponds to
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > >     >     > "message.format.version"
>> > > >     > > > > >     > > >     >     >         >> is set to 1.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> 1) On broker it will
>> be
>> > > set
>> > > > to 1
>> > > >     > > > > > initially.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> 2) When a producer
>> > client
>> > > > sends a
>> > > >     > > > > message
>> > > >     > > > > > with
>> > > >     > > > > >     > > > magic-byte
>> > > >     > > > > >     > > >     > = 2,
>> > > >     > > > > >     > > >     >     > since the
>> > > >     > > > > >     > > >     >     >         >> broker is on
>> magic-byte
>> > =
>> > > > 1, we
>> > > >     > will
>> > > >     > > > > down
>> > > >     > > > > >     > convert
>> > > >     > > > > >     > > > it,
>> > > >     > > > > >     > > >     > which
>> > > >     > > > > >     > > >     >     > means if the
>> > > >     > > > > >     > > >     >     >         >> tombstone bit is set,
>> > the
>> > > > value
>> > > >     > will
>> > > >     > > > be
>> > > >     > > > > > set to
>> > > >     > > > > >     > > > null. A
>> > > >     > > > > >     > > >     > consumer
>> > > >     > > > > >     > > >     >     >         >> understanding
>> > magic-byte =
>> > > > 1, will
>> > > >     > > > still
>> > > >     > > > > > work
>> > > >     > > > > >     > with
>> > > >     > > > > >     > > > this. A
>> > > >     > > > > >     > > >     >     > consumer
>> > > >     > > > > >     > > >     >     >         >> working
>> > > >     > > > > >     > > >     >     >         >> with magic-byte =2
>> will
>> > > > also be
>> > > >     > able
>> > > >     > > > to
>> > > >     > > > > >     > understand
>> > > >     > > > > >     > > > this,
>> > > >     > > > > >     > > >     > since
>> > > >     > > > > >     > > >     >     > it
>> > > >     > > > > >     > > >     >     >         >> understands the
>> > tombstone.
>> > > >     > > > > >     > > >     >     >         >> Now there is still
>> the
>> > > > question of
>> > > >     > > > > > supporting a
>> > > >     > > > > >     > > >     > non-tombstone
>> > > >     > > > > >     > > >     >     > and null
>> > > >     > > > > >     > > >     >     >         >> value from producer
>> > client
>> > > > with
>> > > >     > > > > > magic-byte = 2.*
>> > > >     > > > > >     > > (I
>> > > >     > > > > >     > > > am
>> > > >     > > > > >     > > >     > not sure
>> > > >     > > > > >     > > >     >     > if we
>> > > >     > > > > >     > > >     >     >         >> should support this.
>> > > > Ismael/Becket
>> > > >     > > can
>> > > >     > > > > > comment
>> > > >     > > > > >     > > > here)*
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> 3) When almost all
>> the
>> > > > clients
>> > > >     > have
>> > > >     > > > > > upgraded,
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > >     >     > message.format.version
>> > > >     > > > > >     > > >     >     >         >> on
>> > > >     > > > > >     > > >     >     >         >> the broker can be
>> > changed
>> > > > to 2,
>> > > >     > > where
>> > > >     > > > in
>> > > >     > > > > > the
>> > > >     > > > > >     > down
>> > > >     > > > > >     > > >     > conversion in
>> > > >     > > > > >     > > >     >     > the above
>> > > >     > > > > >     > > >     >     >         >> step will not
>> happen. If
>> > > at
>> > > > this
>> > > >     > > point
>> > > >     > > > > we
>> > > >     > > > > > get a
>> > > >     > > > > >     > > > consumer
>> > > >     > > > > >     > > >     >     > request from a
>> > > >     > > > > >     > > >     >     >         >> older consumer, we
>> might
>> > > > have to
>> > > >     > > down
>> > > >     > > > > > convert
>> > > >     > > > > >     > > where
>> > > >     > > > > >     > > > in we
>> > > >     > > > > >     > > >     > loose
>> > > >     > > > > >     > > >     >     > zero copy,
>> > > >     > > > > >     > > >     >     >         >> but these cases
>> should
>> > be
>> > > > rare.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Becket can you review
>> > this
>> > > > plan
>> > > >     > and
>> > > >     > > > add
>> > > >     > > > > > more
>> > > >     > > > > >     > > > details if I
>> > > >     > > > > >     > > >     > have
>> > > >     > > > > >     > > >     >     >         >> missed/wronged
>> > something,
>> > > > before
>> > > >     > we
>> > > >     > > > put
>> > > >     > > > > > it on
>> > > >     > > > > >     > KIP.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Thanks,
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Mayuresh
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016
>> at
>> > > > 11:07 PM,
>> > > >     > > > > Michael
>> > > >     > > > > >     > Pearce <
>> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > >     >     >         >> wrote:
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> > Thanks guys, for
>> > > > discussing this
>> > > >     > > > > > offline and
>> > > >     > > > > >     > > > getting
>> > > >     > > > > >     > > >     > some
>> > > >     > > > > >     > > >     >     > consensus.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > So its clear for
>> > myself
>> > > > and
>> > > >     > others
>> > > >     > > > > what
>> > > >     > > > > > is
>> > > >     > > > > >     > > > proposed now
>> > > >     > > > > >     > > >     > (i
>> > > >     > > > > >     > > >     >     > think i
>> > > >     > > > > >     > > >     >     >         >> > understand, but
>> want
>> > to
>> > > > make
>> > > >     > sure)
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Could i ask either
>> > > > directly
>> > > >     > update
>> > > >     > > > the
>> > > >     > > > > > kip to
>> > > >     > > > > >     > > > detail the
>> > > >     > > > > >     > > >     >     > migration
>> > > >     > > > > >     > > >     >     >         >> > strategy, or
>> > (re-)state
>> > > > your
>> > > >     > > offline
>> > > >     > > > > > discussed
>> > > >     > > > > >     > > and
>> > > >     > > > > >     > > >     > agreed
>> > > >     > > > > >     > > >     >     > migration
>> > > >     > > > > >     > > >     >     >         >> > strategy based on a
>> > > magic
>> > > > byte
>> > > >     > is
>> > > >     > > in
>> > > >     > > > > > this
>> > > >     > > > > >     > > thread.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > The main original
>> > driver
>> > > > for the
>> > > >     > > KIP
>> > > >     > > > > > was to
>> > > >     > > > > >     > > > support
>> > > >     > > > > >     > > >     >     > compaction where
>> > > >     > > > > >     > > >     >     >         >> value
>> > > >     > > > > >     > > >     >     >         >> > isn't null, based
>> off
>> > > the
>> > > >     > > > discussions
>> > > >     > > > > on
>> > > >     > > > > >     > KIP-82
>> > > >     > > > > >     > > > thread.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > We should be able
>> to
>> > > > support
>> > > >     > > > > > non-tombstone +
>> > > >     > > > > >     > > null
>> > > >     > > > > >     > > > value
>> > > >     > > > > >     > > >     > by the
>> > > >     > > > > >     > > >     >     >         >> completion
>> > > >     > > > > >     > > >     >     >         >> > of the KIP, as we
>> > noted
>> > > > when
>> > > >     > > > > discussing
>> > > >     > > > > > this
>> > > >     > > > > >     > > kip,
>> > > >     > > > > >     > > > having
>> > > >     > > > > >     > > >     >     > logic based on
>> > > >     > > > > >     > > >     >     >         >> a
>> > > >     > > > > >     > > >     >     >         >> > null value isn't
>> very
>> > > > clean and
>> > > >     > > also
>> > > >     > > > > > separates
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > >     > concerns.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > As discussed
>> already
>> > > > though we
>> > > >     > can
>> > > >     > > > > > split this
>> > > >     > > > > >     > > into
>> > > >     > > > > >     > > >     > KIP-87a
>> > > >     > > > > >     > > >     >     > and KIP-87b
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Where we look to
>> > deliver
>> > > > KIP-87a
>> > > >     > > on
>> > > >     > > > a
>> > > >     > > > > >     > compacted
>> > > >     > > > > >     > > > topic
>> > > >     > > > > >     > > >     > (to
>> > > >     > > > > >     > > >     >     > address the
>> > > >     > > > > >     > > >     >     >         >> > immediate issues)
>> > > >     > > > > >     > > >     >     >         >> > * tombstone + null
>> > value
>> > > >     > > > > >     > > >     >     >         >> > * tombstone +
>> non-null
>> > > > value
>> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
>> > > > non-null value
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Then we can discuss
>> > once
>> > > > KIP-87a
>> > > >     > > is
>> > > >     > > > > > completed
>> > > >     > > > > >     > > > options
>> > > >     > > > > >     > > >     > later
>> > > >     > > > > >     > > >     >     > and how we
>> > > >     > > > > >     > > >     >     >         >> > support the second
>> > part
>> > > > KIP-87b
>> > > >     > to
>> > > >     > > > > > deliver:
>> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
>> null
>> > > > value
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Cheers
>> > > >     > > > > >     > > >     >     >         >> > Mike
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > > ______________________________
>> > > >     > > > > > __________
>> > > >     > > > > >     > > >     >     >         >> > From: Becket Qin <
>> > > >     > > > > becket.qin@gmail.com>
>> > > >     > > > > >     > > >     >     >         >> > Sent: Thursday,
>> > November
>> > > > 17,
>> > > >     > 2016
>> > > >     > > > 1:43
>> > > >     > > > > > AM
>> > > >     > > > > >     > > >     >     >         >> > To:
>> > > dev@kafka.apache.org
>> > > >     > > > > >     > > >     >     >         >> > Subject: Re:
>> [DISCUSS]
>> > > > KIP-87 -
>> > > >     > > Add
>> > > >     > > > > > Compaction
>> > > >     > > > > >     > > >     > Tombstone Flag
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and
>> I
>> > had
>> > > > an
>> > > >     > > offline
>> > > >     > > > > >     > discussion,
>> > > >     > > > > >     > > > and
>> > > >     > > > > >     > > >     > following
>> > > >     > > > > >     > > >     >     > is a brief
>> > > >     > > > > >     > > >     >     >         >> > summary.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > 1. We agreed that
>> not
>> > > > bumping up
>> > > >     > > > magic
>> > > >     > > > > > value
>> > > >     > > > > >     > may
>> > > >     > > > > >     > > > result
>> > > >     > > > > >     > > >     > in
>> > > >     > > > > >     > > >     >     > losing zero
>> > > >     > > > > >     > > >     >     >         >> copy
>> > > >     > > > > >     > > >     >     >         >> > during migration.
>> > > >     > > > > >     > > >     >     >         >> > 2. Given that
>> bumping
>> > up
>> > > > magic
>> > > >     > > value
>> > > >     > > > > is
>> > > >     > > > > > almost
>> > > >     > > > > >     > > > free and
>> > > >     > > > > >     > > >     > has
>> > > >     > > > > >     > > >     >     > benefit of
>> > > >     > > > > >     > > >     >     >         >> > avoiding potential
>> > > > performance
>> > > >     > > > issue.
>> > > >     > > > > > It is
>> > > >     > > > > >     > > > probably
>> > > >     > > > > >     > > >     > worth
>> > > >     > > > > >     > > >     >     > doing.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > One issue we still
>> > need
>> > > > to think
>> > > >     > > > about
>> > > >     > > > > > is
>> > > >     > > > > >     > > whether
>> > > >     > > > > >     > > > we
>> > > >     > > > > >     > > >     > want to
>> > > >     > > > > >     > > >     >     > support a
>> > > >     > > > > >     > > >     >     >         >> > non-tombstone
>> message
>> > > > with null
>> > > >     > > > value.
>> > > >     > > > > >     > > >     >     >         >> > Currently it is not
>> > > > supported by
>> > > >     > > > > Kafka.
>> > > >     > > > > > If we
>> > > >     > > > > >     > > > allow a
>> > > >     > > > > >     > > >     >     > non-tombstone null
>> > > >     > > > > >     > > >     >     >         >> > value message to
>> exist
>> > > > after
>> > > >     > > KIP-87.
>> > > >     > > > > The
>> > > >     > > > > >     > problem
>> > > >     > > > > >     > > > is
>> > > >     > > > > >     > > >     > that such
>> > > >     > > > > >     > > >     >     > message
>> > > >     > > > > >     > > >     >     >         >> will
>> > > >     > > > > >     > > >     >     >         >> > not be supported by
>> > the
>> > > >     > consumers
>> > > >     > > > > prior
>> > > >     > > > > > to
>> > > >     > > > > >     > > KIP-87.
>> > > >     > > > > >     > > >     > Because a
>> > > >     > > > > >     > > >     >     > null value
>> > > >     > > > > >     > > >     >     >         >> > will always be
>> > > > interpreted to a
>> > > >     > > > > > tombstone.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > One option is that
>> we
>> > > > keep the
>> > > >     > > > current
>> > > >     > > > > > way,
>> > > >     > > > > >     > i.e.
>> > > >     > > > > >     > > > do not
>> > > >     > > > > >     > > >     >     > support such
>> > > >     > > > > >     > > >     >     >         >> > message. It would
>> be
>> > > good
>> > > > to
>> > > >     > know
>> > > >     > > if
>> > > >     > > > > > there is
>> > > >     > > > > >     > a
>> > > >     > > > > >     > > >     > concrete use
>> > > >     > > > > >     > > >     >     > case for
>> > > >     > > > > >     > > >     >     >         >> such
>> > > >     > > > > >     > > >     >     >         >> > message. If there
>> is
>> > > not,
>> > > > we can
>> > > >     > > > > > probably just
>> > > >     > > > > >     > > not
>> > > >     > > > > >     > > >     > support it.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Thanks,
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > JIangjie (Becket)
>> Qin
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > On Wed, Nov 16,
>> 2016
>> > at
>> > > > 1:28 PM,
>> > > >     > > > > > Mayuresh
>> > > >     > > > > >     > > Gharat <
>> > > >     > > > > >     > > >     >     >         >> >
>> > > > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >     >     >         >> > > wrote:
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > > Hi Ismael,
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > This is
>> something I
>> > > can
>> > > > think
>> > > >     > of
>> > > >     > > > for
>> > > >     > > > > >     > migration
>> > > >     > > > > >     > > > plan:
>> > > >     > > > > >     > > >     >     >         >> > > So the migration
>> > plan
>> > > > can look
>> > > >     > > > > > something
>> > > >     > > > > >     > like
>> > > >     > > > > >     > > > this,
>> > > >     > > > > >     > > >     > with up
>> > > >     > > > > >     > > >     >     >         >> conversion :
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > 1) Currently lets
>> > say
>> > > > we have
>> > > >     > > > Broker
>> > > >     > > > > > at
>> > > >     > > > > >     > > version
>> > > >     > > > > >     > > > x.
>> > > >     > > > > >     > > >     >     >         >> > > 2) Currently we
>> have
>> > > > clients
>> > > >     > at
>> > > >     > > > > > version x.
>> > > >     > > > > >     > > >     >     >         >> > > 3) a) We move the
>> > > > version to
>> > > >     > > > > > Broker(x+1) :
>> > > >     > > > > >     > > > supports
>> > > >     > > > > >     > > >     > both
>> > > >     > > > > >     > > >     >     > tombstone and
>> > > >     > > > > >     > > >     >     >         >> > null
>> > > >     > > > > >     > > >     >     >         >> > > for log
>> compaction.
>> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
>> > the
>> > > > client
>> > > >     > to
>> > > >     > > > > > version
>> > > >     > > > > >     > > > client(x+1) :
>> > > >     > > > > >     > > >     > if in
>> > > >     > > > > >     > > >     >     > the
>> > > >     > > > > >     > > >     >     >         >> producer
>> > > >     > > > > >     > > >     >     >         >> > > client(x+1) the
>> > value
>> > > > is set
>> > > >     > to
>> > > >     > > > > null,
>> > > >     > > > > > we
>> > > >     > > > > >     > will
>> > > >     > > > > >     > > >     > automatically
>> > > >     > > > > >     > > >     >     > set the
>> > > >     > > > > >     > > >     >     >         >> > > Tombstone bit
>> > > > internally. If
>> > > >     > the
>> > > >     > > > > > producer
>> > > >     > > > > >     > > > client(x+1)
>> > > >     > > > > >     > > >     > sets
>> > > >     > > > > >     > > >     >     > the
>> > > >     > > > > >     > > >     >     >         >> tombstone
>> > > >     > > > > >     > > >     >     >         >> > > itself, well and
>> > good.
>> > > > For
>> > > >     > > > producer
>> > > >     > > > > >     > client(x),
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > broker
>> > > >     > > > > >     > > >     >     > will up
>> > > >     > > > > >     > > >     >     >         >> convert
>> > > >     > > > > >     > > >     >     >         >> > > to have the
>> > tombstone
>> > > > bit.
>> > > >     > > > > > Broker(x+1) is
>> > > >     > > > > >     > > > supporting
>> > > >     > > > > >     > > >     > both.
>> > > >     > > > > >     > > >     >     > Consumer
>> > > >     > > > > >     > > >     >     >         >> > > client(x+1) will
>> be
>> > > > aware of
>> > > >     > > this
>> > > >     > > > > and
>> > > >     > > > > > should
>> > > >     > > > > >     > > be
>> > > >     > > > > >     > > > able
>> > > >     > > > > >     > > >     > to
>> > > >     > > > > >     > > >     >     > handle this.
>> > > >     > > > > >     > > >     >     >         >> For
>> > > >     > > > > >     > > >     >     >         >> > > consumer
>> client(x)
>> > we
>> > > > will
>> > > >     > down
>> > > >     > > > > > convert the
>> > > >     > > > > >     > > > message
>> > > >     > > > > >     > > >     > on the
>> > > >     > > > > >     > > >     >     > broker
>> > > >     > > > > >     > > >     >     >         >> side.
>> > > >     > > > > >     > > >     >     >         >> > >     c) At this
>> point
>> > > we
>> > > > will
>> > > >     > > have
>> > > >     > > > to
>> > > >     > > > > >     > specify a
>> > > >     > > > > >     > > >     > warning or
>> > > >     > > > > >     > > >     >     > clearly
>> > > >     > > > > >     > > >     >     >         >> specify
>> > > >     > > > > >     > > >     >     >         >> > > in docs that this
>> > > > behavior is
>> > > >     > > > about
>> > > >     > > > > > to be
>> > > >     > > > > >     > > > changed for
>> > > >     > > > > >     > > >     > log
>> > > >     > > > > >     > > >     >     > compaction.
>> > > >     > > > > >     > > >     >     >         >> > > 4) a) In next
>> > release
>> > > > of the
>> > > >     > > > > > Broker(x+2), we
>> > > >     > > > > >     > > > say that
>> > > >     > > > > >     > > >     > only
>> > > >     > > > > >     > > >     >     > Tombstone
>> > > >     > > > > >     > > >     >     >         >> is
>> > > >     > > > > >     > > >     >     >         >> > > used for log
>> > > compaction
>> > > > on the
>> > > >     > > > > Broker
>> > > >     > > > > > side.
>> > > >     > > > > >     > > >     > Clients(x+1)
>> > > >     > > > > >     > > >     >     > still is
>> > > >     > > > > >     > > >     >     >         >> > > supported.
>> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
>> > the
>> > > > client
>> > > >     > to
>> > > >     > > > > > version
>> > > >     > > > > >     > > > client(x+2) :
>> > > >     > > > > >     > > >     > if
>> > > >     > > > > >     > > >     >     > value is set
>> > > >     > > > > >     > > >     >     >         >> to
>> > > >     > > > > >     > > >     >     >         >> > > null, tombstone
>> will
>> > > > not be
>> > > >     > set
>> > > >     > > > > >     > automatically.
>> > > >     > > > > >     > > > The
>> > > >     > > > > >     > > >     > client
>> > > >     > > > > >     > > >     >     > will have to
>> > > >     > > > > >     > > >     >     >         >> > call
>> > > >     > > > > >     > > >     >     >         >> > > setTombstone() to
>> > > > actually set
>> > > >     > > the
>> > > >     > > > > >     > tombstone.
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > We should compare
>> > this
>> > > >     > migration
>> > > >     > > > > plan
>> > > >     > > > > > with
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > >     > migration
>> > > >     > > > > >     > > >     >     > plan for
>> > > >     > > > > >     > > >     >     >         >> magic
>> > > >     > > > > >     > > >     >     >         >> > > byte bump and do
>> > > > whatever
>> > > >     > looks
>> > > >     > > > > good.
>> > > >     > > > > >     > > >     >     >         >> > > I am just worried
>> > that
>> > > > if we
>> > > >     > go
>> > > >     > > > down
>> > > >     > > > > > magic
>> > > >     > > > > >     > > byte
>> > > >     > > > > >     > > > route,
>> > > >     > > > > >     > > >     >     > unless I am
>> > > >     > > > > >     > > >     >     >         >> > missing
>> > > >     > > > > >     > > >     >     >         >> > > something, it
>> sounds
>> > > > like
>> > > >     > kafka
>> > > >     > > > will
>> > > >     > > > > > be
>> > > >     > > > > >     > stuck
>> > > >     > > > > >     > > > with
>> > > >     > > > > >     > > >     >     > supporting both
>> > > >     > > > > >     > > >     >     >         >> null
>> > > >     > > > > >     > > >     >     >         >> > > value and
>> tombstone
>> > > bit
>> > > > for
>> > > >     > log
>> > > >     > > > > > compaction
>> > > >     > > > > >     > for
>> > > >     > > > > >     > > > life
>> > > >     > > > > >     > > >     > long,
>> > > >     > > > > >     > > >     >     > which does
>> > > >     > > > > >     > > >     >     >         >> not
>> > > >     > > > > >     > > >     >     >         >> > > look like a good
>> end
>> > > > state.
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > Thanks,
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > Mayuresh
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16,
>> 2016
>> > > at
>> > > > 9:32
>> > > >     > AM,
>> > > >     > > > > > Mayuresh
>> > > >     > > > > >     > > > Gharat <
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >     >     >         >> > > > wrote:
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > That's a very
>> good
>> > > > point
>> > > >     > > which I
>> > > >     > > > > > might
>> > > >     > > > > >     > have
>> > > >     > > > > >     > > > not
>> > > >     > > > > >     > > >     >     > considered earlier.
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > Here is a plan
>> > that
>> > > I
>> > > > can
>> > > >     > > think
>> > > >     > > > > of:
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > Stage 1) The
>> > broker
>> > > > from now
>> > > >     > > on,
>> > > >     > > > > up
>> > > >     > > > > >     > converts
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > message
>> > > >     > > > > >     > > >     >     > to have the
>> > > >     > > > > >     > > >     >     >         >> > > > tombstone
>> marker.
>> > > The
>> > > > log
>> > > >     > > > > compaction
>> > > >     > > > > >     > thread
>> > > >     > > > > >     > > > does log
>> > > >     > > > > >     > > >     >     > compaction
>> > > >     > > > > >     > > >     >     >         >> based
>> > > >     > > > > >     > > >     >     >         >> > on
>> > > >     > > > > >     > > >     >     >         >> > > > both null and
>> > > > tombstone
>> > > >     > > marker.
>> > > >     > > > > > This is
>> > > >     > > > > >     > our
>> > > >     > > > > >     > > >     > transition
>> > > >     > > > > >     > > >     >     > period.
>> > > >     > > > > >     > > >     >     >         >> > > > Stage 2) The
>> next
>> > > > release we
>> > > >     > > > only
>> > > >     > > > > > say that
>> > > >     > > > > >     > > log
>> > > >     > > > > >     > > >     > compaction
>> > > >     > > > > >     > > >     >     > is based
>> > > >     > > > > >     > > >     >     >         >> on
>> > > >     > > > > >     > > >     >     >         >> > > > tombstone
>> marker.
>> > > > (Open
>> > > >     > source
>> > > >     > > > > > kafka makes
>> > > >     > > > > >     > > > this as a
>> > > >     > > > > >     > > >     >     > policy). By
>> > > >     > > > > >     > > >     >     >         >> this
>> > > >     > > > > >     > > >     >     >         >> > > time,
>> > > >     > > > > >     > > >     >     >         >> > > > the
>> organization
>> > > > which is
>> > > >     > > moving
>> > > >     > > > > to
>> > > >     > > > > > this
>> > > >     > > > > >     > > > release
>> > > >     > > > > >     > > >     > will be
>> > > >     > > > > >     > > >     >     > sure that
>> > > >     > > > > >     > > >     >     >         >> they
>> > > >     > > > > >     > > >     >     >         >> > > > have gone
>> through
>> > > the
>> > > > entire
>> > > >     > > > > > transition
>> > > >     > > > > >     > > > period.
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > My only goal of
>> > > doing
>> > > > this
>> > > >     > is
>> > > >     > > > that
>> > > >     > > > > > Kafka
>> > > >     > > > > >     > > > clearly
>> > > >     > > > > >     > > >     >     > specifies the end
>> > > >     > > > > >     > > >     >     >         >> > state
>> > > >     > > > > >     > > >     >     >         >> > > > about what log
>> > > > compaction
>> > > >     > > means
>> > > >     > > > > (is
>> > > >     > > > > > it
>> > > >     > > > > >     > null
>> > > >     > > > > >     > > > value
>> > > >     > > > > >     > > >     > or a
>> > > >     > > > > >     > > >     >     > tombstone
>> > > >     > > > > >     > > >     >     >         >> > marker,
>> > > >     > > > > >     > > >     >     >         >> > > > but not both).
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > What do you
>> think?
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > Thanks,
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh
>> > > >     > > > > >     > > >     >     >         >> > > > .
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16,
>> > 2016
>> > > > at 9:17
>> > > >     > > AM,
>> > > >     > > > > > Ismael
>> > > >     > > > > >     > > Juma <
>> > > >     > > > > >     > > >     >     > ismael@juma.me.uk>
>> > > >     > > > > >     > > >     >     >         >> > wrote:
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > >> One comment
>> > below.
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov
>> 16,
>> > > 2016
>> > > > at
>> > > >     > 5:08
>> > > >     > > > PM,
>> > > >     > > > > > Mayuresh
>> > > >     > > > > >     > > > Gharat <
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >     >     >         >> > > >> > wrote:
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >> >    - If we
>> > don't
>> > > > bump up
>> > > >     > > the
>> > > >     > > > > > magic
>> > > >     > > > > >     > byte,
>> > > >     > > > > >     > > > on the
>> > > >     > > > > >     > > >     > broker
>> > > >     > > > > >     > > >     >     > side, the
>> > > >     > > > > >     > > >     >     >         >> > > broker
>> > > >     > > > > >     > > >     >     >         >> > > >> >    will
>> always
>> > > > have to
>> > > >     > look
>> > > >     > > > at
>> > > >     > > > > > both
>> > > >     > > > > >     > > > tombstone
>> > > >     > > > > >     > > >     > bit and
>> > > >     > > > > >     > > >     >     > the value
>> > > >     > > > > >     > > >     >     >         >> when
>> > > >     > > > > >     > > >     >     >         >> > > do
>> > > >     > > > > >     > > >     >     >         >> > > >> the
>> > > >     > > > > >     > > >     >     >         >> > > >> >
>> compaction.
>> > > > Assuming
>> > > >     > we
>> > > >     > > do
>> > > >     > > > > > not bump
>> > > >     > > > > >     > up
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > magic
>> > > >     > > > > >     > > >     >     > byte,
>> > > >     > > > > >     > > >     >     >         >> > > >> >    imagine
>> the
>> > > > broker
>> > > >     > sees
>> > > >     > > a
>> > > >     > > > > > message
>> > > >     > > > > >     > > which
>> > > >     > > > > >     > > > does
>> > > >     > > > > >     > > >     > not
>> > > >     > > > > >     > > >     >     > have a
>> > > >     > > > > >     > > >     >     >         >> tombstone
>> > > >     > > > > >     > > >     >     >         >> > > bit
>> > > >     > > > > >     > > >     >     >         >> > > >> >    set. The
>> > > broker
>> > > > does
>> > > >     > not
>> > > >     > > > > know
>> > > >     > > > > > when
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > >     > message was
>> > > >     > > > > >     > > >     >     > produced
>> > > >     > > > > >     > > >     >     >         >> (i.e.
>> > > >     > > > > >     > > >     >     >         >> > > >> > whether
>> > > >     > > > > >     > > >     >     >         >> > > >> >    the
>> message
>> > > has
>> > > > been
>> > > >     > up
>> > > >     > > > > > converted or
>> > > >     > > > > >     > > > not), it
>> > > >     > > > > >     > > >     > has
>> > > >     > > > > >     > > >     >     > to take a
>> > > >     > > > > >     > > >     >     >         >> > further
>> > > >     > > > > >     > > >     >     >         >> > > >> > look at
>> > > >     > > > > >     > > >     >     >         >> > > >> >    the
>> value to
>> > > > see if it
>> > > >     > > is
>> > > >     > > > > > null or
>> > > >     > > > > >     > not
>> > > >     > > > > >     > > in
>> > > >     > > > > >     > > >     > order to
>> > > >     > > > > >     > > >     >     > determine
>> > > >     > > > > >     > > >     >     >         >> if it
>> > > >     > > > > >     > > >     >     >         >> > > is
>> > > >     > > > > >     > > >     >     >         >> > > >> a
>> > > >     > > > > >     > > >     >     >         >> > > >> >
>> tombstone.
>> > The
>> > > > same
>> > > >     > > logic
>> > > >     > > > > has
>> > > >     > > > > > to be
>> > > >     > > > > >     > > put
>> > > >     > > > > >     > > > on the
>> > > >     > > > > >     > > >     >     > consumer as
>> > > >     > > > > >     > > >     >     >         >> well
>> > > >     > > > > >     > > >     >     >         >> > > >> because
>> > > >     > > > > >     > > >     >     >         >> > > >> > the
>> > > >     > > > > >     > > >     >     >         >> > > >> >    consumer
>> > does
>> > > > not know
>> > > >     > > if
>> > > >     > > > > the
>> > > >     > > > > >     > message
>> > > >     > > > > >     > > > has
>> > > >     > > > > >     > > >     > been up
>> > > >     > > > > >     > > >     >     > converted or
>> > > >     > > > > >     > > >     >     >         >> > not.
>> > > >     > > > > >     > > >     >     >         >> > > >> >       - If
>> we
>> > > > upconvert
>> > > >     > > while
>> > > >     > > > > >     > appending,
>> > > >     > > > > >     > > > this is
>> > > >     > > > > >     > > >     > not
>> > > >     > > > > >     > > >     >     > the case,
>> > > >     > > > > >     > > >     >     >         >> > right?
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >> If I
>> understand
>> > you
>> > > >     > > correctly,
>> > > >     > > > > > this is
>> > > >     > > > > >     > not
>> > > >     > > > > >     > > >     > sufficient
>> > > >     > > > > >     > > >     >     > because the
>> > > >     > > > > >     > > >     >     >         >> log
>> > > >     > > > > >     > > >     >     >         >> > > may
>> > > >     > > > > >     > > >     >     >         >> > > >> have messages
>> > > > appended
>> > > >     > before
>> > > >     > > > it
>> > > >     > > > > > was
>> > > >     > > > > >     > > > upgraded to
>> > > >     > > > > >     > > >     > include
>> > > >     > > > > >     > > >     >     > KIP-87.
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >> Ismael
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > --
>> > > >     > > > > >     > > >     >     >         >> > > > -Regards,
>> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh R.
>> Gharat
>> > > >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > --
>> > > >     > > > > >     > > >     >     >         >> > > -Regards,
>> > > >     > > > > >     > > >     >     >         >> > > Mayuresh R.
>> Gharat
>> > > >     > > > > >     > > >     >     >         >> > > (862) 250-7125
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > The information
>> > > contained
>> > > > in
>> > > >     > this
>> > > >     > > > > email
>> > > >     > > > > > is
>> > > >     > > > > >     > > > strictly
>> > > >     > > > > >     > > >     >     > confidential and for
>> > > >     > > > > >     > > >     >     >         >> > the use of the
>> > addressee
>> > > > only,
>> > > >     > > > unless
>> > > >     > > > > >     > otherwise
>> > > >     > > > > >     > > >     > indicated. If
>> > > >     > > > > >     > > >     >     > you are
>> > > >     > > > > >     > > >     >     >         >> not
>> > > >     > > > > >     > > >     >     >         >> > the intended
>> > recipient,
>> > > > please
>> > > >     > do
>> > > >     > > > not
>> > > >     > > > > > read,
>> > > >     > > > > >     > > copy,
>> > > >     > > > > >     > > > use or
>> > > >     > > > > >     > > >     >     > disclose to
>> > > >     > > > > >     > > >     >     >         >> others
>> > > >     > > > > >     > > >     >     >         >> > this message or any
>> > > > attachment.
>> > > >     > > > Please
>> > > >     > > > > > also
>> > > >     > > > > >     > > > notify the
>> > > >     > > > > >     > > >     > sender
>> > > >     > > > > >     > > >     >     > by
>> > > >     > > > > >     > > >     >     >         >> replying
>> > > >     > > > > >     > > >     >     >         >> > to this email or by
>> > > > telephone
>> > > >     > > > (+44(020
>> > > >     > > > > > 7896
>> > > >     > > > > >     > > 0011)
>> > > >     > > > > >     > > > and
>> > > >     > > > > >     > > >     > then
>> > > >     > > > > >     > > >     >     > delete the
>> > > >     > > > > >     > > >     >     >         >> email
>> > > >     > > > > >     > > >     >     >         >> > and any copies of
>> it.
>> > > > Opinions,
>> > > >     > > > > > conclusion
>> > > >     > > > > >     > (etc)
>> > > >     > > > > >     > > > that
>> > > >     > > > > >     > > >     > do not
>> > > >     > > > > >     > > >     >     > relate to
>> > > >     > > > > >     > > >     >     >         >> the
>> > > >     > > > > >     > > >     >     >         >> > official business
>> of
>> > > this
>> > > >     > company
>> > > >     > > > > shall
>> > > >     > > > > > be
>> > > >     > > > > >     > > > understood as
>> > > >     > > > > >     > > >     >     > neither given
>> > > >     > > > > >     > > >     >     >         >> nor
>> > > >     > > > > >     > > >     >     >         >> > endorsed by it. IG
>> is
>> > a
>> > > > trading
>> > > >     > > name
>> > > >     > > > > of
>> > > >     > > > > > IG
>> > > >     > > > > >     > > Markets
>> > > >     > > > > >     > > >     > Limited (a
>> > > >     > > > > >     > > >     >     > company
>> > > >     > > > > >     > > >     >     >         >> > registered in
>> England
>> > > and
>> > > > Wales,
>> > > >     > > > > company
>> > > >     > > > > >     > number
>> > > >     > > > > >     > > >     > 04008957) and
>> > > >     > > > > >     > > >     >     > IG Index
>> > > >     > > > > >     > > >     >     >         >> > Limited (a company
>> > > > registered in
>> > > >     > > > > > England and
>> > > >     > > > > >     > > > Wales,
>> > > >     > > > > >     > > >     > company
>> > > >     > > > > >     > > >     >     > number
>> > > >     > > > > >     > > >     >     >         >> > 01190902).
>> Registered
>> > > > address at
>> > > >     > > > > Cannon
>> > > >     > > > > > Bridge
>> > > >     > > > > >     > > > House, 25
>> > > >     > > > > >     > > >     >     > Dowgate Hill,
>> > > >     > > > > >     > > >     >     >         >> > London EC4R 2YA.
>> Both
>> > IG
>> > > > Markets
>> > > >     > > > > Limited
>> > > >     > > > > >     > > (register
>> > > >     > > > > >     > > >     > number
>> > > >     > > > > >     > > >     >     > 195355) and IG
>> > > >     > > > > >     > > >     >     >         >> > Index Limited
>> > (register
>> > > > number
>> > > >     > > > 114059)
>> > > >     > > > > > are
>> > > >     > > > > >     > > > authorised
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     > regulated by
>> > > >     > > > > >     > > >     >     >         >> the
>> > > >     > > > > >     > > >     >     >         >> > Financial Conduct
>> > > > Authority.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> --
>> > > >     > > > > >     > > >     >     >         >> -Regards,
>> > > >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
>> > > >     > > > > >     > > >     >     >         >> (862) 250-7125
>> > > >     > > > > >     > > >     >     >         >> The information
>> > contained
>> > > > in this
>> > > >     > > > email
>> > > >     > > > > is
>> > > >     > > > > >     > > strictly
>> > > >     > > > > >     > > >     >     > confidential and for
>> > > >     > > > > >     > > >     >     >         >> the use of the
>> addressee
>> > > > only,
>> > > >     > > unless
>> > > >     > > > > > otherwise
>> > > >     > > > > >     > > >     > indicated. If
>> > > >     > > > > >     > > >     >     > you are not
>> > > >     > > > > >     > > >     >     >         >> the intended
>> recipient,
>> > > > please do
>> > > >     > > not
>> > > >     > > > > > read,
>> > > >     > > > > >     > copy,
>> > > >     > > > > >     > > > use or
>> > > >     > > > > >     > > >     >     > disclose to others
>> > > >     > > > > >     > > >     >     >         >> this message or any
>> > > > attachment.
>> > > >     > > Please
>> > > >     > > > > > also
>> > > >     > > > > >     > notify
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > sender
>> > > >     > > > > >     > > >     >     > by replying
>> > > >     > > > > >     > > >     >     >         >> to this email or by
>> > > > telephone
>> > > >     > > (+44(020
>> > > >     > > > > > 7896
>> > > >     > > > > >     > 0011)
>> > > >     > > > > >     > > > and then
>> > > >     > > > > >     > > >     >     > delete the email
>> > > >     > > > > >     > > >     >     >         >> and any copies of it.
>> > > > Opinions,
>> > > >     > > > > > conclusion (etc)
>> > > >     > > > > >     > > > that do
>> > > >     > > > > >     > > >     > not
>> > > >     > > > > >     > > >     >     > relate to the
>> > > >     > > > > >     > > >     >     >         >> official business of
>> > this
>> > > > company
>> > > >     > > > shall
>> > > >     > > > > be
>> > > >     > > > > >     > > > understood as
>> > > >     > > > > >     > > >     >     > neither given nor
>> > > >     > > > > >     > > >     >     >         >> endorsed by it. IG
>> is a
>> > > > trading
>> > > >     > name
>> > > >     > > > of
>> > > >     > > > > IG
>> > > >     > > > > >     > Markets
>> > > >     > > > > >     > > >     > Limited (a
>> > > >     > > > > >     > > >     >     > company
>> > > >     > > > > >     > > >     >     >         >> registered in England
>> > and
>> > > > Wales,
>> > > >     > > > company
>> > > >     > > > > > number
>> > > >     > > > > >     > > > 04008957)
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     > IG Index
>> > > >     > > > > >     > > >     >     >         >> Limited (a company
>> > > > registered in
>> > > >     > > > England
>> > > >     > > > > > and
>> > > >     > > > > >     > > Wales,
>> > > >     > > > > >     > > >     > company
>> > > >     > > > > >     > > >     >     > number
>> > > >     > > > > >     > > >     >     >         >> 01190902). Registered
>> > > > address at
>> > > >     > > > Cannon
>> > > >     > > > > > Bridge
>> > > >     > > > > >     > > > House, 25
>> > > >     > > > > >     > > >     >     > Dowgate Hill,
>> > > >     > > > > >     > > >     >     >         >> London EC4R 2YA.
>> Both IG
>> > > > Markets
>> > > >     > > > Limited
>> > > >     > > > > >     > (register
>> > > >     > > > > >     > > > number
>> > > >     > > > > >     > > >     >     > 195355) and IG
>> > > >     > > > > >     > > >     >     >         >> Index Limited
>> (register
>> > > > number
>> > > >     > > 114059)
>> > > >     > > > > are
>> > > >     > > > > >     > > > authorised and
>> > > >     > > > > >     > > >     >     > regulated by the
>> > > >     > > > > >     > > >     >     >         >> Financial Conduct
>> > > Authority.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > --
>> > > >     > > > > >     > > >     >     >         > -Regards,
>> > > >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
>> > > >     > > > > >     > > >     >     >         > (862) 250-7125
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         --
>> > > >     > > > > >     > > >     >     >         -Regards,
>> > > >     > > > > >     > > >     >     >         Mayuresh R. Gharat
>> > > >     > > > > >     > > >     >     >         (862) 250-7125
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     The information contained in
>> > this
>> > > > email
>> > > >     > is
>> > > >     > > > > > strictly
>> > > >     > > > > >     > > > confidential
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     > for the use of the addressee
>> only,
>> > > > unless
>> > > >     > > > otherwise
>> > > >     > > > > >     > > indicated.
>> > > >     > > > > >     > > > If
>> > > >     > > > > >     > > >     > you are
>> > > >     > > > > >     > > >     >     > not the intended recipient,
>> please
>> > do
>> > > > not
>> > > >     > read,
>> > > >     > > > > > copy, use
>> > > >     > > > > >     > or
>> > > >     > > > > >     > > >     > disclose to
>> > > >     > > > > >     > > >     >     > others this message or any
>> > > attachment.
>> > > > Please
>> > > >     > > > also
>> > > >     > > > > > notify
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > > sender
>> > > >     > > > > >     > > >     > by
>> > > >     > > > > >     > > >     >     > replying to this email or by
>> > > telephone
>> > > >     > (+44(020
>> > > >     > > > > 7896
>> > > >     > > > > > 0011)
>> > > >     > > > > >     > > and
>> > > >     > > > > >     > > > then
>> > > >     > > > > >     > > >     > delete
>> > > >     > > > > >     > > >     >     > the email and any copies of it.
>> > > > Opinions,
>> > > >     > > > > conclusion
>> > > >     > > > > > (etc)
>> > > >     > > > > >     > > > that do
>> > > >     > > > > >     > > >     > not
>> > > >     > > > > >     > > >     >     > relate to the official business
>> of
>> > > this
>> > > >     > company
>> > > >     > > > > > shall be
>> > > >     > > > > >     > > > understood
>> > > >     > > > > >     > > >     > as
>> > > >     > > > > >     > > >     >     > neither given nor endorsed by
>> it.
>> > IG
>> > > > is a
>> > > >     > > trading
>> > > >     > > > > > name of
>> > > >     > > > > >     > IG
>> > > >     > > > > >     > > > Markets
>> > > >     > > > > >     > > >     >     > Limited (a company registered in
>> > > > England and
>> > > >     > > > Wales,
>> > > >     > > > > > company
>> > > >     > > > > >     > > > number
>> > > >     > > > > >     > > >     >     > 04008957) and IG Index Limited
>> (a
>> > > > company
>> > > >     > > > > registered
>> > > >     > > > > > in
>> > > >     > > > > >     > > > England and
>> > > >     > > > > >     > > >     > Wales,
>> > > >     > > > > >     > > >     >     > company number 01190902).
>> > Registered
>> > > > address
>> > > >     > at
>> > > >     > > > > > Cannon
>> > > >     > > > > >     > Bridge
>> > > >     > > > > >     > > > House,
>> > > >     > > > > >     > > >     > 25
>> > > >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA.
>> Both
>> > > IG
>> > > >     > Markets
>> > > >     > > > > > Limited
>> > > >     > > > > >     > > > (register
>> > > >     > > > > >     > > >     > number
>> > > >     > > > > >     > > >     >     > 195355) and IG Index Limited
>> > > (register
>> > > > number
>> > > >     > > > > > 114059) are
>> > > >     > > > > >     > > > authorised
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     > regulated by the Financial
>> Conduct
>> > > > Authority.
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > The information contained in this
>> email
>> > is
>> > > > strictly
>> > > >     > > > > > confidential
>> > > >     > > > > >     > > and
>> > > >     > > > > >     > > > for
>> > > >     > > > > >     > > >     > the use of the addressee only, unless
>> > > > otherwise
>> > > >     > > > > indicated.
>> > > >     > > > > > If you
>> > > >     > > > > >     > > > are not
>> > > >     > > > > >     > > >     > the intended recipient, please do not
>> > read,
>> > > > copy,
>> > > >     > use
>> > > >     > > > or
>> > > >     > > > > > disclose
>> > > >     > > > > >     > > to
>> > > >     > > > > >     > > > others
>> > > >     > > > > >     > > >     > this message or any attachment. Please
>> > also
>> > > > notify
>> > > >     > > the
>> > > >     > > > > > sender by
>> > > >     > > > > >     > > > replying
>> > > >     > > > > >     > > >     > to this email or by telephone (+44(020
>> > 7896
>> > > > 0011
>> > > >     > <020%207896%200011>) and
>> > > >     > > > > then
>> > > >     > > > > > delete
>> > > >     > > > > >     > > > the email
>> > > >     > > > > >     > > >     > and any copies of it. Opinions,
>> > conclusion
>> > > > (etc)
>> > > >     > that
>> > > >     > > > do
>> > > >     > > > > > not
>> > > >     > > > > >     > relate
>> > > >     > > > > >     > > > to the
>> > > >     > > > > >     > > >     > official business of this company
>> shall
>> > be
>> > > >     > understood
>> > > >     > > > as
>> > > >     > > > > > neither
>> > > >     > > > > >     > > > given nor
>> > > >     > > > > >     > > >     > endorsed by it. IG is a trading name
>> of
>> > IG
>> > > > Markets
>> > > >     > > > > Limited
>> > > >     > > > > > (a
>> > > >     > > > > >     > > company
>> > > >     > > > > >     > > >     > registered in England and Wales,
>> company
>> > > > number
>> > > >     > > > 04008957)
>> > > >     > > > > > and IG
>> > > >     > > > > >     > > > Index
>> > > >     > > > > >     > > >     > Limited (a company registered in
>> England
>> > > and
>> > > > Wales,
>> > > >     > > > > company
>> > > >     > > > > >     > number
>> > > >     > > > > >     > > >     > 01190902). Registered address at
>> Cannon
>> > > > Bridge
>> > > >     > House,
>> > > >     > > > 25
>> > > >     > > > > > Dowgate
>> > > >     > > > > >     > > > Hill,
>> > > >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register
>> > > >     > > > number
>> > > >     > > > > > 195355)
>> > > >     > > > > >     > > > and IG
>> > > >     > > > > >     > > >     > Index Limited (register number 114059)
>> > are
>> > > >     > authorised
>> > > >     > > > and
>> > > >     > > > > >     > regulated
>> > > >     > > > > >     > > > by the
>> > > >     > > > > >     > > >     > Financial Conduct Authority.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > The information contained in this email is
>> > > strictly
>> > > >     > > > > confidential
>> > > >     > > > > > and
>> > > >     > > > > >     > for
>> > > >     > > > > >     > > > the use of the addressee only, unless
>> otherwise
>> > > >     > indicated.
>> > > >     > > If
>> > > >     > > > > > you are
>> > > >     > > > > >     > not
>> > > >     > > > > >     > > > the intended recipient, please do not read,
>> > copy,
>> > > > use or
>> > > >     > > > > > disclose to
>> > > >     > > > > >     > > others
>> > > >     > > > > >     > > > this message or any attachment. Please also
>> > > notify
>> > > > the
>> > > >     > > sender
>> > > >     > > > > by
>> > > >     > > > > >     > replying
>> > > >     > > > > >     > > > to this email or by telephone (+44(020 7896
>> > 0011
>> > > >     > <020%207896%200011>) and then
>> > > >     > > > > > delete the
>> > > >     > > > > >     > > email
>> > > >     > > > > >     > > > and any copies of it. Opinions, conclusion
>> > (etc)
>> > > > that do
>> > > >     > > not
>> > > >     > > > > > relate to
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > > official business of this company shall be
>> > > > understood as
>> > > >     > > > > neither
>> > > >     > > > > > given
>> > > >     > > > > >     > > nor
>> > > >     > > > > >     > > > endorsed by it. IG is a trading name of IG
>> > > Markets
>> > > >     > Limited
>> > > >     > > (a
>> > > >     > > > > > company
>> > > >     > > > > >     > > > registered in England and Wales, company
>> number
>> > > > 04008957)
>> > > >     > > and
>> > > >     > > > > IG
>> > > >     > > > > > Index
>> > > >     > > > > >     > > > Limited (a company registered in England and
>> > > Wales,
>> > > >     > company
>> > > >     > > > > > number
>> > > >     > > > > >     > > > 01190902). Registered address at Cannon
>> Bridge
>> > > > House, 25
>> > > >     > > > > Dowgate
>> > > >     > > > > > Hill,
>> > > >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
>> > > (register
>> > > > number
>> > > >     > > > > > 195355) and
>> > > >     > > > > >     > IG
>> > > >     > > > > >     > > > Index Limited (register number 114059) are
>> > > > authorised and
>> > > >     > > > > > regulated by
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > > Financial Conduct Authority.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > The information contained in this email is
>> > strictly
>> > > >     > > > confidential
>> > > >     > > > > > and for
>> > > >     > > > > >     > > the use of the addressee only, unless
>> otherwise
>> > > > indicated.
>> > > >     > If
>> > > >     > > > you
>> > > >     > > > > > are not
>> > > >     > > > > >     > > the intended recipient, please do not read,
>> copy,
>> > > > use or
>> > > >     > > > disclose
>> > > >     > > > > > to
>> > > >     > > > > >     > others
>> > > >     > > > > >     > > this message or any attachment. Please also
>> > notify
>> > > > the
>> > > >     > sender
>> > > >     > > > by
>> > > >     > > > > > replying
>> > > >     > > > > >     > > to this email or by telephone (+44(020 7896
>> 0011
>> > > >     > <020%207896%200011>) and then
>> > > >     > > > delete
>> > > >     > > > > > the
>> > > >     > > > > >     > email
>> > > >     > > > > >     > > and any copies of it. Opinions, conclusion
>> (etc)
>> > > > that do
>> > > >     > not
>> > > >     > > > > > relate to
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > official business of this company shall be
>> > > > understood as
>> > > >     > > > neither
>> > > >     > > > > > given
>> > > >     > > > > >     > nor
>> > > >     > > > > >     > > endorsed by it. IG is a trading name of IG
>> > Markets
>> > > > Limited
>> > > >     > (a
>> > > >     > > > > > company
>> > > >     > > > > >     > > registered in England and Wales, company
>> number
>> > > > 04008957)
>> > > >     > and
>> > > >     > > > IG
>> > > >     > > > > > Index
>> > > >     > > > > >     > > Limited (a company registered in England and
>> > Wales,
>> > > > company
>> > > >     > > > > number
>> > > >     > > > > >     > > 01190902). Registered address at Cannon Bridge
>> > > > House, 25
>> > > >     > > > Dowgate
>> > > >     > > > > > Hill,
>> > > >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited
>> > (register
>> > > > number
>> > > >     > > > 195355)
>> > > >     > > > > > and IG
>> > > >     > > > > >     > > Index Limited (register number 114059) are
>> > > > authorised and
>> > > >     > > > > > regulated by
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > Financial Conduct Authority.
>> > > >     > > > > >     > >
>> > > >     > > > > >     > >
>> > > >     > > > > >     > The information contained in this email is
>> strictly
>> > > >     > > confidential
>> > > >     > > > > and
>> > > >     > > > > > for
>> > > >     > > > > >     > the use of the addressee only, unless otherwise
>> > > > indicated. If
>> > > >     > > you
>> > > >     > > > > > are not
>> > > >     > > > > >     > the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or
>> > > >     > > disclose
>> > > >     > > > > to
>> > > >     > > > > > others
>> > > >     > > > > >     > this message or any attachment. Please also
>> notify
>> > > the
>> > > > sender
>> > > >     > > by
>> > > >     > > > > > replying
>> > > >     > > > > >     > to this email or by telephone (+44(020 7896 0011
>> > > >     > <020%207896%200011>) and then
>> > > >     > > delete
>> > > >     > > > > > the email
>> > > >     > > > > >     > and any copies of it. Opinions, conclusion (etc)
>> > that
>> > > > do not
>> > > >     > > > relate
>> > > >     > > > > > to the
>> > > >     > > > > >     > official business of this company shall be
>> > understood
>> > > > as
>> > > >     > > neither
>> > > >     > > > > > given nor
>> > > >     > > > > >     > endorsed by it. IG is a trading name of IG
>> Markets
>> > > > Limited (a
>> > > >     > > > > company
>> > > >     > > > > >     > registered in England and Wales, company number
>> > > > 04008957) and
>> > > >     > > IG
>> > > >     > > > > > Index
>> > > >     > > > > >     > Limited (a company registered in England and
>> Wales,
>> > > > company
>> > > >     > > > number
>> > > >     > > > > >     > 01190902). Registered address at Cannon Bridge
>> > House,
>> > > > 25
>> > > >     > > Dowgate
>> > > >     > > > > > Hill,
>> > > >     > > > > >     > London EC4R 2YA. Both IG Markets Limited
>> (register
>> > > > number
>> > > >     > > 195355)
>> > > >     > > > > > and IG
>> > > >     > > > > >     > Index Limited (register number 114059) are
>> > authorised
>> > > > and
>> > > >     > > > regulated
>> > > >     > > > > > by the
>> > > >     > > > > >     > Financial Conduct Authority.
>> > > >     > > > > >     >
>> > > >     > > > > >     >
>> > > >     > > > > >
>> > > >     > > > > >
>> > > >     > > > > > The information contained in this email is strictly
>> > > > confidential
>> > > >     > and
>> > > >     > > > for
>> > > >     > > > > > the use of the addressee only, unless otherwise
>> > indicated.
>> > > > If you
>> > > >     > are
>> > > >     > > > not
>> > > >     > > > > > the intended recipient, please do not read, copy, use
>> or
>> > > > disclose
>> > > >     > to
>> > > >     > > > > others
>> > > >     > > > > > this message or any attachment. Please also notify the
>> > > > sender by
>> > > >     > > > replying
>> > > >     > > > > > to this email or by telephone (+44(020 7896 0011
>> > > >     > <020%207896%200011>) and then delete the
>> > > >     > > > > email
>> > > >     > > > > > and any copies of it. Opinions, conclusion (etc) that
>> do
>> > > not
>> > > > relate
>> > > >     > > to
>> > > >     > > > > the
>> > > >     > > > > > official business of this company shall be understood
>> as
>> > > > neither
>> > > >     > > given
>> > > >     > > > > nor
>> > > >     > > > > > endorsed by it. IG is a trading name of IG Markets
>> > Limited
>> > > (a
>> > > >     > company
>> > > >     > > > > > registered in England and Wales, company number
>> 04008957)
>> > > > and IG
>> > > >     > > Index
>> > > >     > > > > > Limited (a company registered in England and Wales,
>> > company
>> > > > number
>> > > >     > > > > > 01190902). Registered address at Cannon Bridge House,
>> 25
>> > > > Dowgate
>> > > >     > > Hill,
>> > > >     > > > > > London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > > > 195355)
>> > > >     > and
>> > > >     > > > IG
>> > > >     > > > > > Index Limited (register number 114059) are authorised
>> and
>> > > > regulated
>> > > >     > > by
>> > > >     > > > > the
>> > > >     > > > > > Financial Conduct Authority.
>> > > >     > > > > >
>> > > >     > > > > The information contained in this email is strictly
>> > > > confidential and
>> > > >     > > for
>> > > >     > > > > the use of the addressee only, unless otherwise
>> indicated.
>> > If
>> > > > you are
>> > > >     > > not
>> > > >     > > > > the intended recipient, please do not read, copy, use or
>> > > > disclose to
>> > > >     > > > others
>> > > >     > > > > this message or any attachment. Please also notify the
>> > sender
>> > > > by
>> > > >     > > replying
>> > > >     > > > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > >     > and then delete the
>> > > >     > > > email
>> > > >     > > > > and any copies of it. Opinions, conclusion (etc) that do
>> > not
>> > > > relate
>> > > >     > to
>> > > >     > > > the
>> > > >     > > > > official business of this company shall be understood as
>> > > > neither
>> > > >     > given
>> > > >     > > > nor
>> > > >     > > > > endorsed by it. IG is a trading name of IG Markets
>> Limited
>> > (a
>> > > > company
>> > > >     > > > > registered in England and Wales, company number
>> 04008957)
>> > and
>> > > > IG
>> > > >     > Index
>> > > >     > > > > Limited (a company registered in England and Wales,
>> company
>> > > > number
>> > > >     > > > > 01190902). Registered address at Cannon Bridge House, 25
>> > > > Dowgate
>> > > >     > Hill,
>> > > >     > > > > London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > > > 195355) and
>> > > >     > > IG
>> > > >     > > > > Index Limited (register number 114059) are authorised
>> and
>> > > > regulated
>> > > >     > by
>> > > >     > > > the
>> > > >     > > > > Financial Conduct Authority.
>> > > >     > > > >
>> > > >     > > > The information contained in this email is strictly
>> > > confidential
>> > > > and
>> > > >     > for
>> > > >     > > > the use of the addressee only, unless otherwise
>> indicated. If
>> > > > you are
>> > > >     > not
>> > > >     > > > the intended recipient, please do not read, copy, use or
>> > > > disclose to
>> > > >     > > others
>> > > >     > > > this message or any attachment. Please also notify the
>> sender
>> > > by
>> > > >     > replying
>> > > >     > > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > >     > and then delete the
>> > > >     > > email
>> > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
>> not
>> > > > relate to
>> > > >     > > the
>> > > >     > > > official business of this company shall be understood as
>> > > neither
>> > > > given
>> > > >     > > nor
>> > > >     > > > endorsed by it. IG is a trading name of IG Markets
>> Limited (a
>> > > > company
>> > > >     > > > registered in England and Wales, company number 04008957)
>> and
>> > > IG
>> > > > Index
>> > > >     > > > Limited (a company registered in England and Wales,
>> company
>> > > > number
>> > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
>> > > Dowgate
>> > > > Hill,
>> > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
>> > > > 195355) and
>> > > >     > IG
>> > > >     > > > Index Limited (register number 114059) are authorised and
>> > > > regulated by
>> > > >     > > the
>> > > >     > > > Financial Conduct Authority.
>> > > >     > > >
>> > > >     > > The information contained in this email is strictly
>> > confidential
>> > > > and for
>> > > >     > > the use of the addressee only, unless otherwise indicated.
>> If
>> > you
>> > > > are not
>> > > >     > > the intended recipient, please do not read, copy, use or
>> > disclose
>> > > > to
>> > > >     > others
>> > > >     > > this message or any attachment. Please also notify the
>> sender
>> > by
>> > > > replying
>> > > >     > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > >     > and then delete the email
>> > > >     > > and any copies of it. Opinions, conclusion (etc) that do not
>> > > > relate to
>> > > >     > the
>> > > >     > > official business of this company shall be understood as
>> > neither
>> > > > given
>> > > >     > nor
>> > > >     > > endorsed by it. IG is a trading name of IG Markets Limited
>> (a
>> > > > company
>> > > >     > > registered in England and Wales, company number 04008957)
>> and
>> > IG
>> > > > Index
>> > > >     > > Limited (a company registered in England and Wales, company
>> > > number
>> > > >     > > 01190902). Registered address at Cannon Bridge House, 25
>> > Dowgate
>> > > > Hill,
>> > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
>> > 195355)
>> > > > and IG
>> > > >     > > Index Limited (register number 114059) are authorised and
>> > > > regulated by
>> > > >     > the
>> > > >     > > Financial Conduct Authority.
>> > > >     > >
>> > > >     >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If you
>> are
>> > not
>> > > > the intended recipient, please do not read, copy, use or disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then delete
>> the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
>> to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and regulated
>> by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > > The information contained in this email is strictly confidential and
>> for
>> > > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > > the intended recipient, please do not read, copy, use or disclose to
>> > others
>> > > this message or any attachment. Please also notify the sender by
>> replying
>> > > to this email or by telephone (+44(020 7896 0011) and then delete the
>> > email
>> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> > the
>> > > official business of this company shall be understood as neither given
>> > nor
>> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > > registered in England and Wales, company number 04008957) and IG Index
>> > > Limited (a company registered in England and Wales, company number
>> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
>> IG
>> > > Index Limited (register number 114059) are authorised and regulated by
>> > the
>> > > Financial Conduct Authority.
>> > >
>> > The information contained in this email is strictly confidential and for
>> > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > the intended recipient, please do not read, copy, use or disclose to
>> others
>> > this message or any attachment. Please also notify the sender by
>> replying
>> > to this email or by telephone (+44(020 7896 0011) and then delete the
>> email
>> > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> the
>> > official business of this company shall be understood as neither given
>> nor
>> > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > registered in England and Wales, company number 04008957) and IG Index
>> > Limited (a company registered in England and Wales, company number
>> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> > Index Limited (register number 114059) are authorised and regulated by
>> the
>> > Financial Conduct Authority.
>> >
>>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[radai <ra...@gmail.com>]

i didnt mean to sound as insisting. what i actually mean is it would still
be a valid issue but of much less concern.

On Mon, Dec 19, 2016 at 7:50 PM, radai <ra...@gmail.com> wrote:

> this kip fixes a "bug" (quirk?) that arises when people implement headers
> "in V" (in the payload part of a message).
> if you have proper headers you obviously dont need to to stick them in V
> and so wont run into this, but its still a valid issue.
>
> On Mon, Dec 19, 2016 at 3:06 PM, Jay Kreps <ja...@confluent.io> wrote:
>
>> Makes sense!
>>
>> -Jay
>>
>> On Mon, Dec 19, 2016 at 2:40 PM, Michael Pearce <Mi...@ig.com>
>> wrote:
>>
>> > Wow just read that def over tired. Hopefully it makes sense. Or you get
>> > the gist at least.
>> >
>> > ________________________________________
>> > From: Michael Pearce <Mi...@ig.com>
>> > Sent: Monday, December 19, 2016 9:19:02 PM
>> > To: dev@kafka.apache.org
>> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> >
>> > Hi Jay,
>> >
>> > Agreed this stemmed as offshoot from KIP-82.
>> >
>> > Which our main driver for was to be able to have some headers for a null
>> > value as such for our routing, audit, tracing and a few other bits which
>> > currently we are forced to do with a message wrapper, if we all agreed
>> on
>> > KIP-82 that we need native headers and look to implement that the push
>> for
>> > this would dissipate.
>> >
>> > This KIP would allow for though one use case that comes to mind we could
>> > see which is to have business data with a delete. Though as said this
>> isn't
>> > something we are pushing for think really we would have.
>> >
>> > As such in summary yes, if you want to fully support KIP-82 and we can
>> get
>> > that agreed in principle and a target release version, I think quite a
>> few
>> > guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.
>> >
>> > Cheers
>> > Mike
>> >
>> > Sent using OWA for iPhone
>> > ________________________________________
>> > From: Jay Kreps <ja...@confluent.io>
>> > Sent: Monday, December 19, 2016 8:51:23 PM
>> > To: dev@kafka.apache.org
>> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> >
>> > Hey Michael,
>> >
>> > Here is the compatibility concern I have:
>> >
>> >    1. You have a consumer app that relies on value == null to indicate a
>> >    delete (current semantics).
>> >    2. You upgrade Kafka and your clients.
>> >    3. Some producer starts using the tombstone field in combination with
>> >    non-null.
>> >
>> > I share Ismael's dislike of setting tombstones on records with null
>> values.
>> > This makes sense as a transitional state, but as an end state its a bit
>> > weird. You'd expect to be able to mix null values and tombstones, and
>> have
>> > the null values remain and the tombstones get compacted. However what
>> will
>> > happen is both will be compacted and upon debugging this you'll learn
>> that
>> > we sometimes use null in the value to indicate tombstone. Ismael's
>> solution
>> > is a bigger compatibility break, though, so not sure if that is better.
>> >
>> > My other question is the relationship to KIP-82. My read is that this
>> KIP
>> > solves some but not all of the problems KIP-82 is intended for. KIP-82,
>> on
>> > the other hand, seems to address most of the motivating uses for this
>> KIP.
>> > The exception is maybe item (5) on the list where you want to
>> simultaneous
>> > delete and convey some information to subscribers, but I couldn't
>> construct
>> > a concrete examples for that one. Do we need to rationalize these two
>> KIPs?
>> > That is, do you still advocate this proposal if we do KIP-82 and vice
>> > versa? As you may have noticed I'm somewhat emotionally invested in the
>> > simplicity of the core data model, so my default position is let's try
>> to
>> > avoid stuffing more stuff in, but if we have to add stuff I like each of
>> > these individually more than doing both. :-)
>> >
>> > -Jay
>> >
>> >
>> >
>> >
>> > On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Michael.Pearce@ig.com
>> >
>> > wrote:
>> >
>> > > Hi Jay
>> > >
>> > > I disagree here that we are breaking any compatibility, we went
>> through
>> > > this on the discussion thread in fact with the help of that thread is
>> how
>> > > the kip came to the solution.
>> > >
>> > > Also on the supported combinations front you mention, we are not
>> taking
>> > > anything away afaik.
>> > >
>> > > Currently supported are only are:
>> > > Null value = delete
>> > > Non-null value = non delete
>> > >
>> > > With this kip we would support
>> > > Null value + tombstone = delete
>> > > Non null value + tombstone = delete
>> > > Non null value + no tombstone = non delete
>> > >
>> > > As for the alternative idea, this is simply a new policy, how can
>> there
>> > be
>> > > confusion here? For this policy it would be explicit that tombstone
>> > marker
>> > > would need to be set for a delete.
>> > >
>> > > I'm going to vent a little now as starting to get quite frustrated.
>> > >
>> > > We are going round in circles on kip-82 as per use cases there is now
>> > many
>> > > use cases, how many more are needed? just because confluent don't see
>> > these
>> > > doesn't mean they aren't real use cases other have, this is the point
>> of
>> > > the Apache foundation, it shouldn't be the view of just one
>> organisation.
>> > > It really is getting a feeling of the NIH syndrome. Rather than it
>> being
>> > > constructive on discussion of the implementation detail.
>> > >
>> > > kip-87 spawned from as on the kip call we all agreed this was needed.
>> And
>> > > would at least allow a custom wrapper be supported in a compacted
>> topic,
>> > > allowing meta data. Which again now I feel we are spinning wheels, and
>> > > simply finding reasons not support it.
>> > >
>> > > Cheers
>> > > Mike
>> > >
>> > >
>> > >
>> > > Sent using OWA for iPad
>> > > ________________________________________
>> > > From: Jay Kreps <ja...@confluent.io>
>> > > Sent: Friday, December 16, 2016 7:09:23 PM
>> > > To: dev@kafka.apache.org
>> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > >
>> > > Hey Michael,
>> > >
>> > > I do think it might have been better had we started with a separate
>> > concept
>> > > of null vs delete. Given that we are where we are, I'm not sure that
>> the
>> > > possible cures we explored so far are better than the disease.
>> > >
>> > > I apologize for coming to this late, but I didn't really understand
>> the
>> > > implications of the KIP and that we'd be breaking compatibility with
>> > > existing apps until the vote had begun, and in my defense I'm not sure
>> > the
>> > > other folks voting did either.
>> > >
>> > > I think we all agree there are many existing apps that are built with
>> the
>> > > assumption of "null value non-tombstone" and it isn't possible to
>> > > disambiguate these from tombstones on the producer. It isn't that
>> anyone
>> > is
>> > > saying we have to support all four possibilities at once, it's that we
>> > > simply can't orphan one of the existing combinations or our users will
>> > eat
>> > > us!
>> > >
>> > > If I've understood your alternate solution of adding another setting
>> for
>> > > compaction, I think this does fix the compatibility problem, but it
>> adds
>> > an
>> > > odd mode the user has to add on all their topics. While the current
>> state
>> > > is easily explainable, the resulting state where the meaning of
>> tombstone
>> > > and null are overlapping and ambiguous and dependent on a compaction
>> > > setting that could change out of band or not be in sync with your
>> code in
>> > > some environment seems worse to me then where we currently are. I
>> think
>> > the
>> > > question is how would this combination be explained to users and does
>> it
>> > > make sense?
>> > >
>> > > -Jay
>> > >
>> > >
>> > >
>> > > On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > Hi Chaps,
>> > > >
>> > > > Can we either get one more +1 binding (we have 2 already) on the
>> > > existing?
>> > > >
>> > > > Or have some response on the below possible alternative. We are
>> keen to
>> > > > get working on this, so we make next feature release.
>> > > >
>> > > > Cheers
>> > > > Mike
>> > > >
>> > > >
>> > > > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com>
>> wrote:
>> > > >
>> > > >     Hi Ismael
>> > > >
>> > > >     Did you see our email this morning, what's your thoughts on this
>> > > > approach to instead we simply have a brand new policy?
>> > > >
>> > > >     Cheers
>> > > >     Mike
>> > > >
>> > > >
>> > > >     Sent using OWA for iPhone
>> > > >     ________________________________________
>> > > >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
>> > > Juma <
>> > > > ismael@juma.me.uk>
>> > > >     Sent: Tuesday, December 13, 2016 11:30:05 AM
>> > > >     To: dev@kafka.apache.org
>> > > >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > >
>> > > >     Yes, this is actually tricky to do in a way where we both have
>> the
>> > > > desired
>> > > >     semantics and maintain compatibility. When someone creates a
>> > > >     `ProducerRecord` with a `null` value today, the producer doesn't
>> > know
>> > > > if
>> > > >     it's meant to be a tombstone or not. For V3 messages, it's easy
>> > when
>> > > > the
>> > > >     constructor that takes a tombstone is used. However, if any
>> other
>> > > >     constructor is used, it's not clear. A couple of options I can
>> > think
>> > > > of,
>> > > >     none of them particularly nice:
>> > > >
>> > > >     1. Have a third state where tombstone = unknown and the broker
>> > would
>> > > > set
>> > > >     the tombstone bit if the value was null and the topic was
>> > compacted.
>> > > > People
>> > > >     that wanted to pass a non-null value for the tombstone would
>> have
>> > to
>> > > > use
>> > > >     the constructor that takes a tombstone. The drawbacks: third
>> state
>> > > for
>> > > >     tombstone in message format, message conversion at the broker
>> for a
>> > > > common
>> > > >     case.
>> > > >
>> > > >     2. Extend MetadataResponse to optionally include topic configs,
>> > which
>> > > > would
>> > > >     make it possible for the producer to be smarter about setting
>> the
>> > > >     tombstone. It would only do it if a tombstone was not passed
>> > > > explicitly,
>> > > >     the value was null and the topic was compacted. The main
>> drawback
>> > is
>> > > > that
>> > > >     the producer would be getting a bit more data for each topic
>> even
>> > > > though it
>> > > >     probably won't use it most of the time. Extending
>> MetadataResponse
>> > to
>> > > >     return topic configs would be useful for other reasons as well,
>> so
>> > > that
>> > > >     part seems OK.
>> > > >
>> > > >     In addition, for both proposals, we could consider adding
>> warnings
>> > to
>> > > > the
>> > > >     documentation that the behaviour of the constructors that don't
>> > take
>> > > a
>> > > >     tombstone would change in the next major release so that
>> tombstone
>> > =
>> > > > false.
>> > > >     Not sure if this would be worth it though.
>> > > >
>> > > >     Ismael
>> > > >
>> > > >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
>> > > > ewen@confluent.io>
>> > > >     wrote:
>> > > >
>> > > >     > Michael,
>> > > >     >
>> > > >     > It kind of depends on how you want to interpret the tombstone
>> > flag.
>> > > > If it's
>> > > >     > purely a producer-facing Kafka-level thing that we treat as
>> > > internal
>> > > > to the
>> > > >     > broker and log cleaner once the record is sent, then your
>> > approach
>> > > > makes
>> > > >     > sense. You're just moving copying the null-indicates-delete
>> > > behavior
>> > > > of the
>> > > >     > old constructor into the tombstone flag.
>> > > >     >
>> > > >     > However, if you want this change to more generally decouple
>> the
>> > > idea
>> > > > of
>> > > >     > deletion and null values, then you are sometimes converting
>> what
>> > > > might be a
>> > > >     > completely valid null value that doesn't indicate deletion
>> into a
>> > > >     > tombstone. Downstream applications could potentially handle
>> these
>> > > > cases
>> > > >     > differently given the separation of deletion from value.
>> > > >     >
>> > > >     > I guess the question is if we want to try to support the
>> latter
>> > > even
>> > > > for
>> > > >     > topics where we have older produce requests. An example where
>> > this
>> > > > could
>> > > >     > come up is in something like a CDC Connector. If we try to
>> > support
>> > > > the
>> > > >     > semantic difference, a connector might write changes to Kafka
>> > using
>> > > > the
>> > > >     > tombstone flag to indicate when a row was truly deleted (vs an
>> > > > update that
>> > > >     > sets it to null but still present; this probably makes more
>> sense
>> > > > for CDC
>> > > >     > from document stores or extracting single columns). There are
>> > > various
>> > > >     > reasons we might want to maintain the full log and not turn
>> > > > compaction on
>> > > >     > (or just use a time-based retention policy), but downstream
>> > > > applications
>> > > >     > might care to know the difference between a delete and a null
>> > > value.
>> > > > In
>> > > >     > fact both versions of the same log (compacted and
>> time-retention)
>> > > > could be
>> > > >     > useful and I don't think it'll be uncommon to maintain both or
>> > use
>> > > > KIP-71
>> > > >     > to maintain a hybrid compacted/retention topic.
>> > > >     >
>> > > >     > -Ewen
>> > > >     >
>> > > >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > >     > wrote:
>> > > >     >
>> > > >     > > Hi Jay,
>> > > >     > >
>> > > >     > > Why wouldn't that work, the tombstone value is only looked
>> at
>> > by
>> > > > the
>> > > >     > > broker, on a topic configured for compaction as such is
>> benign
>> > on
>> > > > non
>> > > >     > > compacted topics. This is as much as sending a null value
>> > > currently
>> > > >     > >
>> > > >     > >
>> > > >     > > Regards
>> > > >     > > Mike
>> > > >     > >
>> > > >     > >
>> > > >     > >
>> > > >     > > Sent using OWA for iPhone
>> > > >     > > ________________________________________
>> > > >     > > From: Jay Kreps <ja...@confluent.io>
>> > > >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
>> > > >     > > To: dev@kafka.apache.org
>> > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > >     > >
>> > > >     > > Hey Michael,
>> > > >     > >
>> > > >     > > I'm not quite sure that works as that would translate ALL
>> null
>> > > > values to
>> > > >     > > tombstones, even for non-compacted topics that use null as
>> an
>> > > > acceptable
>> > > >     > > value sent by the producer and expected by the consumer.
>> > > >     > >
>> > > >     > > -Jay
>> > > >     > >
>> > > >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > >     > > wrote:
>> > > >     > >
>> > > >     > > > Hi Ewen,
>> > > >     > > >
>> > > >     > > > I think the easiest way to show this is with code.
>> > > >     > > >
>> > > >     > > > As you can see we keep the existing behaviour for
>> > code/binaries
>> > > > calling
>> > > >     > > > the pre-existing constructors, whereby if the value is
>> null
>> > the
>> > > >     > tombstone
>> > > >     > > > is set to true.
>> > > >     > > >
>> > > >     > > > Regards
>> > > >     > > > Mike
>> > > >     > > >
>> > > >     > > >
>> > > >     > > >
>> > > >     > > >     /**
>> > > >     > > >      * Creates a record with a specified timestamp to be
>> sent
>> > > to
>> > > > a
>> > > >     > > > specified topic and partition
>> > > >     > > >      *
>> > > >     > > >      * @param topic The topic the record will be appended
>> to
>> > > >     > > >      * @param partition The partition to which the record
>> > > should
>> > > > be
>> > > >     > sent
>> > > >     > > >      * @param timestamp The timestamp of the record
>> > > >     > > >      * @param tombstone if the record should be treated
>> as a
>> > > > tombstone
>> > > >     > if
>> > > >     > > > the topic is compacted
>> > > >     > > >      * @param key The key that will be included in the
>> record
>> > > >     > > >      * @param value The record contents
>> > > >     > > >      */
>> > > >     > > >     public ProducerRecord(String topic, Integer partition,
>> > > > Boolean
>> > > >     > > > tombstone, Long timestamp, K key, V value) {
>> > > >     > > >         if (topic == null)
>> > > >     > > >             throw new IllegalArgumentException("Topic
>> cannot
>> > > be
>> > > >     > null.");
>> > > >     > > >         if (timestamp != null && timestamp < 0)
>> > > >     > > >             throw new IllegalArgumentException(
>> > > >     > > >                     String.format("Invalid timestamp: %d.
>> > > > Timestamp
>> > > >     > > should
>> > > >     > > > always be non-negative or null.", timestamp));
>> > > >     > > >         if (partition != null && partition < 0)
>> > > >     > > >             throw new IllegalArgumentException(
>> > > >     > > >                     String.format("Invalid partition: %d.
>> > > > Partition
>> > > >     > > number
>> > > >     > > > should always be non-negative or null.", partition));
>> > > >     > > >         if (!tombstone && value == null){
>> > > >     > > >             throw new IllegalArgumentException(
>> > > >     > > >                     String.format("Tombstone must be true
>> if
>> > > null
>> > > >     > > value"));
>> > > >     > > >         }
>> > > >     > > >         this.topic = topic;
>> > > >     > > >         this.partition = partition;
>> > > >     > > >         this.tombstone = tombstone;
>> > > >     > > >         this.key = key;
>> > > >     > > >         this.value = value;
>> > > >     > > >         this.timestamp = timestamp;
>> > > >     > > >     }
>> > > >     > > >
>> > > >     > > >     public ProducerRecord(String topic, Integer partition,
>> > Long
>> > > >     > > timestamp,
>> > > >     > > > K key, V value) {
>> > > >     > > >         this(topic, partition, value == null, timestamp,
>> key,
>> > > > value);
>> > > >     > > >     }
>> > > >     > > > ________________________________________
>> > > >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
>> > > >     > > > Sent: Sunday, December 11, 2016 5:45 AM
>> > > >     > > > To: dev@kafka.apache.org
>> > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > >     > > >
>> > > >     > > > It seemed like this was addressed in the migration
>> section,
>> > > > wasn't it?
>> > > >     > > The
>> > > >     > > > V2 vs V3 requests and the fact that the broker will
>> downgrade
>> > > the
>> > > >     > message
>> > > >     > > > format (losing zero copy) if you issues a V2 request to a
>> > > broker
>> > > > using
>> > > >     > V3
>> > > >     > > > format handles compatibility, does it not? The existing
>> > > > consumers won't
>> > > >     > > see
>> > > >     > > > the extra metadata in the value, but they will get a null
>> > > > instead and
>> > > >     > > treat
>> > > >     > > > it as a tombstone. Certainly there is a performance
>> impact,
>> > but
>> > > > it
>> > > >     > seemed
>> > > >     > > > compatible.
>> > > >     > > >
>> > > >     > > > I'm worried about this though:
>> > > >     > > >
>> > > >     > > >
>> > > >     > > >    - *NOTE *: With the new version of producer client
>> using
>> > > >     > > ProduceRequest
>> > > >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not
>> > set)
>> > > > and
>> > > >     > null
>> > > >     > > > value
>> > > >     > > >    should not be allowed as the older version of consumer
>> > using
>> > > >     > > > FetchRequest
>> > > >     > > >    V2 will think of this as a tombstone when its actually
>> > not.
>> > > >     > > >
>> > > >     > > >
>> > > >     > > > Unless I'm misunderstanding, this ends up breaking binary
>> > > > compatibility
>> > > >     > > for
>> > > >     > > > the Java API. It sounds like this suggests that passing a
>> > null
>> > > > value to
>> > > >     > > the
>> > > >     > > > existing ProducerRecord constructors would generate an
>> > > exception
>> > > > since
>> > > >     > > you
>> > > >     > > > didn't explicitly enable the tombstone (via whatever new
>> > > > constructor is
>> > > >     > > > provided). But that means you can't swap in a newer client
>> > jar
>> > > > without
>> > > >     > > > recompiling and get the same behavior if your app deletes
>> > keys
>> > > > using
>> > > >     > the
>> > > >     > > > current approach because your app will start throwing
>> > > > exceptions. Maybe
>> > > >     > > > this is a tradeoff we're ok with, but we've tried pretty
>> hard
>> > > to
>> > > > avoid
>> > > >     > > > breaking compatibility like this so far -- it makes
>> picking
>> > up
>> > > > bug
>> > > >     > fixes
>> > > >     > > in
>> > > >     > > > the clients harder for users of frameworks that have to
>> pin
>> > to
>> > > > earlier
>> > > >     > > > default versions for compatibility.
>> > > >     > > >
>> > > >     > > > But then later the KIP says:
>> > > >     > > >
>> > > >     > > >
>> > > >     > > >    - The old constructors for ProducerRecord without this
>> > > > parameter
>> > > >     > will
>> > > >     > > be
>> > > >     > > >    kept but updated so that their default behaviour if
>> > setting
>> > > > null
>> > > >     > value
>> > > >     > > > will
>> > > >     > > >    be the tombstone will be set to true to keep existing
>> > > > behaviour.
>> > > >     > > >
>> > > >     > > >
>> > > >     > > > So maybe I am misinterpreting something.
>> > > >     > > >
>> > > >     > > > And just a nit re: motivation section, item 6 would be
>> > clearer
>> > > > for a
>> > > >     > > union
>> > > >     > > > schema with null (or optional schemas in other formats),
>> e.g.
>> > > > [null,
>> > > >     > > > string], in which case losing the schema truly is losing
>> > > > information
>> > > >     > > > (whereas null is already the only valid value for a pure
>> null
>> > > > schema).
>> > > >     > > >
>> > > >     > > > -Ewen
>> > > >     > > >
>> > > >     > > >
>> > > >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com
>> > > >     > >
>> > > >     > > > wrote:
>> > > >     > > >
>> > > >     > > > > Hi Jay,
>> > > >     > > > >
>> > > >     > > > > Good point this detail is missing in the KIP write up.
>> Ive
>> > > > added this
>> > > >     > > > now.
>> > > >     > > > >
>> > > >     > > > > Essentially simply just upgrading the clients we do not
>> > > expect
>> > > > any
>> > > >     > > client
>> > > >     > > > > app code change needed.
>> > > >     > > > >
>> > > >     > > > > Cheers
>> > > >     > > > > Mike
>> > > >     > > > > ________________________________________
>> > > >     > > > > From: Jay Kreps <ja...@confluent.io>
>> > > >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
>> > > >     > > > > To: dev@kafka.apache.org
>> > > >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
>> Flag
>> > > >     > > > >
>> > > >     > > > > Michael,
>> > > >     > > > >
>> > > >     > > > > The compatibility section goes through the migration
>> path,
>> > > but
>> > > > isn't
>> > > >     > > the
>> > > >     > > > > bigger compatibility issue with existing apps? There are
>> > many
>> > > >     > (probably
>> > > >     > > > > thousands) of apps in production that use this feature
>> and
>> > > > send null
>> > > >     > to
>> > > >     > > > > mean delete. It seems like this would break
>> compatibility
>> > > with
>> > > > them,
>> > > >     > > and
>> > > >     > > > > they would have to be rewritten to right?
>> > > >     > > > >
>> > > >     > > > > -Jay
>> > > >     > > > >
>> > > >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
>> > > >     > Michael.Pearce@ig.com
>> > > >     > > >
>> > > >     > > > > wrote:
>> > > >     > > > >
>> > > >     > > > > > Hi Jun,
>> > > >     > > > > >
>> > > >     > > > > > 4) On v3 we honour the tombstone. As such we expect
>> it to
>> > > be
>> > > > set
>> > > >     > > > > correctly
>> > > >     > > > > > as per the KIP.
>> > > >     > > > > >
>> > > >     > > > > > 4.1) why would we want to produce an error when its
>> v3?
>> > > This
>> > > > is the
>> > > >     > > > exact
>> > > >     > > > > > purpose to support non-null tombstone's
>> > > >     > > > > > 4.2) again here im unclear on the question on the v3,
>> > > produce
>> > > >     > request
>> > > >     > > > we
>> > > >     > > > > > expect the tombstone flag to be set correctly.
>> > > >     > > > > >
>> > > >     > > > > > 4.4) compaction only occurs on compacted topics, the
>> bit
>> > > > makes no
>> > > >     > > > > > difference and not looked at on un-compacted
>> (time/size
>> > > based
>> > > >     > > > eviction).
>> > > >     > > > > >
>> > > >     > > > > >
>> > > >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io>
>> > wrote:
>> > > >     > > > > >
>> > > >     > > > > >     Hi, Michael,
>> > > >     > > > > >
>> > > >     > > > > >     4. Then, I think I misunderstood this point. Could
>> > you
>> > > > document
>> > > >     > > the
>> > > >     > > > > >     following points in the wiki?
>> > > >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
>> > > > non-null
>> > > >     > value,
>> > > >     > > > > does
>> > > >     > > > > > the
>> > > >     > > > > >     send() get an error or does the producer
>> > automatically
>> > > > set the
>> > > >     > > > value
>> > > >     > > > > to
>> > > >     > > > > >     null?
>> > > >     > > > > >     4.2 If producer V3 doesn't set tombstone, but
>> > provides
>> > > a
>> > > > null
>> > > >     > > > value,
>> > > >     > > > > > does
>> > > >     > > > > >     the send() get an error or does the producer
>> > > > automatically sets
>> > > >     > > the
>> > > >     > > > > >     tombstone?
>> > > >     > > > > >     4.3 Does the broker only expect messages that (a)
>> > have
>> > > no
>> > > >     > > tombstone
>> > > >     > > > > and
>> > > >     > > > > >     non-null value; (b) have tombstone and null value
>> and
>> > > > reject
>> > > >     > the
>> > > >     > > > > > messages
>> > > >     > > > > >     with an error code otherwise?
>> > > >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic
>> is
>> > > > compacted
>> > > >     > on
>> > > >     > > > > not?
>> > > >     > > > > >
>> > > >     > > > > >     Thanks,
>> > > >     > > > > >
>> > > >     > > > > >     Jun
>> > > >     > > > > >
>> > > >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
>> > > >     > > > > Michael.Pearce@ig.com
>> > > >     > > > > > >
>> > > >     > > > > >     wrote:
>> > > >     > > > > >
>> > > >     > > > > >     > Not at all.  This only acts on compacted topics
>> > just
>> > > > as what
>> > > >     > > > occurs
>> > > >     > > > > > today
>> > > >     > > > > >     >
>> > > >     > > > > >     > Sent using OWA for iPhone
>> > > >     > > > > >     > ________________________________________
>> > > >     > > > > >     > From: Jun Rao <ju...@confluent.io>
>> > > >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
>> > > >     > > > > >     > To: dev@kafka.apache.org
>> > > >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > Tombstone
>> > > > Flag
>> > > >     > > > > >     >
>> > > >     > > > > >     > Hi, Michael,
>> > > >     > > > > >     >
>> > > >     > > > > >     > 4. Hmm, does that mean the new client library
>> can
>> > > > never send
>> > > >     > a
>> > > >     > > > null
>> > > >     > > > > > message
>> > > >     > > > > >     > even to a regular topic? This seems like a
>> change
>> > of
>> > > > the
>> > > >     > > existing
>> > > >     > > > > > behavior.
>> > > >     > > > > >     >
>> > > >     > > > > >     > Thanks,
>> > > >     > > > > >     >
>> > > >     > > > > >     > Jun
>> > > >     > > > > >     >
>> > > >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
>> > > >     > > > > > Michael.Pearce@ig.com>
>> > > >     > > > > >     > wrote:
>> > > >     > > > > >     >
>> > > >     > > > > >     > > Hi Jun,
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Re 4) That's because we expect the tombstone
>> > value
>> > > > to be
>> > > >     > set
>> > > >     > > > > > correctly if
>> > > >     > > > > >     > > message bit is 2, as such if an older client
>> > sends
>> > > > in on
>> > > >     > old
>> > > >     > > > > > message the
>> > > >     > > > > >     > > message is upcast and the bit is set
>> correctly.
>> > And
>> > > > such no
>> > > >     > > > > longer
>> > > >     > > > > > need
>> > > >     > > > > >     > to
>> > > >     > > > > >     > > check the value. Mayuresh can you confirm my
>> > > > thinking and
>> > > >     > > > > > understanding
>> > > >     > > > > >     > of
>> > > >     > > > > >     > > what we've discussed?
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > The second point I understand what you're
>> getting
>> > > at
>> > > > now my
>> > > >     > > > > > apologies.
>> > > >     > > > > >     > Yes
>> > > >     > > > > >     > > this makes sense to save on touching the
>> message,
>> > > if
>> > > > we're
>> > > >     > > the
>> > > >     > > > > > only kip
>> > > >     > > > > >     > > going in, in this release.
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Cheers
>> > > >     > > > > >     > > Mike
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Sent using OWA for iPhone
>> > > >     > > > > >     > > ________________________________________
>> > > >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
>> > > >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>> > > >     > > > > >     > > To: dev@kafka.apache.org
>> > > >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > > > Tombstone Flag
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Hi, Michael,
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > 4. Is this updated in the wiki? The text "If
>> the
>> > > > magic byte
>> > > >     > > on
>> > > >     > > > > > message is
>> > > >     > > > > >     > > 2, the broker should use the tombstone bit for
>> > log
>> > > >     > > compaction."
>> > > >     > > > > > doesn't
>> > > >     > > > > >     > > seem to have changed.
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > 2. My point is that if we change the message
>> > format
>> > > > just
>> > > >     > for
>> > > >     > > > this
>> > > >     > > > > > KIP, we
>> > > >     > > > > >     > > should consider whether it's worth optimizing
>> the
>> > > > down
>> > > >     > > > conversion
>> > > >     > > > > > path
>> > > >     > > > > >     > > (i.e., decide whether a conversion is needed
>> by
>> > > just
>> > > >     > looking
>> > > >     > > at
>> > > >     > > > > the
>> > > >     > > > > >     > > tombstone bit in the wrapper message) since
>> > > > tombstone will
>> > > >     > be
>> > > >     > > > > used
>> > > >     > > > > >     > rarely.
>> > > >     > > > > >     > > However, if the message format change here is
>> > > > combined with
>> > > >     > > > other
>> > > >     > > > > > KIPs,
>> > > >     > > > > >     > > then this optimization likely won't be needed.
>> > The
>> > > > latter
>> > > >     > > > > probably
>> > > >     > > > > > makes
>> > > >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do
>> you
>> > > > think?
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Other than those, +1 from me,
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Thanks,
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > Jun
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael
>> Pearce <
>> > > >     > > > > > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > wrote:
>> > > >     > > > > >     > >
>> > > >     > > > > >     > > > Hi Jun
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > do we have your vote on this now?
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Any other concerns?
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Cheers
>> > > >     > > > > >     > > > Mike
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Sent using OWA for iPhone
>> > > >     > > > > >     > > > ________________________________________
>> > > >     > > > > >     > > > From: Michael Pearce <Michael.Pearce@ig.com
>> >
>> > > >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>> > > >     > > > > >     > > > To: dev@kafka.apache.org
>> > > >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > > > Tombstone
>> > > >     > Flag
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Hi Jun,
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Have updated. Thanks again for the feedback.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Agree yes we should align up when it gets to
>> > > that,
>> > > > I
>> > > >     > assume
>> > > >     > > > > > you've
>> > > >     > > > > >     > > flagged
>> > > >     > > > > >     > > > the same in the other KIP?
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > I think for now let's get this KIP approved,
>> > then
>> > > > we can
>> > > >     > > > start
>> > > >     > > > > > the work
>> > > >     > > > > >     > > to
>> > > >     > > > > >     > > > get an initial PR, then we can discuss how
>> to
>> > > > align the
>> > > >     > two
>> > > >     > > > if
>> > > >     > > > > > needed.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > Cheers,
>> > > >     > > > > >     > > > Mike
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
>> > > jun@confluent.io>
>> > > >     > wrote:
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     Hi, Michael,
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     For 2), this is fine. Could you update
>> the
>> > > KIP
>> > > > wiki
>> > > >     > to
>> > > >     > > > make
>> > > >     > > > > > this
>> > > >     > > > > >     > and
>> > > >     > > > > >     > > > other
>> > > >     > > > > >     > > >     points clearer? Other than that, the KIP
>> > > looks
>> > > > good
>> > > >     > to
>> > > >     > > > me.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     An orthogonal thing is that there are
>> other
>> > > > KIPs such
>> > > >     > > as
>> > > >     > > > > > KIP-98
>> > > >     > > > > >     > that
>> > > >     > > > > >     > > > also
>> > > >     > > > > >     > > >     intend to change the message format. If
>> > they
>> > > > all get
>> > > >     > > > > > approved, we
>> > > >     > > > > >     > > > should
>> > > >     > > > > >     > > >     think about whether it's better to just
>> > bump
>> > > > up the
>> > > >     > > magic
>> > > >     > > > > > byte once
>> > > >     > > > > >     > > to
>> > > >     > > > > >     > > >     incorporate multiple format changes
>> like we
>> > > > did in
>> > > >     > > > > > KIP-31/KIP-32.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     Thanks,
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     Jun
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
>> > > Pearce
>> > > > <
>> > > >     > > > > >     > > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > >     wrote:
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     > Hi Jao
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > Thanks for the response. Sorry for
>> slow
>> > > > reply, both
>> > > >     > > > with
>> > > >     > > > > > personal
>> > > >     > > > > >     > > > sickness
>> > > >     > > > > >     > > >     > and also battling some critical issues
>> > > > encountered
>> > > >     > > > since
>> > > >     > > > > >     > upgrading
>> > > >     > > > > >     > > to
>> > > >     > > > > >     > > >     > 0.10.1.0
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > 1) Thans for spotting, Document error
>> > where
>> > > > we
>> > > >     > > branched
>> > > >     > > > > > this KIP
>> > > >     > > > > >     > > from
>> > > >     > > > > >     > > >     > KIP-82, will get that removed.
>> > > >     > > > > >     > > >     > 2) Intent is to do this just at the
>> > record
>> > > > message
>> > > >     > > > level.
>> > > >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure
>> this
>> > is
>> > > >     > > corrected.
>> > > >     > > > > >     > > >     > 4) As per discussion thread we will
>> > support
>> > > >     > > tombstone +
>> > > >     > > > > > null
>> > > >     > > > > >     > value,
>> > > >     > > > > >     > > >     > tombstone + non null value, no
>> tombstone
>> > +
>> > > > null
>> > > >     > > value.
>> > > >     > > > > >     > > >     > 5) I believe this was in the
>> discussion
>> > > > thread,
>> > > >     > > > @Mayuresh
>> > > >     > > > > > is this
>> > > >     > > > > >     > > >     > something we've overlooked? I thought
>> we
>> > > > would down
>> > > >     > > > > > convert and
>> > > >     > > > > >     > > > remove the
>> > > >     > > > > >     > > >     > value so the old consumer had existing
>> > > > behavior, or
>> > > >     > > is
>> > > >     > > > > > there
>> > > >     > > > > >     > > > something we
>> > > >     > > > > >     > > >     > haven't thought about?
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > Cheers
>> > > >     > > > > >     > > >     > Mike
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
>> > > > jun@confluent.io>
>> > > >     > > > > wrote:
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     Hi, Michael,
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     Thanks for the KIP. A few comments
>> > > below.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     1. The message format change
>> contains
>> > > >     > > > "HeadersLength
>> > > >     > > > > >     > Headers".
>> > > >     > > > > >     > > > Is that
>> > > >     > > > > >     > > >     >     intended?
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     2. For compressed messageset, is
>> the
>> > > > tombstone
>> > > >     > > bit
>> > > >     > > > > > only set
>> > > >     > > > > >     > at
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > shallow
>> > > >     > > > > >     > > >     >     level? Do we always leave that
>> bit in
>> > > the
>> > > >     > wrapper
>> > > >     > > > > > message
>> > > >     > > > > >     > > unset?
>> > > >     > > > > >     > > > An
>> > > >     > > > > >     > > >     >     alternative is to set the
>> tombstone
>> > bit
>> > > > in the
>> > > >     > > > > wrapper
>> > > >     > > > > > if at
>> > > >     > > > > >     > > > least one
>> > > >     > > > > >     > > >     >     inner message has the tombstone
>> bit
>> > > set.
>> > > > This
>> > > >     > > makes
>> > > >     > > > > > things a
>> > > >     > > > > >     > > bit
>> > > >     > > > > >     > > > more
>> > > >     > > > > >     > > >     >     complicated, but we could
>> potentially
>> > > > exploit
>> > > >     > > that
>> > > >     > > > > for
>> > > >     > > > > >     > > > optimizing down
>> > > >     > > > > >     > > >     >     conversion. For example, we only
>> need
>> > > to
>> > > >     > convert
>> > > >     > > > > > messages
>> > > >     > > > > >     > with
>> > > >     > > > > >     > > > magic 2
>> > > >     > > > > >     > > >     > to
>> > > >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone
>> > bit
>> > > > is set
>> > > >     > > > > > (conversion is
>> > > >     > > > > >     > > > always
>> > > >     > > > > >     > > >     > needed
>> > > >     > > > > >     > > >     >     from magic 2 to magic 0). Not
>> sure if
>> > > the
>> > > >     > > > > optimization
>> > > >     > > > > > is
>> > > >     > > > > >     > worth
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     >     complexity though.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     3. The referencing of the new
>> version
>> > > of
>> > > >     > > > > >     > > > ProducerRequest/FetchRequest
>> > > >     > > > > >     > > >     > is
>> > > >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
>> > > > convention
>> > > >     > > > starts
>> > > >     > > > > at
>> > > >     > > > > >     > version
>> > > >     > > > > >     > > > at 0, I
>> > > >     > > > > >     > > >     >     think the new version would be 3.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     4. "If the magic byte on message
>> is
>> > 2,
>> > > > the
>> > > >     > broker
>> > > >     > > > > > should use
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > >     > tombstone
>> > > >     > > > > >     > > >     >     bit for log compaction." What
>> about
>> > > null
>> > > > value?
>> > > >     > > My
>> > > >     > > > > >     > > understanding
>> > > >     > > > > >     > > > is
>> > > >     > > > > >     > > >     > that
>> > > >     > > > > >     > > >     >     null value will be treated the
>> same
>> > as
>> > > > setting
>> > > >     > > the
>> > > >     > > > > > tombstone
>> > > >     > > > > >     > > bit.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     5. For the migration path, it
>> would
>> > be
>> > > > useful
>> > > >     > to
>> > > >     > > > > > describe the
>> > > >     > > > > >     > > > down
>> > > >     > > > > >     > > >     >     conversion path to consumers
>> (i.e.,
>> > > > brokers on
>> > > >     > > > > message
>> > > >     > > > > > format
>> > > >     > > > > >     > > > 0.10.2
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     consumers on older version).
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     Thanks,
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     Jun
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
>> > > Michael
>> > > >     > Pearce <
>> > > >     > > > > >     > > > Michael.Pearce@ig.com
>> > > >     > > > > >     > > >     > >
>> > > >     > > > > >     > > >     >     wrote:
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >     > Hi All,
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > We have been discussing in the
>> > below
>> > > > thread
>> > > >     > and
>> > > >     > > > > final
>> > > >     > > > > >     > changes
>> > > >     > > > > >     > > > have
>> > > >     > > > > >     > > >     > been
>> > > >     > > > > >     > > >     >     > made to the KIP wiki based on
>> these
>> > > >     > > discussions.
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > We would now like to put to the
>> > vote
>> > > > the
>> > > >     > > > following
>> > > >     > > > > > KIP:
>> > > >     > > > > >     > > >     >     > https://cwiki.apache.org/
>> > > >     > > > > > confluence/display/KAFKA/KIP-
>> > > >     > > > > >     > 87+-+
>> > > >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > This kip is for having a
>> distinct
>> > > > compaction
>> > > >     > > > > > attribute
>> > > >     > > > > >     > > > "tombstone"
>> > > >     > > > > >     > > >     > flag
>> > > >     > > > > >     > > >     >     > instead of relying on null
>> value,
>> > > > allowing
>> > > >     > > > non-null
>> > > >     > > > > > value
>> > > >     > > > > >     > > > delete
>> > > >     > > > > >     > > >     > messages.
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > Many thanks,
>> > > >     > > > > >     > > >     >     > Michael
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
>> > > Pearce"
>> > > > <
>> > > >     > > > > >     > > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > >     > wrote:
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     Hi Mayuresh,
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     LGTM. Ive just made one
>> small
>> > > > adjustment
>> > > >     > > > > > updating the
>> > > >     > > > > >     > > wire
>> > > >     > > > > >     > > >     > protocol to
>> > > >     > > > > >     > > >     >     > show the magic byte bump.
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     Do we think we're good to
>> put
>> > to
>> > > a
>> > > > vote?
>> > > >     > Is
>> > > >     > > > > > there any
>> > > >     > > > > >     > > > other bits
>> > > >     > > > > >     > > >     >     > needing discussion?
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     Cheers
>> > > >     > > > > >     > > >     >     >     Mike
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     On 21/11/2016, 18:26,
>> "Mayuresh
>> > > > Gharat" <
>> > > >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
>> > > >     > > > > >     > > >     >     > wrote:
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         Hi Michael,
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         I have updated the
>> > migration
>> > > > section
>> > > >     > of
>> > > >     > > > the
>> > > >     > > > > > KIP.
>> > > >     > > > > >     > Can
>> > > >     > > > > >     > > > you
>> > > >     > > > > >     > > >     > please
>> > > >     > > > > >     > > >     >     > take a look?
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         Thanks,
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         Mayuresh
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at
>> > 9:07
>> > > > AM,
>> > > >     > > Mayuresh
>> > > >     > > > > > Gharat <
>> > > >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >     >     >         > wrote:
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         > Hi Michael,
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > That whilst sending
>> > > > tombstone and
>> > > >     > non
>> > > >     > > > > null
>> > > >     > > > > > value,
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > >     > consumer
>> > > >     > > > > >     > > >     >     > can expect
>> > > >     > > > > >     > > >     >     >         > only to receive the
>> > > non-null
>> > > >     > message
>> > > >     > > > only
>> > > >     > > > > > in step
>> > > >     > > > > >     > > > (3) is
>> > > >     > > > > >     > > >     > this
>> > > >     > > > > >     > > >     >     > correct?
>> > > >     > > > > >     > > >     >     >         > ---> I do agree with
>> you
>> > > > here.
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > Becket, Ismael : can
>> you
>> > > guys
>> > > >     > review
>> > > >     > > > the
>> > > >     > > > > >     > migration
>> > > >     > > > > >     > > > plan
>> > > >     > > > > >     > > >     > listed
>> > > >     > > > > >     > > >     >     > above using
>> > > >     > > > > >     > > >     >     >         > magic byte?
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > Thanks,
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > Mayuresh
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016
>> at
>> > > 8:58
>> > > > AM,
>> > > >     > > > Michael
>> > > >     > > > > > Pearce <
>> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > >     >     >         > wrote:
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         >> Many thanks for this
>> > > > Mayuresh. I
>> > > >     > > don't
>> > > >     > > > > > have any
>> > > >     > > > > >     > > >     > objections.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> I assume we should
>> > state:
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> That whilst sending
>> > > > tombstone and
>> > > >     > > non
>> > > >     > > > > null
>> > > >     > > > > >     > value,
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > consumer
>> > > >     > > > > >     > > >     >     > can expect
>> > > >     > > > > >     > > >     >     >         >> only to receive the
>> > > non-null
>> > > >     > message
>> > > >     > > > > only
>> > > >     > > > > > in
>> > > >     > > > > >     > step
>> > > >     > > > > >     > > > (3) is
>> > > >     > > > > >     > > >     > this
>> > > >     > > > > >     > > >     >     > correct?
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Cheers
>> > > >     > > > > >     > > >     >     >         >> Mike
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Sent using OWA for
>> > iPhone
>> > > >     > > > > >     > > >     >     >         >>
>> > > > ______________________________
>> > > >     > > > > __________
>> > > >     > > > > >     > > >     >     >         >> From: Mayuresh
>> Gharat <
>> > > >     > > > > >     > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >     >     >         >> Sent: Thursday,
>> November
>> > > > 17, 2016
>> > > >     > > > > 5:18:41
>> > > >     > > > > > PM
>> > > >     > > > > >     > > >     >     >         >> To:
>> > dev@kafka.apache.org
>> > > >     > > > > >     > > >     >     >         >> Subject: Re:
>> [DISCUSS]
>> > > > KIP-87 -
>> > > >     > Add
>> > > >     > > > > > Compaction
>> > > >     > > > > >     > > > Tombstone
>> > > >     > > > > >     > > >     > Flag
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Hi Ismael,
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Thanks for the
>> > > explanation.
>> > > >     > > > > >     > > >     >     >         >> Specially I like this
>> > part
>> > > > where
>> > > >     > in
>> > > >     > > > you
>> > > >     > > > > >     > mentioned
>> > > >     > > > > >     > > > we can
>> > > >     > > > > >     > > >     > get
>> > > >     > > > > >     > > >     >     > rid of the
>> > > >     > > > > >     > > >     >     >         >> older null value
>> support
>> > > > for log
>> > > >     > > > > > compaction
>> > > >     > > > > >     > later
>> > > >     > > > > >     > > > on,
>> > > >     > > > > >     > > >     > here :
>> > > >     > > > > >     > > >     >     >         >> We can't change
>> > semantics
>> > > > of the
>> > > >     > > > message
>> > > >     > > > > > format
>> > > >     > > > > >     > > > without
>> > > >     > > > > >     > > >     > having
>> > > >     > > > > >     > > >     >     > a long
>> > > >     > > > > >     > > >     >     >         >> transition period.
>> And
>> > we
>> > > > can't
>> > > >     > rely
>> > > >     > > > > >     > > >     >     >         >> on people reading
>> > > > documentation or
>> > > >     > > > > acting
>> > > >     > > > > > on a
>> > > >     > > > > >     > > > warning for
>> > > >     > > > > >     > > >     >     > something so
>> > > >     > > > > >     > > >     >     >         >> fundamental. As
>> such, my
>> > > > take is
>> > > >     > > that
>> > > >     > > > we
>> > > >     > > > > > need to
>> > > >     > > > > >     > > > bump the
>> > > >     > > > > >     > > >     > magic
>> > > >     > > > > >     > > >     >     > byte. The
>> > > >     > > > > >     > > >     >     >         >> good news is
>> > > >     > > > > >     > > >     >     >         >> that we don't have to
>> > > > support all
>> > > >     > > > > versions
>> > > >     > > > > >     > > forever.
>> > > >     > > > > >     > > > We
>> > > >     > > > > >     > > >     > have
>> > > >     > > > > >     > > >     >     > said that we
>> > > >     > > > > >     > > >     >     >         >> will support direct
>> > > > upgrades for 2
>> > > >     > > > > years.
>> > > >     > > > > > That
>> > > >     > > > > >     > > > means that
>> > > >     > > > > >     > > >     >     > message format
>> > > >     > > > > >     > > >     >     >         >> version n could, in
>> > > theory,
>> > > > be
>> > > >     > > > removed 2
>> > > >     > > > > > years
>> > > >     > > > > >     > > > after the
>> > > >     > > > > >     > > >     > it's
>> > > >     > > > > >     > > >     >     > introduced.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Just a heads up, I
>> would
>> > > > like to
>> > > >     > > > mention
>> > > >     > > > > > that
>> > > >     > > > > >     > even
>> > > >     > > > > >     > > > without
>> > > >     > > > > >     > > >     >     > bumping magic
>> > > >     > > > > >     > > >     >     >         >> byte, we will *NOT*
>> > loose
>> > > > zero
>> > > >     > copy
>> > > >     > > as
>> > > >     > > > > in
>> > > >     > > > > > the
>> > > >     > > > > >     > > > client(x+1)
>> > > >     > > > > >     > > >     > in my
>> > > >     > > > > >     > > >     >     >         >> explanation
>> > > >     > > > > >     > > >     >     >         >> above will convert
>> > > > internally a
>> > > >     > null
>> > > >     > > > > > value to
>> > > >     > > > > >     > > have a
>> > > >     > > > > >     > > >     > tombstone
>> > > >     > > > > >     > > >     >     > bit set and
>> > > >     > > > > >     > > >     >     >         >> a tombstone bit set
>> to
>> > > have
>> > > > a null
>> > > >     > > > value
>> > > >     > > > > >     > > > automatically
>> > > >     > > > > >     > > >     >     > internally and by
>> > > >     > > > > >     > > >     >     >         >> the time we move to
>> > > version
>> > > > (x+2),
>> > > >     > > the
>> > > >     > > > > > clients
>> > > >     > > > > >     > > > would have
>> > > >     > > > > >     > > >     >     > upgraded.
>> > > >     > > > > >     > > >     >     >         >> Obviously if we
>> support
>> > a
>> > > > request
>> > > >     > > from
>> > > >     > > > > >     > > consumer(x),
>> > > >     > > > > >     > > > we
>> > > >     > > > > >     > > >     > will
>> > > >     > > > > >     > > >     >     > loose zero
>> > > >     > > > > >     > > >     >     >         >> copy
>> > > >     > > > > >     > > >     >     >         >> but that is the same
>> > case
>> > > > with
>> > > >     > magic
>> > > >     > > > > byte.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> But if magic byte
>> bump
>> > > > makes life
>> > > >     > > > easier
>> > > >     > > > > > for
>> > > >     > > > > >     > > > transition
>> > > >     > > > > >     > > >     > for the
>> > > >     > > > > >     > > >     >     > above
>> > > >     > > > > >     > > >     >     >         >> reasons that you
>> > > explained,
>> > > > I am
>> > > >     > OK
>> > > >     > > > with
>> > > >     > > > > > it
>> > > >     > > > > >     > since
>> > > >     > > > > >     > > > we are
>> > > >     > > > > >     > > >     > going
>> > > >     > > > > >     > > >     >     > to meet the
>> > > >     > > > > >     > > >     >     >         >> end goal down the
>> road
>> > :)
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> On a side note can we
>> > > > update the
>> > > >     > doc
>> > > >     > > > > here
>> > > >     > > > > > on
>> > > >     > > > > >     > magic
>> > > >     > > > > >     > > > byte
>> > > >     > > > > >     > > >     > to say
>> > > >     > > > > >     > > >     >     > that "*it
>> > > >     > > > > >     > > >     >     >         >> should be bumped
>> > whenever
>> > > > the
>> > > >     > > message
>> > > >     > > > > > format is
>> > > >     > > > > >     > > > changed
>> > > >     > > > > >     > > >     > or the
>> > > >     > > > > >     > > >     >     >         >> interpretation of
>> > message
>> > > > format
>> > > >     > > > (usage
>> > > >     > > > > > of the
>> > > >     > > > > >     > > > reserved
>> > > >     > > > > >     > > >     > bits as
>> > > >     > > > > >     > > >     >     > well) is
>> > > >     > > > > >     > > >     >     >         >> changed*".
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Hi Michael,
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Here is the update
>> plan
>> > > > that we
>> > > >     > > > > discussed
>> > > >     > > > > >     > offline
>> > > >     > > > > >     > > >     > yesterday :
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Currently the
>> magic-byte
>> > > > which
>> > > >     > > > > > corresponds to
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > >     >     > "message.format.version"
>> > > >     > > > > >     > > >     >     >         >> is set to 1.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> 1) On broker it will
>> be
>> > > set
>> > > > to 1
>> > > >     > > > > > initially.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> 2) When a producer
>> > client
>> > > > sends a
>> > > >     > > > > message
>> > > >     > > > > > with
>> > > >     > > > > >     > > > magic-byte
>> > > >     > > > > >     > > >     > = 2,
>> > > >     > > > > >     > > >     >     > since the
>> > > >     > > > > >     > > >     >     >         >> broker is on
>> magic-byte
>> > =
>> > > > 1, we
>> > > >     > will
>> > > >     > > > > down
>> > > >     > > > > >     > convert
>> > > >     > > > > >     > > > it,
>> > > >     > > > > >     > > >     > which
>> > > >     > > > > >     > > >     >     > means if the
>> > > >     > > > > >     > > >     >     >         >> tombstone bit is set,
>> > the
>> > > > value
>> > > >     > will
>> > > >     > > > be
>> > > >     > > > > > set to
>> > > >     > > > > >     > > > null. A
>> > > >     > > > > >     > > >     > consumer
>> > > >     > > > > >     > > >     >     >         >> understanding
>> > magic-byte =
>> > > > 1, will
>> > > >     > > > still
>> > > >     > > > > > work
>> > > >     > > > > >     > with
>> > > >     > > > > >     > > > this. A
>> > > >     > > > > >     > > >     >     > consumer
>> > > >     > > > > >     > > >     >     >         >> working
>> > > >     > > > > >     > > >     >     >         >> with magic-byte =2
>> will
>> > > > also be
>> > > >     > able
>> > > >     > > > to
>> > > >     > > > > >     > understand
>> > > >     > > > > >     > > > this,
>> > > >     > > > > >     > > >     > since
>> > > >     > > > > >     > > >     >     > it
>> > > >     > > > > >     > > >     >     >         >> understands the
>> > tombstone.
>> > > >     > > > > >     > > >     >     >         >> Now there is still
>> the
>> > > > question of
>> > > >     > > > > > supporting a
>> > > >     > > > > >     > > >     > non-tombstone
>> > > >     > > > > >     > > >     >     > and null
>> > > >     > > > > >     > > >     >     >         >> value from producer
>> > client
>> > > > with
>> > > >     > > > > > magic-byte = 2.*
>> > > >     > > > > >     > > (I
>> > > >     > > > > >     > > > am
>> > > >     > > > > >     > > >     > not sure
>> > > >     > > > > >     > > >     >     > if we
>> > > >     > > > > >     > > >     >     >         >> should support this.
>> > > > Ismael/Becket
>> > > >     > > can
>> > > >     > > > > > comment
>> > > >     > > > > >     > > > here)*
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> 3) When almost all
>> the
>> > > > clients
>> > > >     > have
>> > > >     > > > > > upgraded,
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > >     >     > message.format.version
>> > > >     > > > > >     > > >     >     >         >> on
>> > > >     > > > > >     > > >     >     >         >> the broker can be
>> > changed
>> > > > to 2,
>> > > >     > > where
>> > > >     > > > in
>> > > >     > > > > > the
>> > > >     > > > > >     > down
>> > > >     > > > > >     > > >     > conversion in
>> > > >     > > > > >     > > >     >     > the above
>> > > >     > > > > >     > > >     >     >         >> step will not
>> happen. If
>> > > at
>> > > > this
>> > > >     > > point
>> > > >     > > > > we
>> > > >     > > > > > get a
>> > > >     > > > > >     > > > consumer
>> > > >     > > > > >     > > >     >     > request from a
>> > > >     > > > > >     > > >     >     >         >> older consumer, we
>> might
>> > > > have to
>> > > >     > > down
>> > > >     > > > > > convert
>> > > >     > > > > >     > > where
>> > > >     > > > > >     > > > in we
>> > > >     > > > > >     > > >     > loose
>> > > >     > > > > >     > > >     >     > zero copy,
>> > > >     > > > > >     > > >     >     >         >> but these cases
>> should
>> > be
>> > > > rare.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Becket can you review
>> > this
>> > > > plan
>> > > >     > and
>> > > >     > > > add
>> > > >     > > > > > more
>> > > >     > > > > >     > > > details if I
>> > > >     > > > > >     > > >     > have
>> > > >     > > > > >     > > >     >     >         >> missed/wronged
>> > something,
>> > > > before
>> > > >     > we
>> > > >     > > > put
>> > > >     > > > > > it on
>> > > >     > > > > >     > KIP.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Thanks,
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> Mayuresh
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016
>> at
>> > > > 11:07 PM,
>> > > >     > > > > Michael
>> > > >     > > > > >     > Pearce <
>> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
>> > > >     > > > > >     > > >     >     >         >> wrote:
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> > Thanks guys, for
>> > > > discussing this
>> > > >     > > > > > offline and
>> > > >     > > > > >     > > > getting
>> > > >     > > > > >     > > >     > some
>> > > >     > > > > >     > > >     >     > consensus.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > So its clear for
>> > myself
>> > > > and
>> > > >     > others
>> > > >     > > > > what
>> > > >     > > > > > is
>> > > >     > > > > >     > > > proposed now
>> > > >     > > > > >     > > >     > (i
>> > > >     > > > > >     > > >     >     > think i
>> > > >     > > > > >     > > >     >     >         >> > understand, but
>> want
>> > to
>> > > > make
>> > > >     > sure)
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Could i ask either
>> > > > directly
>> > > >     > update
>> > > >     > > > the
>> > > >     > > > > > kip to
>> > > >     > > > > >     > > > detail the
>> > > >     > > > > >     > > >     >     > migration
>> > > >     > > > > >     > > >     >     >         >> > strategy, or
>> > (re-)state
>> > > > your
>> > > >     > > offline
>> > > >     > > > > > discussed
>> > > >     > > > > >     > > and
>> > > >     > > > > >     > > >     > agreed
>> > > >     > > > > >     > > >     >     > migration
>> > > >     > > > > >     > > >     >     >         >> > strategy based on a
>> > > magic
>> > > > byte
>> > > >     > is
>> > > >     > > in
>> > > >     > > > > > this
>> > > >     > > > > >     > > thread.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > The main original
>> > driver
>> > > > for the
>> > > >     > > KIP
>> > > >     > > > > > was to
>> > > >     > > > > >     > > > support
>> > > >     > > > > >     > > >     >     > compaction where
>> > > >     > > > > >     > > >     >     >         >> value
>> > > >     > > > > >     > > >     >     >         >> > isn't null, based
>> off
>> > > the
>> > > >     > > > discussions
>> > > >     > > > > on
>> > > >     > > > > >     > KIP-82
>> > > >     > > > > >     > > > thread.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > We should be able
>> to
>> > > > support
>> > > >     > > > > > non-tombstone +
>> > > >     > > > > >     > > null
>> > > >     > > > > >     > > > value
>> > > >     > > > > >     > > >     > by the
>> > > >     > > > > >     > > >     >     >         >> completion
>> > > >     > > > > >     > > >     >     >         >> > of the KIP, as we
>> > noted
>> > > > when
>> > > >     > > > > discussing
>> > > >     > > > > > this
>> > > >     > > > > >     > > kip,
>> > > >     > > > > >     > > > having
>> > > >     > > > > >     > > >     >     > logic based on
>> > > >     > > > > >     > > >     >     >         >> a
>> > > >     > > > > >     > > >     >     >         >> > null value isn't
>> very
>> > > > clean and
>> > > >     > > also
>> > > >     > > > > > separates
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > >     > concerns.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > As discussed
>> already
>> > > > though we
>> > > >     > can
>> > > >     > > > > > split this
>> > > >     > > > > >     > > into
>> > > >     > > > > >     > > >     > KIP-87a
>> > > >     > > > > >     > > >     >     > and KIP-87b
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Where we look to
>> > deliver
>> > > > KIP-87a
>> > > >     > > on
>> > > >     > > > a
>> > > >     > > > > >     > compacted
>> > > >     > > > > >     > > > topic
>> > > >     > > > > >     > > >     > (to
>> > > >     > > > > >     > > >     >     > address the
>> > > >     > > > > >     > > >     >     >         >> > immediate issues)
>> > > >     > > > > >     > > >     >     >         >> > * tombstone + null
>> > value
>> > > >     > > > > >     > > >     >     >         >> > * tombstone +
>> non-null
>> > > > value
>> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
>> > > > non-null value
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Then we can discuss
>> > once
>> > > > KIP-87a
>> > > >     > > is
>> > > >     > > > > > completed
>> > > >     > > > > >     > > > options
>> > > >     > > > > >     > > >     > later
>> > > >     > > > > >     > > >     >     > and how we
>> > > >     > > > > >     > > >     >     >         >> > support the second
>> > part
>> > > > KIP-87b
>> > > >     > to
>> > > >     > > > > > deliver:
>> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
>> null
>> > > > value
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Cheers
>> > > >     > > > > >     > > >     >     >         >> > Mike
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > > ______________________________
>> > > >     > > > > > __________
>> > > >     > > > > >     > > >     >     >         >> > From: Becket Qin <
>> > > >     > > > > becket.qin@gmail.com>
>> > > >     > > > > >     > > >     >     >         >> > Sent: Thursday,
>> > November
>> > > > 17,
>> > > >     > 2016
>> > > >     > > > 1:43
>> > > >     > > > > > AM
>> > > >     > > > > >     > > >     >     >         >> > To:
>> > > dev@kafka.apache.org
>> > > >     > > > > >     > > >     >     >         >> > Subject: Re:
>> [DISCUSS]
>> > > > KIP-87 -
>> > > >     > > Add
>> > > >     > > > > > Compaction
>> > > >     > > > > >     > > >     > Tombstone Flag
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and
>> I
>> > had
>> > > > an
>> > > >     > > offline
>> > > >     > > > > >     > discussion,
>> > > >     > > > > >     > > > and
>> > > >     > > > > >     > > >     > following
>> > > >     > > > > >     > > >     >     > is a brief
>> > > >     > > > > >     > > >     >     >         >> > summary.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > 1. We agreed that
>> not
>> > > > bumping up
>> > > >     > > > magic
>> > > >     > > > > > value
>> > > >     > > > > >     > may
>> > > >     > > > > >     > > > result
>> > > >     > > > > >     > > >     > in
>> > > >     > > > > >     > > >     >     > losing zero
>> > > >     > > > > >     > > >     >     >         >> copy
>> > > >     > > > > >     > > >     >     >         >> > during migration.
>> > > >     > > > > >     > > >     >     >         >> > 2. Given that
>> bumping
>> > up
>> > > > magic
>> > > >     > > value
>> > > >     > > > > is
>> > > >     > > > > > almost
>> > > >     > > > > >     > > > free and
>> > > >     > > > > >     > > >     > has
>> > > >     > > > > >     > > >     >     > benefit of
>> > > >     > > > > >     > > >     >     >         >> > avoiding potential
>> > > > performance
>> > > >     > > > issue.
>> > > >     > > > > > It is
>> > > >     > > > > >     > > > probably
>> > > >     > > > > >     > > >     > worth
>> > > >     > > > > >     > > >     >     > doing.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > One issue we still
>> > need
>> > > > to think
>> > > >     > > > about
>> > > >     > > > > > is
>> > > >     > > > > >     > > whether
>> > > >     > > > > >     > > > we
>> > > >     > > > > >     > > >     > want to
>> > > >     > > > > >     > > >     >     > support a
>> > > >     > > > > >     > > >     >     >         >> > non-tombstone
>> message
>> > > > with null
>> > > >     > > > value.
>> > > >     > > > > >     > > >     >     >         >> > Currently it is not
>> > > > supported by
>> > > >     > > > > Kafka.
>> > > >     > > > > > If we
>> > > >     > > > > >     > > > allow a
>> > > >     > > > > >     > > >     >     > non-tombstone null
>> > > >     > > > > >     > > >     >     >         >> > value message to
>> exist
>> > > > after
>> > > >     > > KIP-87.
>> > > >     > > > > The
>> > > >     > > > > >     > problem
>> > > >     > > > > >     > > > is
>> > > >     > > > > >     > > >     > that such
>> > > >     > > > > >     > > >     >     > message
>> > > >     > > > > >     > > >     >     >         >> will
>> > > >     > > > > >     > > >     >     >         >> > not be supported by
>> > the
>> > > >     > consumers
>> > > >     > > > > prior
>> > > >     > > > > > to
>> > > >     > > > > >     > > KIP-87.
>> > > >     > > > > >     > > >     > Because a
>> > > >     > > > > >     > > >     >     > null value
>> > > >     > > > > >     > > >     >     >         >> > will always be
>> > > > interpreted to a
>> > > >     > > > > > tombstone.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > One option is that
>> we
>> > > > keep the
>> > > >     > > > current
>> > > >     > > > > > way,
>> > > >     > > > > >     > i.e.
>> > > >     > > > > >     > > > do not
>> > > >     > > > > >     > > >     >     > support such
>> > > >     > > > > >     > > >     >     >         >> > message. It would
>> be
>> > > good
>> > > > to
>> > > >     > know
>> > > >     > > if
>> > > >     > > > > > there is
>> > > >     > > > > >     > a
>> > > >     > > > > >     > > >     > concrete use
>> > > >     > > > > >     > > >     >     > case for
>> > > >     > > > > >     > > >     >     >         >> such
>> > > >     > > > > >     > > >     >     >         >> > message. If there
>> is
>> > > not,
>> > > > we can
>> > > >     > > > > > probably just
>> > > >     > > > > >     > > not
>> > > >     > > > > >     > > >     > support it.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > Thanks,
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > JIangjie (Becket)
>> Qin
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > On Wed, Nov 16,
>> 2016
>> > at
>> > > > 1:28 PM,
>> > > >     > > > > > Mayuresh
>> > > >     > > > > >     > > Gharat <
>> > > >     > > > > >     > > >     >     >         >> >
>> > > > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >     >     >         >> > > wrote:
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >> > > Hi Ismael,
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > This is
>> something I
>> > > can
>> > > > think
>> > > >     > of
>> > > >     > > > for
>> > > >     > > > > >     > migration
>> > > >     > > > > >     > > > plan:
>> > > >     > > > > >     > > >     >     >         >> > > So the migration
>> > plan
>> > > > can look
>> > > >     > > > > > something
>> > > >     > > > > >     > like
>> > > >     > > > > >     > > > this,
>> > > >     > > > > >     > > >     > with up
>> > > >     > > > > >     > > >     >     >         >> conversion :
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > 1) Currently lets
>> > say
>> > > > we have
>> > > >     > > > Broker
>> > > >     > > > > > at
>> > > >     > > > > >     > > version
>> > > >     > > > > >     > > > x.
>> > > >     > > > > >     > > >     >     >         >> > > 2) Currently we
>> have
>> > > > clients
>> > > >     > at
>> > > >     > > > > > version x.
>> > > >     > > > > >     > > >     >     >         >> > > 3) a) We move the
>> > > > version to
>> > > >     > > > > > Broker(x+1) :
>> > > >     > > > > >     > > > supports
>> > > >     > > > > >     > > >     > both
>> > > >     > > > > >     > > >     >     > tombstone and
>> > > >     > > > > >     > > >     >     >         >> > null
>> > > >     > > > > >     > > >     >     >         >> > > for log
>> compaction.
>> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
>> > the
>> > > > client
>> > > >     > to
>> > > >     > > > > > version
>> > > >     > > > > >     > > > client(x+1) :
>> > > >     > > > > >     > > >     > if in
>> > > >     > > > > >     > > >     >     > the
>> > > >     > > > > >     > > >     >     >         >> producer
>> > > >     > > > > >     > > >     >     >         >> > > client(x+1) the
>> > value
>> > > > is set
>> > > >     > to
>> > > >     > > > > null,
>> > > >     > > > > > we
>> > > >     > > > > >     > will
>> > > >     > > > > >     > > >     > automatically
>> > > >     > > > > >     > > >     >     > set the
>> > > >     > > > > >     > > >     >     >         >> > > Tombstone bit
>> > > > internally. If
>> > > >     > the
>> > > >     > > > > > producer
>> > > >     > > > > >     > > > client(x+1)
>> > > >     > > > > >     > > >     > sets
>> > > >     > > > > >     > > >     >     > the
>> > > >     > > > > >     > > >     >     >         >> tombstone
>> > > >     > > > > >     > > >     >     >         >> > > itself, well and
>> > good.
>> > > > For
>> > > >     > > > producer
>> > > >     > > > > >     > client(x),
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > broker
>> > > >     > > > > >     > > >     >     > will up
>> > > >     > > > > >     > > >     >     >         >> convert
>> > > >     > > > > >     > > >     >     >         >> > > to have the
>> > tombstone
>> > > > bit.
>> > > >     > > > > > Broker(x+1) is
>> > > >     > > > > >     > > > supporting
>> > > >     > > > > >     > > >     > both.
>> > > >     > > > > >     > > >     >     > Consumer
>> > > >     > > > > >     > > >     >     >         >> > > client(x+1) will
>> be
>> > > > aware of
>> > > >     > > this
>> > > >     > > > > and
>> > > >     > > > > > should
>> > > >     > > > > >     > > be
>> > > >     > > > > >     > > > able
>> > > >     > > > > >     > > >     > to
>> > > >     > > > > >     > > >     >     > handle this.
>> > > >     > > > > >     > > >     >     >         >> For
>> > > >     > > > > >     > > >     >     >         >> > > consumer
>> client(x)
>> > we
>> > > > will
>> > > >     > down
>> > > >     > > > > > convert the
>> > > >     > > > > >     > > > message
>> > > >     > > > > >     > > >     > on the
>> > > >     > > > > >     > > >     >     > broker
>> > > >     > > > > >     > > >     >     >         >> side.
>> > > >     > > > > >     > > >     >     >         >> > >     c) At this
>> point
>> > > we
>> > > > will
>> > > >     > > have
>> > > >     > > > to
>> > > >     > > > > >     > specify a
>> > > >     > > > > >     > > >     > warning or
>> > > >     > > > > >     > > >     >     > clearly
>> > > >     > > > > >     > > >     >     >         >> specify
>> > > >     > > > > >     > > >     >     >         >> > > in docs that this
>> > > > behavior is
>> > > >     > > > about
>> > > >     > > > > > to be
>> > > >     > > > > >     > > > changed for
>> > > >     > > > > >     > > >     > log
>> > > >     > > > > >     > > >     >     > compaction.
>> > > >     > > > > >     > > >     >     >         >> > > 4) a) In next
>> > release
>> > > > of the
>> > > >     > > > > > Broker(x+2), we
>> > > >     > > > > >     > > > say that
>> > > >     > > > > >     > > >     > only
>> > > >     > > > > >     > > >     >     > Tombstone
>> > > >     > > > > >     > > >     >     >         >> is
>> > > >     > > > > >     > > >     >     >         >> > > used for log
>> > > compaction
>> > > > on the
>> > > >     > > > > Broker
>> > > >     > > > > > side.
>> > > >     > > > > >     > > >     > Clients(x+1)
>> > > >     > > > > >     > > >     >     > still is
>> > > >     > > > > >     > > >     >     >         >> > > supported.
>> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
>> > the
>> > > > client
>> > > >     > to
>> > > >     > > > > > version
>> > > >     > > > > >     > > > client(x+2) :
>> > > >     > > > > >     > > >     > if
>> > > >     > > > > >     > > >     >     > value is set
>> > > >     > > > > >     > > >     >     >         >> to
>> > > >     > > > > >     > > >     >     >         >> > > null, tombstone
>> will
>> > > > not be
>> > > >     > set
>> > > >     > > > > >     > automatically.
>> > > >     > > > > >     > > > The
>> > > >     > > > > >     > > >     > client
>> > > >     > > > > >     > > >     >     > will have to
>> > > >     > > > > >     > > >     >     >         >> > call
>> > > >     > > > > >     > > >     >     >         >> > > setTombstone() to
>> > > > actually set
>> > > >     > > the
>> > > >     > > > > >     > tombstone.
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > We should compare
>> > this
>> > > >     > migration
>> > > >     > > > > plan
>> > > >     > > > > > with
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > >     > migration
>> > > >     > > > > >     > > >     >     > plan for
>> > > >     > > > > >     > > >     >     >         >> magic
>> > > >     > > > > >     > > >     >     >         >> > > byte bump and do
>> > > > whatever
>> > > >     > looks
>> > > >     > > > > good.
>> > > >     > > > > >     > > >     >     >         >> > > I am just worried
>> > that
>> > > > if we
>> > > >     > go
>> > > >     > > > down
>> > > >     > > > > > magic
>> > > >     > > > > >     > > byte
>> > > >     > > > > >     > > > route,
>> > > >     > > > > >     > > >     >     > unless I am
>> > > >     > > > > >     > > >     >     >         >> > missing
>> > > >     > > > > >     > > >     >     >         >> > > something, it
>> sounds
>> > > > like
>> > > >     > kafka
>> > > >     > > > will
>> > > >     > > > > > be
>> > > >     > > > > >     > stuck
>> > > >     > > > > >     > > > with
>> > > >     > > > > >     > > >     >     > supporting both
>> > > >     > > > > >     > > >     >     >         >> null
>> > > >     > > > > >     > > >     >     >         >> > > value and
>> tombstone
>> > > bit
>> > > > for
>> > > >     > log
>> > > >     > > > > > compaction
>> > > >     > > > > >     > for
>> > > >     > > > > >     > > > life
>> > > >     > > > > >     > > >     > long,
>> > > >     > > > > >     > > >     >     > which does
>> > > >     > > > > >     > > >     >     >         >> not
>> > > >     > > > > >     > > >     >     >         >> > > look like a good
>> end
>> > > > state.
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > Thanks,
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > Mayuresh
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16,
>> 2016
>> > > at
>> > > > 9:32
>> > > >     > AM,
>> > > >     > > > > > Mayuresh
>> > > >     > > > > >     > > > Gharat <
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >     >     >         >> > > > wrote:
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > That's a very
>> good
>> > > > point
>> > > >     > > which I
>> > > >     > > > > > might
>> > > >     > > > > >     > have
>> > > >     > > > > >     > > > not
>> > > >     > > > > >     > > >     >     > considered earlier.
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > Here is a plan
>> > that
>> > > I
>> > > > can
>> > > >     > > think
>> > > >     > > > > of:
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > Stage 1) The
>> > broker
>> > > > from now
>> > > >     > > on,
>> > > >     > > > > up
>> > > >     > > > > >     > converts
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > message
>> > > >     > > > > >     > > >     >     > to have the
>> > > >     > > > > >     > > >     >     >         >> > > > tombstone
>> marker.
>> > > The
>> > > > log
>> > > >     > > > > compaction
>> > > >     > > > > >     > thread
>> > > >     > > > > >     > > > does log
>> > > >     > > > > >     > > >     >     > compaction
>> > > >     > > > > >     > > >     >     >         >> based
>> > > >     > > > > >     > > >     >     >         >> > on
>> > > >     > > > > >     > > >     >     >         >> > > > both null and
>> > > > tombstone
>> > > >     > > marker.
>> > > >     > > > > > This is
>> > > >     > > > > >     > our
>> > > >     > > > > >     > > >     > transition
>> > > >     > > > > >     > > >     >     > period.
>> > > >     > > > > >     > > >     >     >         >> > > > Stage 2) The
>> next
>> > > > release we
>> > > >     > > > only
>> > > >     > > > > > say that
>> > > >     > > > > >     > > log
>> > > >     > > > > >     > > >     > compaction
>> > > >     > > > > >     > > >     >     > is based
>> > > >     > > > > >     > > >     >     >         >> on
>> > > >     > > > > >     > > >     >     >         >> > > > tombstone
>> marker.
>> > > > (Open
>> > > >     > source
>> > > >     > > > > > kafka makes
>> > > >     > > > > >     > > > this as a
>> > > >     > > > > >     > > >     >     > policy). By
>> > > >     > > > > >     > > >     >     >         >> this
>> > > >     > > > > >     > > >     >     >         >> > > time,
>> > > >     > > > > >     > > >     >     >         >> > > > the
>> organization
>> > > > which is
>> > > >     > > moving
>> > > >     > > > > to
>> > > >     > > > > > this
>> > > >     > > > > >     > > > release
>> > > >     > > > > >     > > >     > will be
>> > > >     > > > > >     > > >     >     > sure that
>> > > >     > > > > >     > > >     >     >         >> they
>> > > >     > > > > >     > > >     >     >         >> > > > have gone
>> through
>> > > the
>> > > > entire
>> > > >     > > > > > transition
>> > > >     > > > > >     > > > period.
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > My only goal of
>> > > doing
>> > > > this
>> > > >     > is
>> > > >     > > > that
>> > > >     > > > > > Kafka
>> > > >     > > > > >     > > > clearly
>> > > >     > > > > >     > > >     >     > specifies the end
>> > > >     > > > > >     > > >     >     >         >> > state
>> > > >     > > > > >     > > >     >     >         >> > > > about what log
>> > > > compaction
>> > > >     > > means
>> > > >     > > > > (is
>> > > >     > > > > > it
>> > > >     > > > > >     > null
>> > > >     > > > > >     > > > value
>> > > >     > > > > >     > > >     > or a
>> > > >     > > > > >     > > >     >     > tombstone
>> > > >     > > > > >     > > >     >     >         >> > marker,
>> > > >     > > > > >     > > >     >     >         >> > > > but not both).
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > What do you
>> think?
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > Thanks,
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh
>> > > >     > > > > >     > > >     >     >         >> > > > .
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16,
>> > 2016
>> > > > at 9:17
>> > > >     > > AM,
>> > > >     > > > > > Ismael
>> > > >     > > > > >     > > Juma <
>> > > >     > > > > >     > > >     >     > ismael@juma.me.uk>
>> > > >     > > > > >     > > >     >     >         >> > wrote:
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > >> One comment
>> > below.
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov
>> 16,
>> > > 2016
>> > > > at
>> > > >     > 5:08
>> > > >     > > > PM,
>> > > >     > > > > > Mayuresh
>> > > >     > > > > >     > > > Gharat <
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > > gharatmayuresh15@gmail.com
>> > > >     > > > > >     > > >     >     >         >> > > >> > wrote:
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >> >    - If we
>> > don't
>> > > > bump up
>> > > >     > > the
>> > > >     > > > > > magic
>> > > >     > > > > >     > byte,
>> > > >     > > > > >     > > > on the
>> > > >     > > > > >     > > >     > broker
>> > > >     > > > > >     > > >     >     > side, the
>> > > >     > > > > >     > > >     >     >         >> > > broker
>> > > >     > > > > >     > > >     >     >         >> > > >> >    will
>> always
>> > > > have to
>> > > >     > look
>> > > >     > > > at
>> > > >     > > > > > both
>> > > >     > > > > >     > > > tombstone
>> > > >     > > > > >     > > >     > bit and
>> > > >     > > > > >     > > >     >     > the value
>> > > >     > > > > >     > > >     >     >         >> when
>> > > >     > > > > >     > > >     >     >         >> > > do
>> > > >     > > > > >     > > >     >     >         >> > > >> the
>> > > >     > > > > >     > > >     >     >         >> > > >> >
>> compaction.
>> > > > Assuming
>> > > >     > we
>> > > >     > > do
>> > > >     > > > > > not bump
>> > > >     > > > > >     > up
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > magic
>> > > >     > > > > >     > > >     >     > byte,
>> > > >     > > > > >     > > >     >     >         >> > > >> >    imagine
>> the
>> > > > broker
>> > > >     > sees
>> > > >     > > a
>> > > >     > > > > > message
>> > > >     > > > > >     > > which
>> > > >     > > > > >     > > > does
>> > > >     > > > > >     > > >     > not
>> > > >     > > > > >     > > >     >     > have a
>> > > >     > > > > >     > > >     >     >         >> tombstone
>> > > >     > > > > >     > > >     >     >         >> > > bit
>> > > >     > > > > >     > > >     >     >         >> > > >> >    set. The
>> > > broker
>> > > > does
>> > > >     > not
>> > > >     > > > > know
>> > > >     > > > > > when
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > >     > message was
>> > > >     > > > > >     > > >     >     > produced
>> > > >     > > > > >     > > >     >     >         >> (i.e.
>> > > >     > > > > >     > > >     >     >         >> > > >> > whether
>> > > >     > > > > >     > > >     >     >         >> > > >> >    the
>> message
>> > > has
>> > > > been
>> > > >     > up
>> > > >     > > > > > converted or
>> > > >     > > > > >     > > > not), it
>> > > >     > > > > >     > > >     > has
>> > > >     > > > > >     > > >     >     > to take a
>> > > >     > > > > >     > > >     >     >         >> > further
>> > > >     > > > > >     > > >     >     >         >> > > >> > look at
>> > > >     > > > > >     > > >     >     >         >> > > >> >    the
>> value to
>> > > > see if it
>> > > >     > > is
>> > > >     > > > > > null or
>> > > >     > > > > >     > not
>> > > >     > > > > >     > > in
>> > > >     > > > > >     > > >     > order to
>> > > >     > > > > >     > > >     >     > determine
>> > > >     > > > > >     > > >     >     >         >> if it
>> > > >     > > > > >     > > >     >     >         >> > > is
>> > > >     > > > > >     > > >     >     >         >> > > >> a
>> > > >     > > > > >     > > >     >     >         >> > > >> >
>> tombstone.
>> > The
>> > > > same
>> > > >     > > logic
>> > > >     > > > > has
>> > > >     > > > > > to be
>> > > >     > > > > >     > > put
>> > > >     > > > > >     > > > on the
>> > > >     > > > > >     > > >     >     > consumer as
>> > > >     > > > > >     > > >     >     >         >> well
>> > > >     > > > > >     > > >     >     >         >> > > >> because
>> > > >     > > > > >     > > >     >     >         >> > > >> > the
>> > > >     > > > > >     > > >     >     >         >> > > >> >    consumer
>> > does
>> > > > not know
>> > > >     > > if
>> > > >     > > > > the
>> > > >     > > > > >     > message
>> > > >     > > > > >     > > > has
>> > > >     > > > > >     > > >     > been up
>> > > >     > > > > >     > > >     >     > converted or
>> > > >     > > > > >     > > >     >     >         >> > not.
>> > > >     > > > > >     > > >     >     >         >> > > >> >       - If
>> we
>> > > > upconvert
>> > > >     > > while
>> > > >     > > > > >     > appending,
>> > > >     > > > > >     > > > this is
>> > > >     > > > > >     > > >     > not
>> > > >     > > > > >     > > >     >     > the case,
>> > > >     > > > > >     > > >     >     >         >> > right?
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >> If I
>> understand
>> > you
>> > > >     > > correctly,
>> > > >     > > > > > this is
>> > > >     > > > > >     > not
>> > > >     > > > > >     > > >     > sufficient
>> > > >     > > > > >     > > >     >     > because the
>> > > >     > > > > >     > > >     >     >         >> log
>> > > >     > > > > >     > > >     >     >         >> > > may
>> > > >     > > > > >     > > >     >     >         >> > > >> have messages
>> > > > appended
>> > > >     > before
>> > > >     > > > it
>> > > >     > > > > > was
>> > > >     > > > > >     > > > upgraded to
>> > > >     > > > > >     > > >     > include
>> > > >     > > > > >     > > >     >     > KIP-87.
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >> Ismael
>> > > >     > > > > >     > > >     >     >         >> > > >>
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > > > --
>> > > >     > > > > >     > > >     >     >         >> > > > -Regards,
>> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh R.
>> Gharat
>> > > >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
>> > > >     > > > > >     > > >     >     >         >> > > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > > --
>> > > >     > > > > >     > > >     >     >         >> > > -Regards,
>> > > >     > > > > >     > > >     >     >         >> > > Mayuresh R.
>> Gharat
>> > > >     > > > > >     > > >     >     >         >> > > (862) 250-7125
>> > > >     > > > > >     > > >     >     >         >> > >
>> > > >     > > > > >     > > >     >     >         >> > The information
>> > > contained
>> > > > in
>> > > >     > this
>> > > >     > > > > email
>> > > >     > > > > > is
>> > > >     > > > > >     > > > strictly
>> > > >     > > > > >     > > >     >     > confidential and for
>> > > >     > > > > >     > > >     >     >         >> > the use of the
>> > addressee
>> > > > only,
>> > > >     > > > unless
>> > > >     > > > > >     > otherwise
>> > > >     > > > > >     > > >     > indicated. If
>> > > >     > > > > >     > > >     >     > you are
>> > > >     > > > > >     > > >     >     >         >> not
>> > > >     > > > > >     > > >     >     >         >> > the intended
>> > recipient,
>> > > > please
>> > > >     > do
>> > > >     > > > not
>> > > >     > > > > > read,
>> > > >     > > > > >     > > copy,
>> > > >     > > > > >     > > > use or
>> > > >     > > > > >     > > >     >     > disclose to
>> > > >     > > > > >     > > >     >     >         >> others
>> > > >     > > > > >     > > >     >     >         >> > this message or any
>> > > > attachment.
>> > > >     > > > Please
>> > > >     > > > > > also
>> > > >     > > > > >     > > > notify the
>> > > >     > > > > >     > > >     > sender
>> > > >     > > > > >     > > >     >     > by
>> > > >     > > > > >     > > >     >     >         >> replying
>> > > >     > > > > >     > > >     >     >         >> > to this email or by
>> > > > telephone
>> > > >     > > > (+44(020
>> > > >     > > > > > 7896
>> > > >     > > > > >     > > 0011)
>> > > >     > > > > >     > > > and
>> > > >     > > > > >     > > >     > then
>> > > >     > > > > >     > > >     >     > delete the
>> > > >     > > > > >     > > >     >     >         >> email
>> > > >     > > > > >     > > >     >     >         >> > and any copies of
>> it.
>> > > > Opinions,
>> > > >     > > > > > conclusion
>> > > >     > > > > >     > (etc)
>> > > >     > > > > >     > > > that
>> > > >     > > > > >     > > >     > do not
>> > > >     > > > > >     > > >     >     > relate to
>> > > >     > > > > >     > > >     >     >         >> the
>> > > >     > > > > >     > > >     >     >         >> > official business
>> of
>> > > this
>> > > >     > company
>> > > >     > > > > shall
>> > > >     > > > > > be
>> > > >     > > > > >     > > > understood as
>> > > >     > > > > >     > > >     >     > neither given
>> > > >     > > > > >     > > >     >     >         >> nor
>> > > >     > > > > >     > > >     >     >         >> > endorsed by it. IG
>> is
>> > a
>> > > > trading
>> > > >     > > name
>> > > >     > > > > of
>> > > >     > > > > > IG
>> > > >     > > > > >     > > Markets
>> > > >     > > > > >     > > >     > Limited (a
>> > > >     > > > > >     > > >     >     > company
>> > > >     > > > > >     > > >     >     >         >> > registered in
>> England
>> > > and
>> > > > Wales,
>> > > >     > > > > company
>> > > >     > > > > >     > number
>> > > >     > > > > >     > > >     > 04008957) and
>> > > >     > > > > >     > > >     >     > IG Index
>> > > >     > > > > >     > > >     >     >         >> > Limited (a company
>> > > > registered in
>> > > >     > > > > > England and
>> > > >     > > > > >     > > > Wales,
>> > > >     > > > > >     > > >     > company
>> > > >     > > > > >     > > >     >     > number
>> > > >     > > > > >     > > >     >     >         >> > 01190902).
>> Registered
>> > > > address at
>> > > >     > > > > Cannon
>> > > >     > > > > > Bridge
>> > > >     > > > > >     > > > House, 25
>> > > >     > > > > >     > > >     >     > Dowgate Hill,
>> > > >     > > > > >     > > >     >     >         >> > London EC4R 2YA.
>> Both
>> > IG
>> > > > Markets
>> > > >     > > > > Limited
>> > > >     > > > > >     > > (register
>> > > >     > > > > >     > > >     > number
>> > > >     > > > > >     > > >     >     > 195355) and IG
>> > > >     > > > > >     > > >     >     >         >> > Index Limited
>> > (register
>> > > > number
>> > > >     > > > 114059)
>> > > >     > > > > > are
>> > > >     > > > > >     > > > authorised
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     > regulated by
>> > > >     > > > > >     > > >     >     >         >> the
>> > > >     > > > > >     > > >     >     >         >> > Financial Conduct
>> > > > Authority.
>> > > >     > > > > >     > > >     >     >         >> >
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >> --
>> > > >     > > > > >     > > >     >     >         >> -Regards,
>> > > >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
>> > > >     > > > > >     > > >     >     >         >> (862) 250-7125
>> > > >     > > > > >     > > >     >     >         >> The information
>> > contained
>> > > > in this
>> > > >     > > > email
>> > > >     > > > > is
>> > > >     > > > > >     > > strictly
>> > > >     > > > > >     > > >     >     > confidential and for
>> > > >     > > > > >     > > >     >     >         >> the use of the
>> addressee
>> > > > only,
>> > > >     > > unless
>> > > >     > > > > > otherwise
>> > > >     > > > > >     > > >     > indicated. If
>> > > >     > > > > >     > > >     >     > you are not
>> > > >     > > > > >     > > >     >     >         >> the intended
>> recipient,
>> > > > please do
>> > > >     > > not
>> > > >     > > > > > read,
>> > > >     > > > > >     > copy,
>> > > >     > > > > >     > > > use or
>> > > >     > > > > >     > > >     >     > disclose to others
>> > > >     > > > > >     > > >     >     >         >> this message or any
>> > > > attachment.
>> > > >     > > Please
>> > > >     > > > > > also
>> > > >     > > > > >     > notify
>> > > >     > > > > >     > > > the
>> > > >     > > > > >     > > >     > sender
>> > > >     > > > > >     > > >     >     > by replying
>> > > >     > > > > >     > > >     >     >         >> to this email or by
>> > > > telephone
>> > > >     > > (+44(020
>> > > >     > > > > > 7896
>> > > >     > > > > >     > 0011)
>> > > >     > > > > >     > > > and then
>> > > >     > > > > >     > > >     >     > delete the email
>> > > >     > > > > >     > > >     >     >         >> and any copies of it.
>> > > > Opinions,
>> > > >     > > > > > conclusion (etc)
>> > > >     > > > > >     > > > that do
>> > > >     > > > > >     > > >     > not
>> > > >     > > > > >     > > >     >     > relate to the
>> > > >     > > > > >     > > >     >     >         >> official business of
>> > this
>> > > > company
>> > > >     > > > shall
>> > > >     > > > > be
>> > > >     > > > > >     > > > understood as
>> > > >     > > > > >     > > >     >     > neither given nor
>> > > >     > > > > >     > > >     >     >         >> endorsed by it. IG
>> is a
>> > > > trading
>> > > >     > name
>> > > >     > > > of
>> > > >     > > > > IG
>> > > >     > > > > >     > Markets
>> > > >     > > > > >     > > >     > Limited (a
>> > > >     > > > > >     > > >     >     > company
>> > > >     > > > > >     > > >     >     >         >> registered in England
>> > and
>> > > > Wales,
>> > > >     > > > company
>> > > >     > > > > > number
>> > > >     > > > > >     > > > 04008957)
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     > IG Index
>> > > >     > > > > >     > > >     >     >         >> Limited (a company
>> > > > registered in
>> > > >     > > > England
>> > > >     > > > > > and
>> > > >     > > > > >     > > Wales,
>> > > >     > > > > >     > > >     > company
>> > > >     > > > > >     > > >     >     > number
>> > > >     > > > > >     > > >     >     >         >> 01190902). Registered
>> > > > address at
>> > > >     > > > Cannon
>> > > >     > > > > > Bridge
>> > > >     > > > > >     > > > House, 25
>> > > >     > > > > >     > > >     >     > Dowgate Hill,
>> > > >     > > > > >     > > >     >     >         >> London EC4R 2YA.
>> Both IG
>> > > > Markets
>> > > >     > > > Limited
>> > > >     > > > > >     > (register
>> > > >     > > > > >     > > > number
>> > > >     > > > > >     > > >     >     > 195355) and IG
>> > > >     > > > > >     > > >     >     >         >> Index Limited
>> (register
>> > > > number
>> > > >     > > 114059)
>> > > >     > > > > are
>> > > >     > > > > >     > > > authorised and
>> > > >     > > > > >     > > >     >     > regulated by the
>> > > >     > > > > >     > > >     >     >         >> Financial Conduct
>> > > Authority.
>> > > >     > > > > >     > > >     >     >         >>
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >         > --
>> > > >     > > > > >     > > >     >     >         > -Regards,
>> > > >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
>> > > >     > > > > >     > > >     >     >         > (862) 250-7125
>> > > >     > > > > >     > > >     >     >         >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >         --
>> > > >     > > > > >     > > >     >     >         -Regards,
>> > > >     > > > > >     > > >     >     >         Mayuresh R. Gharat
>> > > >     > > > > >     > > >     >     >         (862) 250-7125
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >     The information contained in
>> > this
>> > > > email
>> > > >     > is
>> > > >     > > > > > strictly
>> > > >     > > > > >     > > > confidential
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     > for the use of the addressee
>> only,
>> > > > unless
>> > > >     > > > otherwise
>> > > >     > > > > >     > > indicated.
>> > > >     > > > > >     > > > If
>> > > >     > > > > >     > > >     > you are
>> > > >     > > > > >     > > >     >     > not the intended recipient,
>> please
>> > do
>> > > > not
>> > > >     > read,
>> > > >     > > > > > copy, use
>> > > >     > > > > >     > or
>> > > >     > > > > >     > > >     > disclose to
>> > > >     > > > > >     > > >     >     > others this message or any
>> > > attachment.
>> > > > Please
>> > > >     > > > also
>> > > >     > > > > > notify
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > > sender
>> > > >     > > > > >     > > >     > by
>> > > >     > > > > >     > > >     >     > replying to this email or by
>> > > telephone
>> > > >     > (+44(020
>> > > >     > > > > 7896
>> > > >     > > > > > 0011)
>> > > >     > > > > >     > > and
>> > > >     > > > > >     > > > then
>> > > >     > > > > >     > > >     > delete
>> > > >     > > > > >     > > >     >     > the email and any copies of it.
>> > > > Opinions,
>> > > >     > > > > conclusion
>> > > >     > > > > > (etc)
>> > > >     > > > > >     > > > that do
>> > > >     > > > > >     > > >     > not
>> > > >     > > > > >     > > >     >     > relate to the official business
>> of
>> > > this
>> > > >     > company
>> > > >     > > > > > shall be
>> > > >     > > > > >     > > > understood
>> > > >     > > > > >     > > >     > as
>> > > >     > > > > >     > > >     >     > neither given nor endorsed by
>> it.
>> > IG
>> > > > is a
>> > > >     > > trading
>> > > >     > > > > > name of
>> > > >     > > > > >     > IG
>> > > >     > > > > >     > > > Markets
>> > > >     > > > > >     > > >     >     > Limited (a company registered in
>> > > > England and
>> > > >     > > > Wales,
>> > > >     > > > > > company
>> > > >     > > > > >     > > > number
>> > > >     > > > > >     > > >     >     > 04008957) and IG Index Limited
>> (a
>> > > > company
>> > > >     > > > > registered
>> > > >     > > > > > in
>> > > >     > > > > >     > > > England and
>> > > >     > > > > >     > > >     > Wales,
>> > > >     > > > > >     > > >     >     > company number 01190902).
>> > Registered
>> > > > address
>> > > >     > at
>> > > >     > > > > > Cannon
>> > > >     > > > > >     > Bridge
>> > > >     > > > > >     > > > House,
>> > > >     > > > > >     > > >     > 25
>> > > >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA.
>> Both
>> > > IG
>> > > >     > Markets
>> > > >     > > > > > Limited
>> > > >     > > > > >     > > > (register
>> > > >     > > > > >     > > >     > number
>> > > >     > > > > >     > > >     >     > 195355) and IG Index Limited
>> > > (register
>> > > > number
>> > > >     > > > > > 114059) are
>> > > >     > > > > >     > > > authorised
>> > > >     > > > > >     > > >     > and
>> > > >     > > > > >     > > >     >     > regulated by the Financial
>> Conduct
>> > > > Authority.
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >     >
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >     > The information contained in this
>> email
>> > is
>> > > > strictly
>> > > >     > > > > > confidential
>> > > >     > > > > >     > > and
>> > > >     > > > > >     > > > for
>> > > >     > > > > >     > > >     > the use of the addressee only, unless
>> > > > otherwise
>> > > >     > > > > indicated.
>> > > >     > > > > > If you
>> > > >     > > > > >     > > > are not
>> > > >     > > > > >     > > >     > the intended recipient, please do not
>> > read,
>> > > > copy,
>> > > >     > use
>> > > >     > > > or
>> > > >     > > > > > disclose
>> > > >     > > > > >     > > to
>> > > >     > > > > >     > > > others
>> > > >     > > > > >     > > >     > this message or any attachment. Please
>> > also
>> > > > notify
>> > > >     > > the
>> > > >     > > > > > sender by
>> > > >     > > > > >     > > > replying
>> > > >     > > > > >     > > >     > to this email or by telephone (+44(020
>> > 7896
>> > > > 0011
>> > > >     > <020%207896%200011>) and
>> > > >     > > > > then
>> > > >     > > > > > delete
>> > > >     > > > > >     > > > the email
>> > > >     > > > > >     > > >     > and any copies of it. Opinions,
>> > conclusion
>> > > > (etc)
>> > > >     > that
>> > > >     > > > do
>> > > >     > > > > > not
>> > > >     > > > > >     > relate
>> > > >     > > > > >     > > > to the
>> > > >     > > > > >     > > >     > official business of this company
>> shall
>> > be
>> > > >     > understood
>> > > >     > > > as
>> > > >     > > > > > neither
>> > > >     > > > > >     > > > given nor
>> > > >     > > > > >     > > >     > endorsed by it. IG is a trading name
>> of
>> > IG
>> > > > Markets
>> > > >     > > > > Limited
>> > > >     > > > > > (a
>> > > >     > > > > >     > > company
>> > > >     > > > > >     > > >     > registered in England and Wales,
>> company
>> > > > number
>> > > >     > > > 04008957)
>> > > >     > > > > > and IG
>> > > >     > > > > >     > > > Index
>> > > >     > > > > >     > > >     > Limited (a company registered in
>> England
>> > > and
>> > > > Wales,
>> > > >     > > > > company
>> > > >     > > > > >     > number
>> > > >     > > > > >     > > >     > 01190902). Registered address at
>> Cannon
>> > > > Bridge
>> > > >     > House,
>> > > >     > > > 25
>> > > >     > > > > > Dowgate
>> > > >     > > > > >     > > > Hill,
>> > > >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register
>> > > >     > > > number
>> > > >     > > > > > 195355)
>> > > >     > > > > >     > > > and IG
>> > > >     > > > > >     > > >     > Index Limited (register number 114059)
>> > are
>> > > >     > authorised
>> > > >     > > > and
>> > > >     > > > > >     > regulated
>> > > >     > > > > >     > > > by the
>> > > >     > > > > >     > > >     > Financial Conduct Authority.
>> > > >     > > > > >     > > >     >
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > > The information contained in this email is
>> > > strictly
>> > > >     > > > > confidential
>> > > >     > > > > > and
>> > > >     > > > > >     > for
>> > > >     > > > > >     > > > the use of the addressee only, unless
>> otherwise
>> > > >     > indicated.
>> > > >     > > If
>> > > >     > > > > > you are
>> > > >     > > > > >     > not
>> > > >     > > > > >     > > > the intended recipient, please do not read,
>> > copy,
>> > > > use or
>> > > >     > > > > > disclose to
>> > > >     > > > > >     > > others
>> > > >     > > > > >     > > > this message or any attachment. Please also
>> > > notify
>> > > > the
>> > > >     > > sender
>> > > >     > > > > by
>> > > >     > > > > >     > replying
>> > > >     > > > > >     > > > to this email or by telephone (+44(020 7896
>> > 0011
>> > > >     > <020%207896%200011>) and then
>> > > >     > > > > > delete the
>> > > >     > > > > >     > > email
>> > > >     > > > > >     > > > and any copies of it. Opinions, conclusion
>> > (etc)
>> > > > that do
>> > > >     > > not
>> > > >     > > > > > relate to
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > > official business of this company shall be
>> > > > understood as
>> > > >     > > > > neither
>> > > >     > > > > > given
>> > > >     > > > > >     > > nor
>> > > >     > > > > >     > > > endorsed by it. IG is a trading name of IG
>> > > Markets
>> > > >     > Limited
>> > > >     > > (a
>> > > >     > > > > > company
>> > > >     > > > > >     > > > registered in England and Wales, company
>> number
>> > > > 04008957)
>> > > >     > > and
>> > > >     > > > > IG
>> > > >     > > > > > Index
>> > > >     > > > > >     > > > Limited (a company registered in England and
>> > > Wales,
>> > > >     > company
>> > > >     > > > > > number
>> > > >     > > > > >     > > > 01190902). Registered address at Cannon
>> Bridge
>> > > > House, 25
>> > > >     > > > > Dowgate
>> > > >     > > > > > Hill,
>> > > >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
>> > > (register
>> > > > number
>> > > >     > > > > > 195355) and
>> > > >     > > > > >     > IG
>> > > >     > > > > >     > > > Index Limited (register number 114059) are
>> > > > authorised and
>> > > >     > > > > > regulated by
>> > > >     > > > > >     > > the
>> > > >     > > > > >     > > > Financial Conduct Authority.
>> > > >     > > > > >     > > >
>> > > >     > > > > >     > > The information contained in this email is
>> > strictly
>> > > >     > > > confidential
>> > > >     > > > > > and for
>> > > >     > > > > >     > > the use of the addressee only, unless
>> otherwise
>> > > > indicated.
>> > > >     > If
>> > > >     > > > you
>> > > >     > > > > > are not
>> > > >     > > > > >     > > the intended recipient, please do not read,
>> copy,
>> > > > use or
>> > > >     > > > disclose
>> > > >     > > > > > to
>> > > >     > > > > >     > others
>> > > >     > > > > >     > > this message or any attachment. Please also
>> > notify
>> > > > the
>> > > >     > sender
>> > > >     > > > by
>> > > >     > > > > > replying
>> > > >     > > > > >     > > to this email or by telephone (+44(020 7896
>> 0011
>> > > >     > <020%207896%200011>) and then
>> > > >     > > > delete
>> > > >     > > > > > the
>> > > >     > > > > >     > email
>> > > >     > > > > >     > > and any copies of it. Opinions, conclusion
>> (etc)
>> > > > that do
>> > > >     > not
>> > > >     > > > > > relate to
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > official business of this company shall be
>> > > > understood as
>> > > >     > > > neither
>> > > >     > > > > > given
>> > > >     > > > > >     > nor
>> > > >     > > > > >     > > endorsed by it. IG is a trading name of IG
>> > Markets
>> > > > Limited
>> > > >     > (a
>> > > >     > > > > > company
>> > > >     > > > > >     > > registered in England and Wales, company
>> number
>> > > > 04008957)
>> > > >     > and
>> > > >     > > > IG
>> > > >     > > > > > Index
>> > > >     > > > > >     > > Limited (a company registered in England and
>> > Wales,
>> > > > company
>> > > >     > > > > number
>> > > >     > > > > >     > > 01190902). Registered address at Cannon Bridge
>> > > > House, 25
>> > > >     > > > Dowgate
>> > > >     > > > > > Hill,
>> > > >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited
>> > (register
>> > > > number
>> > > >     > > > 195355)
>> > > >     > > > > > and IG
>> > > >     > > > > >     > > Index Limited (register number 114059) are
>> > > > authorised and
>> > > >     > > > > > regulated by
>> > > >     > > > > >     > the
>> > > >     > > > > >     > > Financial Conduct Authority.
>> > > >     > > > > >     > >
>> > > >     > > > > >     > >
>> > > >     > > > > >     > The information contained in this email is
>> strictly
>> > > >     > > confidential
>> > > >     > > > > and
>> > > >     > > > > > for
>> > > >     > > > > >     > the use of the addressee only, unless otherwise
>> > > > indicated. If
>> > > >     > > you
>> > > >     > > > > > are not
>> > > >     > > > > >     > the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or
>> > > >     > > disclose
>> > > >     > > > > to
>> > > >     > > > > > others
>> > > >     > > > > >     > this message or any attachment. Please also
>> notify
>> > > the
>> > > > sender
>> > > >     > > by
>> > > >     > > > > > replying
>> > > >     > > > > >     > to this email or by telephone (+44(020 7896 0011
>> > > >     > <020%207896%200011>) and then
>> > > >     > > delete
>> > > >     > > > > > the email
>> > > >     > > > > >     > and any copies of it. Opinions, conclusion (etc)
>> > that
>> > > > do not
>> > > >     > > > relate
>> > > >     > > > > > to the
>> > > >     > > > > >     > official business of this company shall be
>> > understood
>> > > > as
>> > > >     > > neither
>> > > >     > > > > > given nor
>> > > >     > > > > >     > endorsed by it. IG is a trading name of IG
>> Markets
>> > > > Limited (a
>> > > >     > > > > company
>> > > >     > > > > >     > registered in England and Wales, company number
>> > > > 04008957) and
>> > > >     > > IG
>> > > >     > > > > > Index
>> > > >     > > > > >     > Limited (a company registered in England and
>> Wales,
>> > > > company
>> > > >     > > > number
>> > > >     > > > > >     > 01190902). Registered address at Cannon Bridge
>> > House,
>> > > > 25
>> > > >     > > Dowgate
>> > > >     > > > > > Hill,
>> > > >     > > > > >     > London EC4R 2YA. Both IG Markets Limited
>> (register
>> > > > number
>> > > >     > > 195355)
>> > > >     > > > > > and IG
>> > > >     > > > > >     > Index Limited (register number 114059) are
>> > authorised
>> > > > and
>> > > >     > > > regulated
>> > > >     > > > > > by the
>> > > >     > > > > >     > Financial Conduct Authority.
>> > > >     > > > > >     >
>> > > >     > > > > >     >
>> > > >     > > > > >
>> > > >     > > > > >
>> > > >     > > > > > The information contained in this email is strictly
>> > > > confidential
>> > > >     > and
>> > > >     > > > for
>> > > >     > > > > > the use of the addressee only, unless otherwise
>> > indicated.
>> > > > If you
>> > > >     > are
>> > > >     > > > not
>> > > >     > > > > > the intended recipient, please do not read, copy, use
>> or
>> > > > disclose
>> > > >     > to
>> > > >     > > > > others
>> > > >     > > > > > this message or any attachment. Please also notify the
>> > > > sender by
>> > > >     > > > replying
>> > > >     > > > > > to this email or by telephone (+44(020 7896 0011
>> > > >     > <020%207896%200011>) and then delete the
>> > > >     > > > > email
>> > > >     > > > > > and any copies of it. Opinions, conclusion (etc) that
>> do
>> > > not
>> > > > relate
>> > > >     > > to
>> > > >     > > > > the
>> > > >     > > > > > official business of this company shall be understood
>> as
>> > > > neither
>> > > >     > > given
>> > > >     > > > > nor
>> > > >     > > > > > endorsed by it. IG is a trading name of IG Markets
>> > Limited
>> > > (a
>> > > >     > company
>> > > >     > > > > > registered in England and Wales, company number
>> 04008957)
>> > > > and IG
>> > > >     > > Index
>> > > >     > > > > > Limited (a company registered in England and Wales,
>> > company
>> > > > number
>> > > >     > > > > > 01190902). Registered address at Cannon Bridge House,
>> 25
>> > > > Dowgate
>> > > >     > > Hill,
>> > > >     > > > > > London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > > > 195355)
>> > > >     > and
>> > > >     > > > IG
>> > > >     > > > > > Index Limited (register number 114059) are authorised
>> and
>> > > > regulated
>> > > >     > > by
>> > > >     > > > > the
>> > > >     > > > > > Financial Conduct Authority.
>> > > >     > > > > >
>> > > >     > > > > The information contained in this email is strictly
>> > > > confidential and
>> > > >     > > for
>> > > >     > > > > the use of the addressee only, unless otherwise
>> indicated.
>> > If
>> > > > you are
>> > > >     > > not
>> > > >     > > > > the intended recipient, please do not read, copy, use or
>> > > > disclose to
>> > > >     > > > others
>> > > >     > > > > this message or any attachment. Please also notify the
>> > sender
>> > > > by
>> > > >     > > replying
>> > > >     > > > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > >     > and then delete the
>> > > >     > > > email
>> > > >     > > > > and any copies of it. Opinions, conclusion (etc) that do
>> > not
>> > > > relate
>> > > >     > to
>> > > >     > > > the
>> > > >     > > > > official business of this company shall be understood as
>> > > > neither
>> > > >     > given
>> > > >     > > > nor
>> > > >     > > > > endorsed by it. IG is a trading name of IG Markets
>> Limited
>> > (a
>> > > > company
>> > > >     > > > > registered in England and Wales, company number
>> 04008957)
>> > and
>> > > > IG
>> > > >     > Index
>> > > >     > > > > Limited (a company registered in England and Wales,
>> company
>> > > > number
>> > > >     > > > > 01190902). Registered address at Cannon Bridge House, 25
>> > > > Dowgate
>> > > >     > Hill,
>> > > >     > > > > London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > > > 195355) and
>> > > >     > > IG
>> > > >     > > > > Index Limited (register number 114059) are authorised
>> and
>> > > > regulated
>> > > >     > by
>> > > >     > > > the
>> > > >     > > > > Financial Conduct Authority.
>> > > >     > > > >
>> > > >     > > > The information contained in this email is strictly
>> > > confidential
>> > > > and
>> > > >     > for
>> > > >     > > > the use of the addressee only, unless otherwise
>> indicated. If
>> > > > you are
>> > > >     > not
>> > > >     > > > the intended recipient, please do not read, copy, use or
>> > > > disclose to
>> > > >     > > others
>> > > >     > > > this message or any attachment. Please also notify the
>> sender
>> > > by
>> > > >     > replying
>> > > >     > > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > >     > and then delete the
>> > > >     > > email
>> > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
>> not
>> > > > relate to
>> > > >     > > the
>> > > >     > > > official business of this company shall be understood as
>> > > neither
>> > > > given
>> > > >     > > nor
>> > > >     > > > endorsed by it. IG is a trading name of IG Markets
>> Limited (a
>> > > > company
>> > > >     > > > registered in England and Wales, company number 04008957)
>> and
>> > > IG
>> > > > Index
>> > > >     > > > Limited (a company registered in England and Wales,
>> company
>> > > > number
>> > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
>> > > Dowgate
>> > > > Hill,
>> > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
>> > > > 195355) and
>> > > >     > IG
>> > > >     > > > Index Limited (register number 114059) are authorised and
>> > > > regulated by
>> > > >     > > the
>> > > >     > > > Financial Conduct Authority.
>> > > >     > > >
>> > > >     > > The information contained in this email is strictly
>> > confidential
>> > > > and for
>> > > >     > > the use of the addressee only, unless otherwise indicated.
>> If
>> > you
>> > > > are not
>> > > >     > > the intended recipient, please do not read, copy, use or
>> > disclose
>> > > > to
>> > > >     > others
>> > > >     > > this message or any attachment. Please also notify the
>> sender
>> > by
>> > > > replying
>> > > >     > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > >     > and then delete the email
>> > > >     > > and any copies of it. Opinions, conclusion (etc) that do not
>> > > > relate to
>> > > >     > the
>> > > >     > > official business of this company shall be understood as
>> > neither
>> > > > given
>> > > >     > nor
>> > > >     > > endorsed by it. IG is a trading name of IG Markets Limited
>> (a
>> > > > company
>> > > >     > > registered in England and Wales, company number 04008957)
>> and
>> > IG
>> > > > Index
>> > > >     > > Limited (a company registered in England and Wales, company
>> > > number
>> > > >     > > 01190902). Registered address at Cannon Bridge House, 25
>> > Dowgate
>> > > > Hill,
>> > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
>> > 195355)
>> > > > and IG
>> > > >     > > Index Limited (register number 114059) are authorised and
>> > > > regulated by
>> > > >     > the
>> > > >     > > Financial Conduct Authority.
>> > > >     > >
>> > > >     >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If you
>> are
>> > not
>> > > > the intended recipient, please do not read, copy, use or disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then delete
>> the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
>> to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and regulated
>> by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > > The information contained in this email is strictly confidential and
>> for
>> > > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > > the intended recipient, please do not read, copy, use or disclose to
>> > others
>> > > this message or any attachment. Please also notify the sender by
>> replying
>> > > to this email or by telephone (+44(020 7896 0011) and then delete the
>> > email
>> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> > the
>> > > official business of this company shall be understood as neither given
>> > nor
>> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > > registered in England and Wales, company number 04008957) and IG Index
>> > > Limited (a company registered in England and Wales, company number
>> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
>> IG
>> > > Index Limited (register number 114059) are authorised and regulated by
>> > the
>> > > Financial Conduct Authority.
>> > >
>> > The information contained in this email is strictly confidential and for
>> > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > the intended recipient, please do not read, copy, use or disclose to
>> others
>> > this message or any attachment. Please also notify the sender by
>> replying
>> > to this email or by telephone (+44(020 7896 0011) and then delete the
>> email
>> > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> the
>> > official business of this company shall be understood as neither given
>> nor
>> > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > registered in England and Wales, company number 04008957) and IG Index
>> > Limited (a company registered in England and Wales, company number
>> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> > Index Limited (register number 114059) are authorised and regulated by
>> the
>> > Financial Conduct Authority.
>> >
>>
>
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[radai <ra...@gmail.com>]

this kip fixes a "bug" (quirk?) that arises when people implement headers
"in V" (in the payload part of a message).
if you have proper headers you obviously dont need to to stick them in V
and so wont run into this, but its still a valid issue.

On Mon, Dec 19, 2016 at 3:06 PM, Jay Kreps <ja...@confluent.io> wrote:

> Makes sense!
>
> -Jay
>
> On Mon, Dec 19, 2016 at 2:40 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Wow just read that def over tired. Hopefully it makes sense. Or you get
> > the gist at least.
> >
> > ________________________________________
> > From: Michael Pearce <Mi...@ig.com>
> > Sent: Monday, December 19, 2016 9:19:02 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hi Jay,
> >
> > Agreed this stemmed as offshoot from KIP-82.
> >
> > Which our main driver for was to be able to have some headers for a null
> > value as such for our routing, audit, tracing and a few other bits which
> > currently we are forced to do with a message wrapper, if we all agreed on
> > KIP-82 that we need native headers and look to implement that the push
> for
> > this would dissipate.
> >
> > This KIP would allow for though one use case that comes to mind we could
> > see which is to have business data with a delete. Though as said this
> isn't
> > something we are pushing for think really we would have.
> >
> > As such in summary yes, if you want to fully support KIP-82 and we can
> get
> > that agreed in principle and a target release version, I think quite a
> few
> > guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.
> >
> > Cheers
> > Mike
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jay Kreps <ja...@confluent.io>
> > Sent: Monday, December 19, 2016 8:51:23 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hey Michael,
> >
> > Here is the compatibility concern I have:
> >
> >    1. You have a consumer app that relies on value == null to indicate a
> >    delete (current semantics).
> >    2. You upgrade Kafka and your clients.
> >    3. Some producer starts using the tombstone field in combination with
> >    non-null.
> >
> > I share Ismael's dislike of setting tombstones on records with null
> values.
> > This makes sense as a transitional state, but as an end state its a bit
> > weird. You'd expect to be able to mix null values and tombstones, and
> have
> > the null values remain and the tombstones get compacted. However what
> will
> > happen is both will be compacted and upon debugging this you'll learn
> that
> > we sometimes use null in the value to indicate tombstone. Ismael's
> solution
> > is a bigger compatibility break, though, so not sure if that is better.
> >
> > My other question is the relationship to KIP-82. My read is that this KIP
> > solves some but not all of the problems KIP-82 is intended for. KIP-82,
> on
> > the other hand, seems to address most of the motivating uses for this
> KIP.
> > The exception is maybe item (5) on the list where you want to
> simultaneous
> > delete and convey some information to subscribers, but I couldn't
> construct
> > a concrete examples for that one. Do we need to rationalize these two
> KIPs?
> > That is, do you still advocate this proposal if we do KIP-82 and vice
> > versa? As you may have noticed I'm somewhat emotionally invested in the
> > simplicity of the core data model, so my default position is let's try to
> > avoid stuffing more stuff in, but if we have to add stuff I like each of
> > these individually more than doing both. :-)
> >
> > -Jay
> >
> >
> >
> >
> > On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jay
> > >
> > > I disagree here that we are breaking any compatibility, we went through
> > > this on the discussion thread in fact with the help of that thread is
> how
> > > the kip came to the solution.
> > >
> > > Also on the supported combinations front you mention, we are not taking
> > > anything away afaik.
> > >
> > > Currently supported are only are:
> > > Null value = delete
> > > Non-null value = non delete
> > >
> > > With this kip we would support
> > > Null value + tombstone = delete
> > > Non null value + tombstone = delete
> > > Non null value + no tombstone = non delete
> > >
> > > As for the alternative idea, this is simply a new policy, how can there
> > be
> > > confusion here? For this policy it would be explicit that tombstone
> > marker
> > > would need to be set for a delete.
> > >
> > > I'm going to vent a little now as starting to get quite frustrated.
> > >
> > > We are going round in circles on kip-82 as per use cases there is now
> > many
> > > use cases, how many more are needed? just because confluent don't see
> > these
> > > doesn't mean they aren't real use cases other have, this is the point
> of
> > > the Apache foundation, it shouldn't be the view of just one
> organisation.
> > > It really is getting a feeling of the NIH syndrome. Rather than it
> being
> > > constructive on discussion of the implementation detail.
> > >
> > > kip-87 spawned from as on the kip call we all agreed this was needed.
> And
> > > would at least allow a custom wrapper be supported in a compacted
> topic,
> > > allowing meta data. Which again now I feel we are spinning wheels, and
> > > simply finding reasons not support it.
> > >
> > > Cheers
> > > Mike
> > >
> > >
> > >
> > > Sent using OWA for iPad
> > > ________________________________________
> > > From: Jay Kreps <ja...@confluent.io>
> > > Sent: Friday, December 16, 2016 7:09:23 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > > Hey Michael,
> > >
> > > I do think it might have been better had we started with a separate
> > concept
> > > of null vs delete. Given that we are where we are, I'm not sure that
> the
> > > possible cures we explored so far are better than the disease.
> > >
> > > I apologize for coming to this late, but I didn't really understand the
> > > implications of the KIP and that we'd be breaking compatibility with
> > > existing apps until the vote had begun, and in my defense I'm not sure
> > the
> > > other folks voting did either.
> > >
> > > I think we all agree there are many existing apps that are built with
> the
> > > assumption of "null value non-tombstone" and it isn't possible to
> > > disambiguate these from tombstones on the producer. It isn't that
> anyone
> > is
> > > saying we have to support all four possibilities at once, it's that we
> > > simply can't orphan one of the existing combinations or our users will
> > eat
> > > us!
> > >
> > > If I've understood your alternate solution of adding another setting
> for
> > > compaction, I think this does fix the compatibility problem, but it
> adds
> > an
> > > odd mode the user has to add on all their topics. While the current
> state
> > > is easily explainable, the resulting state where the meaning of
> tombstone
> > > and null are overlapping and ambiguous and dependent on a compaction
> > > setting that could change out of band or not be in sync with your code
> in
> > > some environment seems worse to me then where we currently are. I think
> > the
> > > question is how would this combination be explained to users and does
> it
> > > make sense?
> > >
> > > -Jay
> > >
> > >
> > >
> > > On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> > > wrote:
> > >
> > > > Hi Chaps,
> > > >
> > > > Can we either get one more +1 binding (we have 2 already) on the
> > > existing?
> > > >
> > > > Or have some response on the below possible alternative. We are keen
> to
> > > > get working on this, so we make next feature release.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > >
> > > > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com>
> wrote:
> > > >
> > > >     Hi Ismael
> > > >
> > > >     Did you see our email this morning, what's your thoughts on this
> > > > approach to instead we simply have a brand new policy?
> > > >
> > > >     Cheers
> > > >     Mike
> > > >
> > > >
> > > >     Sent using OWA for iPhone
> > > >     ________________________________________
> > > >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
> > > Juma <
> > > > ismael@juma.me.uk>
> > > >     Sent: Tuesday, December 13, 2016 11:30:05 AM
> > > >     To: dev@kafka.apache.org
> > > >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >
> > > >     Yes, this is actually tricky to do in a way where we both have
> the
> > > > desired
> > > >     semantics and maintain compatibility. When someone creates a
> > > >     `ProducerRecord` with a `null` value today, the producer doesn't
> > know
> > > > if
> > > >     it's meant to be a tombstone or not. For V3 messages, it's easy
> > when
> > > > the
> > > >     constructor that takes a tombstone is used. However, if any other
> > > >     constructor is used, it's not clear. A couple of options I can
> > think
> > > > of,
> > > >     none of them particularly nice:
> > > >
> > > >     1. Have a third state where tombstone = unknown and the broker
> > would
> > > > set
> > > >     the tombstone bit if the value was null and the topic was
> > compacted.
> > > > People
> > > >     that wanted to pass a non-null value for the tombstone would have
> > to
> > > > use
> > > >     the constructor that takes a tombstone. The drawbacks: third
> state
> > > for
> > > >     tombstone in message format, message conversion at the broker
> for a
> > > > common
> > > >     case.
> > > >
> > > >     2. Extend MetadataResponse to optionally include topic configs,
> > which
> > > > would
> > > >     make it possible for the producer to be smarter about setting the
> > > >     tombstone. It would only do it if a tombstone was not passed
> > > > explicitly,
> > > >     the value was null and the topic was compacted. The main drawback
> > is
> > > > that
> > > >     the producer would be getting a bit more data for each topic even
> > > > though it
> > > >     probably won't use it most of the time. Extending
> MetadataResponse
> > to
> > > >     return topic configs would be useful for other reasons as well,
> so
> > > that
> > > >     part seems OK.
> > > >
> > > >     In addition, for both proposals, we could consider adding
> warnings
> > to
> > > > the
> > > >     documentation that the behaviour of the constructors that don't
> > take
> > > a
> > > >     tombstone would change in the next major release so that
> tombstone
> > =
> > > > false.
> > > >     Not sure if this would be worth it though.
> > > >
> > > >     Ismael
> > > >
> > > >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> > > > ewen@confluent.io>
> > > >     wrote:
> > > >
> > > >     > Michael,
> > > >     >
> > > >     > It kind of depends on how you want to interpret the tombstone
> > flag.
> > > > If it's
> > > >     > purely a producer-facing Kafka-level thing that we treat as
> > > internal
> > > > to the
> > > >     > broker and log cleaner once the record is sent, then your
> > approach
> > > > makes
> > > >     > sense. You're just moving copying the null-indicates-delete
> > > behavior
> > > > of the
> > > >     > old constructor into the tombstone flag.
> > > >     >
> > > >     > However, if you want this change to more generally decouple the
> > > idea
> > > > of
> > > >     > deletion and null values, then you are sometimes converting
> what
> > > > might be a
> > > >     > completely valid null value that doesn't indicate deletion
> into a
> > > >     > tombstone. Downstream applications could potentially handle
> these
> > > > cases
> > > >     > differently given the separation of deletion from value.
> > > >     >
> > > >     > I guess the question is if we want to try to support the latter
> > > even
> > > > for
> > > >     > topics where we have older produce requests. An example where
> > this
> > > > could
> > > >     > come up is in something like a CDC Connector. If we try to
> > support
> > > > the
> > > >     > semantic difference, a connector might write changes to Kafka
> > using
> > > > the
> > > >     > tombstone flag to indicate when a row was truly deleted (vs an
> > > > update that
> > > >     > sets it to null but still present; this probably makes more
> sense
> > > > for CDC
> > > >     > from document stores or extracting single columns). There are
> > > various
> > > >     > reasons we might want to maintain the full log and not turn
> > > > compaction on
> > > >     > (or just use a time-based retention policy), but downstream
> > > > applications
> > > >     > might care to know the difference between a delete and a null
> > > value.
> > > > In
> > > >     > fact both versions of the same log (compacted and
> time-retention)
> > > > could be
> > > >     > useful and I don't think it'll be uncommon to maintain both or
> > use
> > > > KIP-71
> > > >     > to maintain a hybrid compacted/retention topic.
> > > >     >
> > > >     > -Ewen
> > > >     >
> > > >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > >     > wrote:
> > > >     >
> > > >     > > Hi Jay,
> > > >     > >
> > > >     > > Why wouldn't that work, the tombstone value is only looked at
> > by
> > > > the
> > > >     > > broker, on a topic configured for compaction as such is
> benign
> > on
> > > > non
> > > >     > > compacted topics. This is as much as sending a null value
> > > currently
> > > >     > >
> > > >     > >
> > > >     > > Regards
> > > >     > > Mike
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > > Sent using OWA for iPhone
> > > >     > > ________________________________________
> > > >     > > From: Jay Kreps <ja...@confluent.io>
> > > >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
> > > >     > > To: dev@kafka.apache.org
> > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >     > >
> > > >     > > Hey Michael,
> > > >     > >
> > > >     > > I'm not quite sure that works as that would translate ALL
> null
> > > > values to
> > > >     > > tombstones, even for non-compacted topics that use null as an
> > > > acceptable
> > > >     > > value sent by the producer and expected by the consumer.
> > > >     > >
> > > >     > > -Jay
> > > >     > >
> > > >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > >     > > wrote:
> > > >     > >
> > > >     > > > Hi Ewen,
> > > >     > > >
> > > >     > > > I think the easiest way to show this is with code.
> > > >     > > >
> > > >     > > > As you can see we keep the existing behaviour for
> > code/binaries
> > > > calling
> > > >     > > > the pre-existing constructors, whereby if the value is null
> > the
> > > >     > tombstone
> > > >     > > > is set to true.
> > > >     > > >
> > > >     > > > Regards
> > > >     > > > Mike
> > > >     > > >
> > > >     > > >
> > > >     > > >
> > > >     > > >     /**
> > > >     > > >      * Creates a record with a specified timestamp to be
> sent
> > > to
> > > > a
> > > >     > > > specified topic and partition
> > > >     > > >      *
> > > >     > > >      * @param topic The topic the record will be appended
> to
> > > >     > > >      * @param partition The partition to which the record
> > > should
> > > > be
> > > >     > sent
> > > >     > > >      * @param timestamp The timestamp of the record
> > > >     > > >      * @param tombstone if the record should be treated as
> a
> > > > tombstone
> > > >     > if
> > > >     > > > the topic is compacted
> > > >     > > >      * @param key The key that will be included in the
> record
> > > >     > > >      * @param value The record contents
> > > >     > > >      */
> > > >     > > >     public ProducerRecord(String topic, Integer partition,
> > > > Boolean
> > > >     > > > tombstone, Long timestamp, K key, V value) {
> > > >     > > >         if (topic == null)
> > > >     > > >             throw new IllegalArgumentException("Topic
> cannot
> > > be
> > > >     > null.");
> > > >     > > >         if (timestamp != null && timestamp < 0)
> > > >     > > >             throw new IllegalArgumentException(
> > > >     > > >                     String.format("Invalid timestamp: %d.
> > > > Timestamp
> > > >     > > should
> > > >     > > > always be non-negative or null.", timestamp));
> > > >     > > >         if (partition != null && partition < 0)
> > > >     > > >             throw new IllegalArgumentException(
> > > >     > > >                     String.format("Invalid partition: %d.
> > > > Partition
> > > >     > > number
> > > >     > > > should always be non-negative or null.", partition));
> > > >     > > >         if (!tombstone && value == null){
> > > >     > > >             throw new IllegalArgumentException(
> > > >     > > >                     String.format("Tombstone must be true
> if
> > > null
> > > >     > > value"));
> > > >     > > >         }
> > > >     > > >         this.topic = topic;
> > > >     > > >         this.partition = partition;
> > > >     > > >         this.tombstone = tombstone;
> > > >     > > >         this.key = key;
> > > >     > > >         this.value = value;
> > > >     > > >         this.timestamp = timestamp;
> > > >     > > >     }
> > > >     > > >
> > > >     > > >     public ProducerRecord(String topic, Integer partition,
> > Long
> > > >     > > timestamp,
> > > >     > > > K key, V value) {
> > > >     > > >         this(topic, partition, value == null, timestamp,
> key,
> > > > value);
> > > >     > > >     }
> > > >     > > > ________________________________________
> > > >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> > > >     > > > Sent: Sunday, December 11, 2016 5:45 AM
> > > >     > > > To: dev@kafka.apache.org
> > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >     > > >
> > > >     > > > It seemed like this was addressed in the migration section,
> > > > wasn't it?
> > > >     > > The
> > > >     > > > V2 vs V3 requests and the fact that the broker will
> downgrade
> > > the
> > > >     > message
> > > >     > > > format (losing zero copy) if you issues a V2 request to a
> > > broker
> > > > using
> > > >     > V3
> > > >     > > > format handles compatibility, does it not? The existing
> > > > consumers won't
> > > >     > > see
> > > >     > > > the extra metadata in the value, but they will get a null
> > > > instead and
> > > >     > > treat
> > > >     > > > it as a tombstone. Certainly there is a performance impact,
> > but
> > > > it
> > > >     > seemed
> > > >     > > > compatible.
> > > >     > > >
> > > >     > > > I'm worried about this though:
> > > >     > > >
> > > >     > > >
> > > >     > > >    - *NOTE *: With the new version of producer client using
> > > >     > > ProduceRequest
> > > >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not
> > set)
> > > > and
> > > >     > null
> > > >     > > > value
> > > >     > > >    should not be allowed as the older version of consumer
> > using
> > > >     > > > FetchRequest
> > > >     > > >    V2 will think of this as a tombstone when its actually
> > not.
> > > >     > > >
> > > >     > > >
> > > >     > > > Unless I'm misunderstanding, this ends up breaking binary
> > > > compatibility
> > > >     > > for
> > > >     > > > the Java API. It sounds like this suggests that passing a
> > null
> > > > value to
> > > >     > > the
> > > >     > > > existing ProducerRecord constructors would generate an
> > > exception
> > > > since
> > > >     > > you
> > > >     > > > didn't explicitly enable the tombstone (via whatever new
> > > > constructor is
> > > >     > > > provided). But that means you can't swap in a newer client
> > jar
> > > > without
> > > >     > > > recompiling and get the same behavior if your app deletes
> > keys
> > > > using
> > > >     > the
> > > >     > > > current approach because your app will start throwing
> > > > exceptions. Maybe
> > > >     > > > this is a tradeoff we're ok with, but we've tried pretty
> hard
> > > to
> > > > avoid
> > > >     > > > breaking compatibility like this so far -- it makes picking
> > up
> > > > bug
> > > >     > fixes
> > > >     > > in
> > > >     > > > the clients harder for users of frameworks that have to pin
> > to
> > > > earlier
> > > >     > > > default versions for compatibility.
> > > >     > > >
> > > >     > > > But then later the KIP says:
> > > >     > > >
> > > >     > > >
> > > >     > > >    - The old constructors for ProducerRecord without this
> > > > parameter
> > > >     > will
> > > >     > > be
> > > >     > > >    kept but updated so that their default behaviour if
> > setting
> > > > null
> > > >     > value
> > > >     > > > will
> > > >     > > >    be the tombstone will be set to true to keep existing
> > > > behaviour.
> > > >     > > >
> > > >     > > >
> > > >     > > > So maybe I am misinterpreting something.
> > > >     > > >
> > > >     > > > And just a nit re: motivation section, item 6 would be
> > clearer
> > > > for a
> > > >     > > union
> > > >     > > > schema with null (or optional schemas in other formats),
> e.g.
> > > > [null,
> > > >     > > > string], in which case losing the schema truly is losing
> > > > information
> > > >     > > > (whereas null is already the only valid value for a pure
> null
> > > > schema).
> > > >     > > >
> > > >     > > > -Ewen
> > > >     > > >
> > > >     > > >
> > > >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > >     > >
> > > >     > > > wrote:
> > > >     > > >
> > > >     > > > > Hi Jay,
> > > >     > > > >
> > > >     > > > > Good point this detail is missing in the KIP write up.
> Ive
> > > > added this
> > > >     > > > now.
> > > >     > > > >
> > > >     > > > > Essentially simply just upgrading the clients we do not
> > > expect
> > > > any
> > > >     > > client
> > > >     > > > > app code change needed.
> > > >     > > > >
> > > >     > > > > Cheers
> > > >     > > > > Mike
> > > >     > > > > ________________________________________
> > > >     > > > > From: Jay Kreps <ja...@confluent.io>
> > > >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
> > > >     > > > > To: dev@kafka.apache.org
> > > >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> Flag
> > > >     > > > >
> > > >     > > > > Michael,
> > > >     > > > >
> > > >     > > > > The compatibility section goes through the migration
> path,
> > > but
> > > > isn't
> > > >     > > the
> > > >     > > > > bigger compatibility issue with existing apps? There are
> > many
> > > >     > (probably
> > > >     > > > > thousands) of apps in production that use this feature
> and
> > > > send null
> > > >     > to
> > > >     > > > > mean delete. It seems like this would break compatibility
> > > with
> > > > them,
> > > >     > > and
> > > >     > > > > they would have to be rewritten to right?
> > > >     > > > >
> > > >     > > > > -Jay
> > > >     > > > >
> > > >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> > > >     > Michael.Pearce@ig.com
> > > >     > > >
> > > >     > > > > wrote:
> > > >     > > > >
> > > >     > > > > > Hi Jun,
> > > >     > > > > >
> > > >     > > > > > 4) On v3 we honour the tombstone. As such we expect it
> to
> > > be
> > > > set
> > > >     > > > > correctly
> > > >     > > > > > as per the KIP.
> > > >     > > > > >
> > > >     > > > > > 4.1) why would we want to produce an error when its v3?
> > > This
> > > > is the
> > > >     > > > exact
> > > >     > > > > > purpose to support non-null tombstone's
> > > >     > > > > > 4.2) again here im unclear on the question on the v3,
> > > produce
> > > >     > request
> > > >     > > > we
> > > >     > > > > > expect the tombstone flag to be set correctly.
> > > >     > > > > >
> > > >     > > > > > 4.4) compaction only occurs on compacted topics, the
> bit
> > > > makes no
> > > >     > > > > > difference and not looked at on un-compacted (time/size
> > > based
> > > >     > > > eviction).
> > > >     > > > > >
> > > >     > > > > >
> > > >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io>
> > wrote:
> > > >     > > > > >
> > > >     > > > > >     Hi, Michael,
> > > >     > > > > >
> > > >     > > > > >     4. Then, I think I misunderstood this point. Could
> > you
> > > > document
> > > >     > > the
> > > >     > > > > >     following points in the wiki?
> > > >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
> > > > non-null
> > > >     > value,
> > > >     > > > > does
> > > >     > > > > > the
> > > >     > > > > >     send() get an error or does the producer
> > automatically
> > > > set the
> > > >     > > > value
> > > >     > > > > to
> > > >     > > > > >     null?
> > > >     > > > > >     4.2 If producer V3 doesn't set tombstone, but
> > provides
> > > a
> > > > null
> > > >     > > > value,
> > > >     > > > > > does
> > > >     > > > > >     the send() get an error or does the producer
> > > > automatically sets
> > > >     > > the
> > > >     > > > > >     tombstone?
> > > >     > > > > >     4.3 Does the broker only expect messages that (a)
> > have
> > > no
> > > >     > > tombstone
> > > >     > > > > and
> > > >     > > > > >     non-null value; (b) have tombstone and null value
> and
> > > > reject
> > > >     > the
> > > >     > > > > > messages
> > > >     > > > > >     with an error code otherwise?
> > > >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic
> is
> > > > compacted
> > > >     > on
> > > >     > > > > not?
> > > >     > > > > >
> > > >     > > > > >     Thanks,
> > > >     > > > > >
> > > >     > > > > >     Jun
> > > >     > > > > >
> > > >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > > >     > > > > Michael.Pearce@ig.com
> > > >     > > > > > >
> > > >     > > > > >     wrote:
> > > >     > > > > >
> > > >     > > > > >     > Not at all.  This only acts on compacted topics
> > just
> > > > as what
> > > >     > > > occurs
> > > >     > > > > > today
> > > >     > > > > >     >
> > > >     > > > > >     > Sent using OWA for iPhone
> > > >     > > > > >     > ________________________________________
> > > >     > > > > >     > From: Jun Rao <ju...@confluent.io>
> > > >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > > >     > > > > >     > To: dev@kafka.apache.org
> > > >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone
> > > > Flag
> > > >     > > > > >     >
> > > >     > > > > >     > Hi, Michael,
> > > >     > > > > >     >
> > > >     > > > > >     > 4. Hmm, does that mean the new client library can
> > > > never send
> > > >     > a
> > > >     > > > null
> > > >     > > > > > message
> > > >     > > > > >     > even to a regular topic? This seems like a change
> > of
> > > > the
> > > >     > > existing
> > > >     > > > > > behavior.
> > > >     > > > > >     >
> > > >     > > > > >     > Thanks,
> > > >     > > > > >     >
> > > >     > > > > >     > Jun
> > > >     > > > > >     >
> > > >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > >     > > > > > Michael.Pearce@ig.com>
> > > >     > > > > >     > wrote:
> > > >     > > > > >     >
> > > >     > > > > >     > > Hi Jun,
> > > >     > > > > >     > >
> > > >     > > > > >     > > Re 4) That's because we expect the tombstone
> > value
> > > > to be
> > > >     > set
> > > >     > > > > > correctly if
> > > >     > > > > >     > > message bit is 2, as such if an older client
> > sends
> > > > in on
> > > >     > old
> > > >     > > > > > message the
> > > >     > > > > >     > > message is upcast and the bit is set correctly.
> > And
> > > > such no
> > > >     > > > > longer
> > > >     > > > > > need
> > > >     > > > > >     > to
> > > >     > > > > >     > > check the value. Mayuresh can you confirm my
> > > > thinking and
> > > >     > > > > > understanding
> > > >     > > > > >     > of
> > > >     > > > > >     > > what we've discussed?
> > > >     > > > > >     > >
> > > >     > > > > >     > > The second point I understand what you're
> getting
> > > at
> > > > now my
> > > >     > > > > > apologies.
> > > >     > > > > >     > Yes
> > > >     > > > > >     > > this makes sense to save on touching the
> message,
> > > if
> > > > we're
> > > >     > > the
> > > >     > > > > > only kip
> > > >     > > > > >     > > going in, in this release.
> > > >     > > > > >     > >
> > > >     > > > > >     > > Cheers
> > > >     > > > > >     > > Mike
> > > >     > > > > >     > >
> > > >     > > > > >     > > Sent using OWA for iPhone
> > > >     > > > > >     > > ________________________________________
> > > >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
> > > >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > > >     > > > > >     > > To: dev@kafka.apache.org
> > > >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > > > Tombstone Flag
> > > >     > > > > >     > >
> > > >     > > > > >     > > Hi, Michael,
> > > >     > > > > >     > >
> > > >     > > > > >     > > 4. Is this updated in the wiki? The text "If
> the
> > > > magic byte
> > > >     > > on
> > > >     > > > > > message is
> > > >     > > > > >     > > 2, the broker should use the tombstone bit for
> > log
> > > >     > > compaction."
> > > >     > > > > > doesn't
> > > >     > > > > >     > > seem to have changed.
> > > >     > > > > >     > >
> > > >     > > > > >     > > 2. My point is that if we change the message
> > format
> > > > just
> > > >     > for
> > > >     > > > this
> > > >     > > > > > KIP, we
> > > >     > > > > >     > > should consider whether it's worth optimizing
> the
> > > > down
> > > >     > > > conversion
> > > >     > > > > > path
> > > >     > > > > >     > > (i.e., decide whether a conversion is needed by
> > > just
> > > >     > looking
> > > >     > > at
> > > >     > > > > the
> > > >     > > > > >     > > tombstone bit in the wrapper message) since
> > > > tombstone will
> > > >     > be
> > > >     > > > > used
> > > >     > > > > >     > rarely.
> > > >     > > > > >     > > However, if the message format change here is
> > > > combined with
> > > >     > > > other
> > > >     > > > > > KIPs,
> > > >     > > > > >     > > then this optimization likely won't be needed.
> > The
> > > > latter
> > > >     > > > > probably
> > > >     > > > > > makes
> > > >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do
> you
> > > > think?
> > > >     > > > > >     > >
> > > >     > > > > >     > > Other than those, +1 from me,
> > > >     > > > > >     > >
> > > >     > > > > >     > > Thanks,
> > > >     > > > > >     > >
> > > >     > > > > >     > > Jun
> > > >     > > > > >     > >
> > > >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce
> <
> > > >     > > > > > Michael.Pearce@ig.com>
> > > >     > > > > >     > > wrote:
> > > >     > > > > >     > >
> > > >     > > > > >     > > > Hi Jun
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > do we have your vote on this now?
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > Any other concerns?
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > Cheers
> > > >     > > > > >     > > > Mike
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > Sent using OWA for iPhone
> > > >     > > > > >     > > > ________________________________________
> > > >     > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> > > >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > > >     > > > > >     > > > To: dev@kafka.apache.org
> > > >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > > > Tombstone
> > > >     > Flag
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > Hi Jun,
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > Have updated. Thanks again for the feedback.
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > Agree yes we should align up when it gets to
> > > that,
> > > > I
> > > >     > assume
> > > >     > > > > > you've
> > > >     > > > > >     > > flagged
> > > >     > > > > >     > > > the same in the other KIP?
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > I think for now let's get this KIP approved,
> > then
> > > > we can
> > > >     > > > start
> > > >     > > > > > the work
> > > >     > > > > >     > > to
> > > >     > > > > >     > > > get an initial PR, then we can discuss how to
> > > > align the
> > > >     > two
> > > >     > > > if
> > > >     > > > > > needed.
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > Cheers,
> > > >     > > > > >     > > > Mike
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
> > > jun@confluent.io>
> > > >     > wrote:
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >     Hi, Michael,
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >     For 2), this is fine. Could you update
> the
> > > KIP
> > > > wiki
> > > >     > to
> > > >     > > > make
> > > >     > > > > > this
> > > >     > > > > >     > and
> > > >     > > > > >     > > > other
> > > >     > > > > >     > > >     points clearer? Other than that, the KIP
> > > looks
> > > > good
> > > >     > to
> > > >     > > > me.
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >     An orthogonal thing is that there are
> other
> > > > KIPs such
> > > >     > > as
> > > >     > > > > > KIP-98
> > > >     > > > > >     > that
> > > >     > > > > >     > > > also
> > > >     > > > > >     > > >     intend to change the message format. If
> > they
> > > > all get
> > > >     > > > > > approved, we
> > > >     > > > > >     > > > should
> > > >     > > > > >     > > >     think about whether it's better to just
> > bump
> > > > up the
> > > >     > > magic
> > > >     > > > > > byte once
> > > >     > > > > >     > > to
> > > >     > > > > >     > > >     incorporate multiple format changes like
> we
> > > > did in
> > > >     > > > > > KIP-31/KIP-32.
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >     Thanks,
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >     Jun
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
> > > Pearce
> > > > <
> > > >     > > > > >     > > Michael.Pearce@ig.com>
> > > >     > > > > >     > > >     wrote:
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >     > Hi Jao
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     > Thanks for the response. Sorry for slow
> > > > reply, both
> > > >     > > > with
> > > >     > > > > > personal
> > > >     > > > > >     > > > sickness
> > > >     > > > > >     > > >     > and also battling some critical issues
> > > > encountered
> > > >     > > > since
> > > >     > > > > >     > upgrading
> > > >     > > > > >     > > to
> > > >     > > > > >     > > >     > 0.10.1.0
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     > 1) Thans for spotting, Document error
> > where
> > > > we
> > > >     > > branched
> > > >     > > > > > this KIP
> > > >     > > > > >     > > from
> > > >     > > > > >     > > >     > KIP-82, will get that removed.
> > > >     > > > > >     > > >     > 2) Intent is to do this just at the
> > record
> > > > message
> > > >     > > > level.
> > > >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure
> this
> > is
> > > >     > > corrected.
> > > >     > > > > >     > > >     > 4) As per discussion thread we will
> > support
> > > >     > > tombstone +
> > > >     > > > > > null
> > > >     > > > > >     > value,
> > > >     > > > > >     > > >     > tombstone + non null value, no
> tombstone
> > +
> > > > null
> > > >     > > value.
> > > >     > > > > >     > > >     > 5) I believe this was in the discussion
> > > > thread,
> > > >     > > > @Mayuresh
> > > >     > > > > > is this
> > > >     > > > > >     > > >     > something we've overlooked? I thought
> we
> > > > would down
> > > >     > > > > > convert and
> > > >     > > > > >     > > > remove the
> > > >     > > > > >     > > >     > value so the old consumer had existing
> > > > behavior, or
> > > >     > > is
> > > >     > > > > > there
> > > >     > > > > >     > > > something we
> > > >     > > > > >     > > >     > haven't thought about?
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     > Cheers
> > > >     > > > > >     > > >     > Mike
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> > > > jun@confluent.io>
> > > >     > > > > wrote:
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     Hi, Michael,
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     Thanks for the KIP. A few comments
> > > below.
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     1. The message format change
> contains
> > > >     > > > "HeadersLength
> > > >     > > > > >     > Headers".
> > > >     > > > > >     > > > Is that
> > > >     > > > > >     > > >     >     intended?
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     2. For compressed messageset, is
> the
> > > > tombstone
> > > >     > > bit
> > > >     > > > > > only set
> > > >     > > > > >     > at
> > > >     > > > > >     > > > the
> > > >     > > > > >     > > >     > shallow
> > > >     > > > > >     > > >     >     level? Do we always leave that bit
> in
> > > the
> > > >     > wrapper
> > > >     > > > > > message
> > > >     > > > > >     > > unset?
> > > >     > > > > >     > > > An
> > > >     > > > > >     > > >     >     alternative is to set the tombstone
> > bit
> > > > in the
> > > >     > > > > wrapper
> > > >     > > > > > if at
> > > >     > > > > >     > > > least one
> > > >     > > > > >     > > >     >     inner message has the tombstone bit
> > > set.
> > > > This
> > > >     > > makes
> > > >     > > > > > things a
> > > >     > > > > >     > > bit
> > > >     > > > > >     > > > more
> > > >     > > > > >     > > >     >     complicated, but we could
> potentially
> > > > exploit
> > > >     > > that
> > > >     > > > > for
> > > >     > > > > >     > > > optimizing down
> > > >     > > > > >     > > >     >     conversion. For example, we only
> need
> > > to
> > > >     > convert
> > > >     > > > > > messages
> > > >     > > > > >     > with
> > > >     > > > > >     > > > magic 2
> > > >     > > > > >     > > >     > to
> > > >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone
> > bit
> > > > is set
> > > >     > > > > > (conversion is
> > > >     > > > > >     > > > always
> > > >     > > > > >     > > >     > needed
> > > >     > > > > >     > > >     >     from magic 2 to magic 0). Not sure
> if
> > > the
> > > >     > > > > optimization
> > > >     > > > > > is
> > > >     > > > > >     > worth
> > > >     > > > > >     > > > the
> > > >     > > > > >     > > >     >     complexity though.
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     3. The referencing of the new
> version
> > > of
> > > >     > > > > >     > > > ProducerRequest/FetchRequest
> > > >     > > > > >     > > >     > is
> > > >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> > > > convention
> > > >     > > > starts
> > > >     > > > > at
> > > >     > > > > >     > version
> > > >     > > > > >     > > > at 0, I
> > > >     > > > > >     > > >     >     think the new version would be 3.
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     4. "If the magic byte on message is
> > 2,
> > > > the
> > > >     > broker
> > > >     > > > > > should use
> > > >     > > > > >     > > the
> > > >     > > > > >     > > >     > tombstone
> > > >     > > > > >     > > >     >     bit for log compaction." What about
> > > null
> > > > value?
> > > >     > > My
> > > >     > > > > >     > > understanding
> > > >     > > > > >     > > > is
> > > >     > > > > >     > > >     > that
> > > >     > > > > >     > > >     >     null value will be treated the same
> > as
> > > > setting
> > > >     > > the
> > > >     > > > > > tombstone
> > > >     > > > > >     > > bit.
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     5. For the migration path, it would
> > be
> > > > useful
> > > >     > to
> > > >     > > > > > describe the
> > > >     > > > > >     > > > down
> > > >     > > > > >     > > >     >     conversion path to consumers (i.e.,
> > > > brokers on
> > > >     > > > > message
> > > >     > > > > > format
> > > >     > > > > >     > > > 0.10.2
> > > >     > > > > >     > > >     > and
> > > >     > > > > >     > > >     >     consumers on older version).
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     Thanks,
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     Jun
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
> > > Michael
> > > >     > Pearce <
> > > >     > > > > >     > > > Michael.Pearce@ig.com
> > > >     > > > > >     > > >     > >
> > > >     > > > > >     > > >     >     wrote:
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >     > Hi All,
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     > We have been discussing in the
> > below
> > > > thread
> > > >     > and
> > > >     > > > > final
> > > >     > > > > >     > changes
> > > >     > > > > >     > > > have
> > > >     > > > > >     > > >     > been
> > > >     > > > > >     > > >     >     > made to the KIP wiki based on
> these
> > > >     > > discussions.
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     > We would now like to put to the
> > vote
> > > > the
> > > >     > > > following
> > > >     > > > > > KIP:
> > > >     > > > > >     > > >     >     > https://cwiki.apache.org/
> > > >     > > > > > confluence/display/KAFKA/KIP-
> > > >     > > > > >     > 87+-+
> > > >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     > This kip is for having a distinct
> > > > compaction
> > > >     > > > > > attribute
> > > >     > > > > >     > > > "tombstone"
> > > >     > > > > >     > > >     > flag
> > > >     > > > > >     > > >     >     > instead of relying on null value,
> > > > allowing
> > > >     > > > non-null
> > > >     > > > > > value
> > > >     > > > > >     > > > delete
> > > >     > > > > >     > > >     > messages.
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     > Many thanks,
> > > >     > > > > >     > > >     >     > Michael
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
> > > Pearce"
> > > > <
> > > >     > > > > >     > > Michael.Pearce@ig.com>
> > > >     > > > > >     > > >     > wrote:
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >     Hi Mayuresh,
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >     LGTM. Ive just made one small
> > > > adjustment
> > > >     > > > > > updating the
> > > >     > > > > >     > > wire
> > > >     > > > > >     > > >     > protocol to
> > > >     > > > > >     > > >     >     > show the magic byte bump.
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >     Do we think we're good to put
> > to
> > > a
> > > > vote?
> > > >     > Is
> > > >     > > > > > there any
> > > >     > > > > >     > > > other bits
> > > >     > > > > >     > > >     >     > needing discussion?
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >     Cheers
> > > >     > > > > >     > > >     >     >     Mike
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >     On 21/11/2016, 18:26,
> "Mayuresh
> > > > Gharat" <
> > > >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
> > > >     > > > > >     > > >     >     > wrote:
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >         Hi Michael,
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >         I have updated the
> > migration
> > > > section
> > > >     > of
> > > >     > > > the
> > > >     > > > > > KIP.
> > > >     > > > > >     > Can
> > > >     > > > > >     > > > you
> > > >     > > > > >     > > >     > please
> > > >     > > > > >     > > >     >     > take a look?
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >         Thanks,
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >         Mayuresh
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at
> > 9:07
> > > > AM,
> > > >     > > Mayuresh
> > > >     > > > > > Gharat <
> > > >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
> > > >     > > > > >     > > >     >     >         > wrote:
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >         > Hi Michael,
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         > That whilst sending
> > > > tombstone and
> > > >     > non
> > > >     > > > > null
> > > >     > > > > > value,
> > > >     > > > > >     > > the
> > > >     > > > > >     > > >     > consumer
> > > >     > > > > >     > > >     >     > can expect
> > > >     > > > > >     > > >     >     >         > only to receive the
> > > non-null
> > > >     > message
> > > >     > > > only
> > > >     > > > > > in step
> > > >     > > > > >     > > > (3) is
> > > >     > > > > >     > > >     > this
> > > >     > > > > >     > > >     >     > correct?
> > > >     > > > > >     > > >     >     >         > ---> I do agree with
> you
> > > > here.
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         > Becket, Ismael : can
> you
> > > guys
> > > >     > review
> > > >     > > > the
> > > >     > > > > >     > migration
> > > >     > > > > >     > > > plan
> > > >     > > > > >     > > >     > listed
> > > >     > > > > >     > > >     >     > above using
> > > >     > > > > >     > > >     >     >         > magic byte?
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         > Thanks,
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         > Mayuresh
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at
> > > 8:58
> > > > AM,
> > > >     > > > Michael
> > > >     > > > > > Pearce <
> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> > > >     > > > > >     > > >     >     >         > wrote:
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         >> Many thanks for this
> > > > Mayuresh. I
> > > >     > > don't
> > > >     > > > > > have any
> > > >     > > > > >     > > >     > objections.
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> I assume we should
> > state:
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> That whilst sending
> > > > tombstone and
> > > >     > > non
> > > >     > > > > null
> > > >     > > > > >     > value,
> > > >     > > > > >     > > > the
> > > >     > > > > >     > > >     > consumer
> > > >     > > > > >     > > >     >     > can expect
> > > >     > > > > >     > > >     >     >         >> only to receive the
> > > non-null
> > > >     > message
> > > >     > > > > only
> > > >     > > > > > in
> > > >     > > > > >     > step
> > > >     > > > > >     > > > (3) is
> > > >     > > > > >     > > >     > this
> > > >     > > > > >     > > >     >     > correct?
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Cheers
> > > >     > > > > >     > > >     >     >         >> Mike
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Sent using OWA for
> > iPhone
> > > >     > > > > >     > > >     >     >         >>
> > > > ______________________________
> > > >     > > > > __________
> > > >     > > > > >     > > >     >     >         >> From: Mayuresh Gharat
> <
> > > >     > > > > >     > gharatmayuresh15@gmail.com
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >     >     >         >> Sent: Thursday,
> November
> > > > 17, 2016
> > > >     > > > > 5:18:41
> > > >     > > > > > PM
> > > >     > > > > >     > > >     >     >         >> To:
> > dev@kafka.apache.org
> > > >     > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> > > > KIP-87 -
> > > >     > Add
> > > >     > > > > > Compaction
> > > >     > > > > >     > > > Tombstone
> > > >     > > > > >     > > >     > Flag
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Hi Ismael,
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Thanks for the
> > > explanation.
> > > >     > > > > >     > > >     >     >         >> Specially I like this
> > part
> > > > where
> > > >     > in
> > > >     > > > you
> > > >     > > > > >     > mentioned
> > > >     > > > > >     > > > we can
> > > >     > > > > >     > > >     > get
> > > >     > > > > >     > > >     >     > rid of the
> > > >     > > > > >     > > >     >     >         >> older null value
> support
> > > > for log
> > > >     > > > > > compaction
> > > >     > > > > >     > later
> > > >     > > > > >     > > > on,
> > > >     > > > > >     > > >     > here :
> > > >     > > > > >     > > >     >     >         >> We can't change
> > semantics
> > > > of the
> > > >     > > > message
> > > >     > > > > > format
> > > >     > > > > >     > > > without
> > > >     > > > > >     > > >     > having
> > > >     > > > > >     > > >     >     > a long
> > > >     > > > > >     > > >     >     >         >> transition period. And
> > we
> > > > can't
> > > >     > rely
> > > >     > > > > >     > > >     >     >         >> on people reading
> > > > documentation or
> > > >     > > > > acting
> > > >     > > > > > on a
> > > >     > > > > >     > > > warning for
> > > >     > > > > >     > > >     >     > something so
> > > >     > > > > >     > > >     >     >         >> fundamental. As such,
> my
> > > > take is
> > > >     > > that
> > > >     > > > we
> > > >     > > > > > need to
> > > >     > > > > >     > > > bump the
> > > >     > > > > >     > > >     > magic
> > > >     > > > > >     > > >     >     > byte. The
> > > >     > > > > >     > > >     >     >         >> good news is
> > > >     > > > > >     > > >     >     >         >> that we don't have to
> > > > support all
> > > >     > > > > versions
> > > >     > > > > >     > > forever.
> > > >     > > > > >     > > > We
> > > >     > > > > >     > > >     > have
> > > >     > > > > >     > > >     >     > said that we
> > > >     > > > > >     > > >     >     >         >> will support direct
> > > > upgrades for 2
> > > >     > > > > years.
> > > >     > > > > > That
> > > >     > > > > >     > > > means that
> > > >     > > > > >     > > >     >     > message format
> > > >     > > > > >     > > >     >     >         >> version n could, in
> > > theory,
> > > > be
> > > >     > > > removed 2
> > > >     > > > > > years
> > > >     > > > > >     > > > after the
> > > >     > > > > >     > > >     > it's
> > > >     > > > > >     > > >     >     > introduced.
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Just a heads up, I
> would
> > > > like to
> > > >     > > > mention
> > > >     > > > > > that
> > > >     > > > > >     > even
> > > >     > > > > >     > > > without
> > > >     > > > > >     > > >     >     > bumping magic
> > > >     > > > > >     > > >     >     >         >> byte, we will *NOT*
> > loose
> > > > zero
> > > >     > copy
> > > >     > > as
> > > >     > > > > in
> > > >     > > > > > the
> > > >     > > > > >     > > > client(x+1)
> > > >     > > > > >     > > >     > in my
> > > >     > > > > >     > > >     >     >         >> explanation
> > > >     > > > > >     > > >     >     >         >> above will convert
> > > > internally a
> > > >     > null
> > > >     > > > > > value to
> > > >     > > > > >     > > have a
> > > >     > > > > >     > > >     > tombstone
> > > >     > > > > >     > > >     >     > bit set and
> > > >     > > > > >     > > >     >     >         >> a tombstone bit set to
> > > have
> > > > a null
> > > >     > > > value
> > > >     > > > > >     > > > automatically
> > > >     > > > > >     > > >     >     > internally and by
> > > >     > > > > >     > > >     >     >         >> the time we move to
> > > version
> > > > (x+2),
> > > >     > > the
> > > >     > > > > > clients
> > > >     > > > > >     > > > would have
> > > >     > > > > >     > > >     >     > upgraded.
> > > >     > > > > >     > > >     >     >         >> Obviously if we
> support
> > a
> > > > request
> > > >     > > from
> > > >     > > > > >     > > consumer(x),
> > > >     > > > > >     > > > we
> > > >     > > > > >     > > >     > will
> > > >     > > > > >     > > >     >     > loose zero
> > > >     > > > > >     > > >     >     >         >> copy
> > > >     > > > > >     > > >     >     >         >> but that is the same
> > case
> > > > with
> > > >     > magic
> > > >     > > > > byte.
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> But if magic byte bump
> > > > makes life
> > > >     > > > easier
> > > >     > > > > > for
> > > >     > > > > >     > > > transition
> > > >     > > > > >     > > >     > for the
> > > >     > > > > >     > > >     >     > above
> > > >     > > > > >     > > >     >     >         >> reasons that you
> > > explained,
> > > > I am
> > > >     > OK
> > > >     > > > with
> > > >     > > > > > it
> > > >     > > > > >     > since
> > > >     > > > > >     > > > we are
> > > >     > > > > >     > > >     > going
> > > >     > > > > >     > > >     >     > to meet the
> > > >     > > > > >     > > >     >     >         >> end goal down the road
> > :)
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> On a side note can we
> > > > update the
> > > >     > doc
> > > >     > > > > here
> > > >     > > > > > on
> > > >     > > > > >     > magic
> > > >     > > > > >     > > > byte
> > > >     > > > > >     > > >     > to say
> > > >     > > > > >     > > >     >     > that "*it
> > > >     > > > > >     > > >     >     >         >> should be bumped
> > whenever
> > > > the
> > > >     > > message
> > > >     > > > > > format is
> > > >     > > > > >     > > > changed
> > > >     > > > > >     > > >     > or the
> > > >     > > > > >     > > >     >     >         >> interpretation of
> > message
> > > > format
> > > >     > > > (usage
> > > >     > > > > > of the
> > > >     > > > > >     > > > reserved
> > > >     > > > > >     > > >     > bits as
> > > >     > > > > >     > > >     >     > well) is
> > > >     > > > > >     > > >     >     >         >> changed*".
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Hi Michael,
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Here is the update
> plan
> > > > that we
> > > >     > > > > discussed
> > > >     > > > > >     > offline
> > > >     > > > > >     > > >     > yesterday :
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Currently the
> magic-byte
> > > > which
> > > >     > > > > > corresponds to
> > > >     > > > > >     > the
> > > >     > > > > >     > > >     >     > "message.format.version"
> > > >     > > > > >     > > >     >     >         >> is set to 1.
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> 1) On broker it will
> be
> > > set
> > > > to 1
> > > >     > > > > > initially.
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> 2) When a producer
> > client
> > > > sends a
> > > >     > > > > message
> > > >     > > > > > with
> > > >     > > > > >     > > > magic-byte
> > > >     > > > > >     > > >     > = 2,
> > > >     > > > > >     > > >     >     > since the
> > > >     > > > > >     > > >     >     >         >> broker is on
> magic-byte
> > =
> > > > 1, we
> > > >     > will
> > > >     > > > > down
> > > >     > > > > >     > convert
> > > >     > > > > >     > > > it,
> > > >     > > > > >     > > >     > which
> > > >     > > > > >     > > >     >     > means if the
> > > >     > > > > >     > > >     >     >         >> tombstone bit is set,
> > the
> > > > value
> > > >     > will
> > > >     > > > be
> > > >     > > > > > set to
> > > >     > > > > >     > > > null. A
> > > >     > > > > >     > > >     > consumer
> > > >     > > > > >     > > >     >     >         >> understanding
> > magic-byte =
> > > > 1, will
> > > >     > > > still
> > > >     > > > > > work
> > > >     > > > > >     > with
> > > >     > > > > >     > > > this. A
> > > >     > > > > >     > > >     >     > consumer
> > > >     > > > > >     > > >     >     >         >> working
> > > >     > > > > >     > > >     >     >         >> with magic-byte =2
> will
> > > > also be
> > > >     > able
> > > >     > > > to
> > > >     > > > > >     > understand
> > > >     > > > > >     > > > this,
> > > >     > > > > >     > > >     > since
> > > >     > > > > >     > > >     >     > it
> > > >     > > > > >     > > >     >     >         >> understands the
> > tombstone.
> > > >     > > > > >     > > >     >     >         >> Now there is still the
> > > > question of
> > > >     > > > > > supporting a
> > > >     > > > > >     > > >     > non-tombstone
> > > >     > > > > >     > > >     >     > and null
> > > >     > > > > >     > > >     >     >         >> value from producer
> > client
> > > > with
> > > >     > > > > > magic-byte = 2.*
> > > >     > > > > >     > > (I
> > > >     > > > > >     > > > am
> > > >     > > > > >     > > >     > not sure
> > > >     > > > > >     > > >     >     > if we
> > > >     > > > > >     > > >     >     >         >> should support this.
> > > > Ismael/Becket
> > > >     > > can
> > > >     > > > > > comment
> > > >     > > > > >     > > > here)*
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> 3) When almost all the
> > > > clients
> > > >     > have
> > > >     > > > > > upgraded,
> > > >     > > > > >     > the
> > > >     > > > > >     > > >     >     > message.format.version
> > > >     > > > > >     > > >     >     >         >> on
> > > >     > > > > >     > > >     >     >         >> the broker can be
> > changed
> > > > to 2,
> > > >     > > where
> > > >     > > > in
> > > >     > > > > > the
> > > >     > > > > >     > down
> > > >     > > > > >     > > >     > conversion in
> > > >     > > > > >     > > >     >     > the above
> > > >     > > > > >     > > >     >     >         >> step will not happen.
> If
> > > at
> > > > this
> > > >     > > point
> > > >     > > > > we
> > > >     > > > > > get a
> > > >     > > > > >     > > > consumer
> > > >     > > > > >     > > >     >     > request from a
> > > >     > > > > >     > > >     >     >         >> older consumer, we
> might
> > > > have to
> > > >     > > down
> > > >     > > > > > convert
> > > >     > > > > >     > > where
> > > >     > > > > >     > > > in we
> > > >     > > > > >     > > >     > loose
> > > >     > > > > >     > > >     >     > zero copy,
> > > >     > > > > >     > > >     >     >         >> but these cases should
> > be
> > > > rare.
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Becket can you review
> > this
> > > > plan
> > > >     > and
> > > >     > > > add
> > > >     > > > > > more
> > > >     > > > > >     > > > details if I
> > > >     > > > > >     > > >     > have
> > > >     > > > > >     > > >     >     >         >> missed/wronged
> > something,
> > > > before
> > > >     > we
> > > >     > > > put
> > > >     > > > > > it on
> > > >     > > > > >     > KIP.
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Thanks,
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> Mayuresh
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016
> at
> > > > 11:07 PM,
> > > >     > > > > Michael
> > > >     > > > > >     > Pearce <
> > > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> > > >     > > > > >     > > >     >     >         >> wrote:
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> > Thanks guys, for
> > > > discussing this
> > > >     > > > > > offline and
> > > >     > > > > >     > > > getting
> > > >     > > > > >     > > >     > some
> > > >     > > > > >     > > >     >     > consensus.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > So its clear for
> > myself
> > > > and
> > > >     > others
> > > >     > > > > what
> > > >     > > > > > is
> > > >     > > > > >     > > > proposed now
> > > >     > > > > >     > > >     > (i
> > > >     > > > > >     > > >     >     > think i
> > > >     > > > > >     > > >     >     >         >> > understand, but want
> > to
> > > > make
> > > >     > sure)
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > Could i ask either
> > > > directly
> > > >     > update
> > > >     > > > the
> > > >     > > > > > kip to
> > > >     > > > > >     > > > detail the
> > > >     > > > > >     > > >     >     > migration
> > > >     > > > > >     > > >     >     >         >> > strategy, or
> > (re-)state
> > > > your
> > > >     > > offline
> > > >     > > > > > discussed
> > > >     > > > > >     > > and
> > > >     > > > > >     > > >     > agreed
> > > >     > > > > >     > > >     >     > migration
> > > >     > > > > >     > > >     >     >         >> > strategy based on a
> > > magic
> > > > byte
> > > >     > is
> > > >     > > in
> > > >     > > > > > this
> > > >     > > > > >     > > thread.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > The main original
> > driver
> > > > for the
> > > >     > > KIP
> > > >     > > > > > was to
> > > >     > > > > >     > > > support
> > > >     > > > > >     > > >     >     > compaction where
> > > >     > > > > >     > > >     >     >         >> value
> > > >     > > > > >     > > >     >     >         >> > isn't null, based
> off
> > > the
> > > >     > > > discussions
> > > >     > > > > on
> > > >     > > > > >     > KIP-82
> > > >     > > > > >     > > > thread.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > We should be able to
> > > > support
> > > >     > > > > > non-tombstone +
> > > >     > > > > >     > > null
> > > >     > > > > >     > > > value
> > > >     > > > > >     > > >     > by the
> > > >     > > > > >     > > >     >     >         >> completion
> > > >     > > > > >     > > >     >     >         >> > of the KIP, as we
> > noted
> > > > when
> > > >     > > > > discussing
> > > >     > > > > > this
> > > >     > > > > >     > > kip,
> > > >     > > > > >     > > > having
> > > >     > > > > >     > > >     >     > logic based on
> > > >     > > > > >     > > >     >     >         >> a
> > > >     > > > > >     > > >     >     >         >> > null value isn't
> very
> > > > clean and
> > > >     > > also
> > > >     > > > > > separates
> > > >     > > > > >     > > the
> > > >     > > > > >     > > >     > concerns.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > As discussed already
> > > > though we
> > > >     > can
> > > >     > > > > > split this
> > > >     > > > > >     > > into
> > > >     > > > > >     > > >     > KIP-87a
> > > >     > > > > >     > > >     >     > and KIP-87b
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > Where we look to
> > deliver
> > > > KIP-87a
> > > >     > > on
> > > >     > > > a
> > > >     > > > > >     > compacted
> > > >     > > > > >     > > > topic
> > > >     > > > > >     > > >     > (to
> > > >     > > > > >     > > >     >     > address the
> > > >     > > > > >     > > >     >     >         >> > immediate issues)
> > > >     > > > > >     > > >     >     >         >> > * tombstone + null
> > value
> > > >     > > > > >     > > >     >     >         >> > * tombstone +
> non-null
> > > > value
> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
> > > > non-null value
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > Then we can discuss
> > once
> > > > KIP-87a
> > > >     > > is
> > > >     > > > > > completed
> > > >     > > > > >     > > > options
> > > >     > > > > >     > > >     > later
> > > >     > > > > >     > > >     >     > and how we
> > > >     > > > > >     > > >     >     >         >> > support the second
> > part
> > > > KIP-87b
> > > >     > to
> > > >     > > > > > deliver:
> > > >     > > > > >     > > >     >     >         >> > * non-tombstone +
> null
> > > > value
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > Cheers
> > > >     > > > > >     > > >     >     >         >> > Mike
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> >
> > > > ______________________________
> > > >     > > > > > __________
> > > >     > > > > >     > > >     >     >         >> > From: Becket Qin <
> > > >     > > > > becket.qin@gmail.com>
> > > >     > > > > >     > > >     >     >         >> > Sent: Thursday,
> > November
> > > > 17,
> > > >     > 2016
> > > >     > > > 1:43
> > > >     > > > > > AM
> > > >     > > > > >     > > >     >     >         >> > To:
> > > dev@kafka.apache.org
> > > >     > > > > >     > > >     >     >         >> > Subject: Re:
> [DISCUSS]
> > > > KIP-87 -
> > > >     > > Add
> > > >     > > > > > Compaction
> > > >     > > > > >     > > >     > Tombstone Flag
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and I
> > had
> > > > an
> > > >     > > offline
> > > >     > > > > >     > discussion,
> > > >     > > > > >     > > > and
> > > >     > > > > >     > > >     > following
> > > >     > > > > >     > > >     >     > is a brief
> > > >     > > > > >     > > >     >     >         >> > summary.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > 1. We agreed that
> not
> > > > bumping up
> > > >     > > > magic
> > > >     > > > > > value
> > > >     > > > > >     > may
> > > >     > > > > >     > > > result
> > > >     > > > > >     > > >     > in
> > > >     > > > > >     > > >     >     > losing zero
> > > >     > > > > >     > > >     >     >         >> copy
> > > >     > > > > >     > > >     >     >         >> > during migration.
> > > >     > > > > >     > > >     >     >         >> > 2. Given that
> bumping
> > up
> > > > magic
> > > >     > > value
> > > >     > > > > is
> > > >     > > > > > almost
> > > >     > > > > >     > > > free and
> > > >     > > > > >     > > >     > has
> > > >     > > > > >     > > >     >     > benefit of
> > > >     > > > > >     > > >     >     >         >> > avoiding potential
> > > > performance
> > > >     > > > issue.
> > > >     > > > > > It is
> > > >     > > > > >     > > > probably
> > > >     > > > > >     > > >     > worth
> > > >     > > > > >     > > >     >     > doing.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > One issue we still
> > need
> > > > to think
> > > >     > > > about
> > > >     > > > > > is
> > > >     > > > > >     > > whether
> > > >     > > > > >     > > > we
> > > >     > > > > >     > > >     > want to
> > > >     > > > > >     > > >     >     > support a
> > > >     > > > > >     > > >     >     >         >> > non-tombstone
> message
> > > > with null
> > > >     > > > value.
> > > >     > > > > >     > > >     >     >         >> > Currently it is not
> > > > supported by
> > > >     > > > > Kafka.
> > > >     > > > > > If we
> > > >     > > > > >     > > > allow a
> > > >     > > > > >     > > >     >     > non-tombstone null
> > > >     > > > > >     > > >     >     >         >> > value message to
> exist
> > > > after
> > > >     > > KIP-87.
> > > >     > > > > The
> > > >     > > > > >     > problem
> > > >     > > > > >     > > > is
> > > >     > > > > >     > > >     > that such
> > > >     > > > > >     > > >     >     > message
> > > >     > > > > >     > > >     >     >         >> will
> > > >     > > > > >     > > >     >     >         >> > not be supported by
> > the
> > > >     > consumers
> > > >     > > > > prior
> > > >     > > > > > to
> > > >     > > > > >     > > KIP-87.
> > > >     > > > > >     > > >     > Because a
> > > >     > > > > >     > > >     >     > null value
> > > >     > > > > >     > > >     >     >         >> > will always be
> > > > interpreted to a
> > > >     > > > > > tombstone.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > One option is that
> we
> > > > keep the
> > > >     > > > current
> > > >     > > > > > way,
> > > >     > > > > >     > i.e.
> > > >     > > > > >     > > > do not
> > > >     > > > > >     > > >     >     > support such
> > > >     > > > > >     > > >     >     >         >> > message. It would be
> > > good
> > > > to
> > > >     > know
> > > >     > > if
> > > >     > > > > > there is
> > > >     > > > > >     > a
> > > >     > > > > >     > > >     > concrete use
> > > >     > > > > >     > > >     >     > case for
> > > >     > > > > >     > > >     >     >         >> such
> > > >     > > > > >     > > >     >     >         >> > message. If there is
> > > not,
> > > > we can
> > > >     > > > > > probably just
> > > >     > > > > >     > > not
> > > >     > > > > >     > > >     > support it.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > Thanks,
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > JIangjie (Becket)
> Qin
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016
> > at
> > > > 1:28 PM,
> > > >     > > > > > Mayuresh
> > > >     > > > > >     > > Gharat <
> > > >     > > > > >     > > >     >     >         >> >
> > > > gharatmayuresh15@gmail.com
> > > >     > > > > >     > > >     >     >         >> > > wrote:
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >> > > Hi Ismael,
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > > This is something
> I
> > > can
> > > > think
> > > >     > of
> > > >     > > > for
> > > >     > > > > >     > migration
> > > >     > > > > >     > > > plan:
> > > >     > > > > >     > > >     >     >         >> > > So the migration
> > plan
> > > > can look
> > > >     > > > > > something
> > > >     > > > > >     > like
> > > >     > > > > >     > > > this,
> > > >     > > > > >     > > >     > with up
> > > >     > > > > >     > > >     >     >         >> conversion :
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > > 1) Currently lets
> > say
> > > > we have
> > > >     > > > Broker
> > > >     > > > > > at
> > > >     > > > > >     > > version
> > > >     > > > > >     > > > x.
> > > >     > > > > >     > > >     >     >         >> > > 2) Currently we
> have
> > > > clients
> > > >     > at
> > > >     > > > > > version x.
> > > >     > > > > >     > > >     >     >         >> > > 3) a) We move the
> > > > version to
> > > >     > > > > > Broker(x+1) :
> > > >     > > > > >     > > > supports
> > > >     > > > > >     > > >     > both
> > > >     > > > > >     > > >     >     > tombstone and
> > > >     > > > > >     > > >     >     >         >> > null
> > > >     > > > > >     > > >     >     >         >> > > for log
> compaction.
> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
> > the
> > > > client
> > > >     > to
> > > >     > > > > > version
> > > >     > > > > >     > > > client(x+1) :
> > > >     > > > > >     > > >     > if in
> > > >     > > > > >     > > >     >     > the
> > > >     > > > > >     > > >     >     >         >> producer
> > > >     > > > > >     > > >     >     >         >> > > client(x+1) the
> > value
> > > > is set
> > > >     > to
> > > >     > > > > null,
> > > >     > > > > > we
> > > >     > > > > >     > will
> > > >     > > > > >     > > >     > automatically
> > > >     > > > > >     > > >     >     > set the
> > > >     > > > > >     > > >     >     >         >> > > Tombstone bit
> > > > internally. If
> > > >     > the
> > > >     > > > > > producer
> > > >     > > > > >     > > > client(x+1)
> > > >     > > > > >     > > >     > sets
> > > >     > > > > >     > > >     >     > the
> > > >     > > > > >     > > >     >     >         >> tombstone
> > > >     > > > > >     > > >     >     >         >> > > itself, well and
> > good.
> > > > For
> > > >     > > > producer
> > > >     > > > > >     > client(x),
> > > >     > > > > >     > > > the
> > > >     > > > > >     > > >     > broker
> > > >     > > > > >     > > >     >     > will up
> > > >     > > > > >     > > >     >     >         >> convert
> > > >     > > > > >     > > >     >     >         >> > > to have the
> > tombstone
> > > > bit.
> > > >     > > > > > Broker(x+1) is
> > > >     > > > > >     > > > supporting
> > > >     > > > > >     > > >     > both.
> > > >     > > > > >     > > >     >     > Consumer
> > > >     > > > > >     > > >     >     >         >> > > client(x+1) will
> be
> > > > aware of
> > > >     > > this
> > > >     > > > > and
> > > >     > > > > > should
> > > >     > > > > >     > > be
> > > >     > > > > >     > > > able
> > > >     > > > > >     > > >     > to
> > > >     > > > > >     > > >     >     > handle this.
> > > >     > > > > >     > > >     >     >         >> For
> > > >     > > > > >     > > >     >     >         >> > > consumer client(x)
> > we
> > > > will
> > > >     > down
> > > >     > > > > > convert the
> > > >     > > > > >     > > > message
> > > >     > > > > >     > > >     > on the
> > > >     > > > > >     > > >     >     > broker
> > > >     > > > > >     > > >     >     >         >> side.
> > > >     > > > > >     > > >     >     >         >> > >     c) At this
> point
> > > we
> > > > will
> > > >     > > have
> > > >     > > > to
> > > >     > > > > >     > specify a
> > > >     > > > > >     > > >     > warning or
> > > >     > > > > >     > > >     >     > clearly
> > > >     > > > > >     > > >     >     >         >> specify
> > > >     > > > > >     > > >     >     >         >> > > in docs that this
> > > > behavior is
> > > >     > > > about
> > > >     > > > > > to be
> > > >     > > > > >     > > > changed for
> > > >     > > > > >     > > >     > log
> > > >     > > > > >     > > >     >     > compaction.
> > > >     > > > > >     > > >     >     >         >> > > 4) a) In next
> > release
> > > > of the
> > > >     > > > > > Broker(x+2), we
> > > >     > > > > >     > > > say that
> > > >     > > > > >     > > >     > only
> > > >     > > > > >     > > >     >     > Tombstone
> > > >     > > > > >     > > >     >     >         >> is
> > > >     > > > > >     > > >     >     >         >> > > used for log
> > > compaction
> > > > on the
> > > >     > > > > Broker
> > > >     > > > > > side.
> > > >     > > > > >     > > >     > Clients(x+1)
> > > >     > > > > >     > > >     >     > still is
> > > >     > > > > >     > > >     >     >         >> > > supported.
> > > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
> > the
> > > > client
> > > >     > to
> > > >     > > > > > version
> > > >     > > > > >     > > > client(x+2) :
> > > >     > > > > >     > > >     > if
> > > >     > > > > >     > > >     >     > value is set
> > > >     > > > > >     > > >     >     >         >> to
> > > >     > > > > >     > > >     >     >         >> > > null, tombstone
> will
> > > > not be
> > > >     > set
> > > >     > > > > >     > automatically.
> > > >     > > > > >     > > > The
> > > >     > > > > >     > > >     > client
> > > >     > > > > >     > > >     >     > will have to
> > > >     > > > > >     > > >     >     >         >> > call
> > > >     > > > > >     > > >     >     >         >> > > setTombstone() to
> > > > actually set
> > > >     > > the
> > > >     > > > > >     > tombstone.
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > > We should compare
> > this
> > > >     > migration
> > > >     > > > > plan
> > > >     > > > > > with
> > > >     > > > > >     > the
> > > >     > > > > >     > > >     > migration
> > > >     > > > > >     > > >     >     > plan for
> > > >     > > > > >     > > >     >     >         >> magic
> > > >     > > > > >     > > >     >     >         >> > > byte bump and do
> > > > whatever
> > > >     > looks
> > > >     > > > > good.
> > > >     > > > > >     > > >     >     >         >> > > I am just worried
> > that
> > > > if we
> > > >     > go
> > > >     > > > down
> > > >     > > > > > magic
> > > >     > > > > >     > > byte
> > > >     > > > > >     > > > route,
> > > >     > > > > >     > > >     >     > unless I am
> > > >     > > > > >     > > >     >     >         >> > missing
> > > >     > > > > >     > > >     >     >         >> > > something, it
> sounds
> > > > like
> > > >     > kafka
> > > >     > > > will
> > > >     > > > > > be
> > > >     > > > > >     > stuck
> > > >     > > > > >     > > > with
> > > >     > > > > >     > > >     >     > supporting both
> > > >     > > > > >     > > >     >     >         >> null
> > > >     > > > > >     > > >     >     >         >> > > value and
> tombstone
> > > bit
> > > > for
> > > >     > log
> > > >     > > > > > compaction
> > > >     > > > > >     > for
> > > >     > > > > >     > > > life
> > > >     > > > > >     > > >     > long,
> > > >     > > > > >     > > >     >     > which does
> > > >     > > > > >     > > >     >     >         >> not
> > > >     > > > > >     > > >     >     >         >> > > look like a good
> end
> > > > state.
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > > Thanks,
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > > Mayuresh
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16,
> 2016
> > > at
> > > > 9:32
> > > >     > AM,
> > > >     > > > > > Mayuresh
> > > >     > > > > >     > > > Gharat <
> > > >     > > > > >     > > >     >     >         >> > >
> > > > gharatmayuresh15@gmail.com
> > > >     > > > > >     > > >     >     >         >> > > > wrote:
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > That's a very
> good
> > > > point
> > > >     > > which I
> > > >     > > > > > might
> > > >     > > > > >     > have
> > > >     > > > > >     > > > not
> > > >     > > > > >     > > >     >     > considered earlier.
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > Here is a plan
> > that
> > > I
> > > > can
> > > >     > > think
> > > >     > > > > of:
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > Stage 1) The
> > broker
> > > > from now
> > > >     > > on,
> > > >     > > > > up
> > > >     > > > > >     > converts
> > > >     > > > > >     > > > the
> > > >     > > > > >     > > >     > message
> > > >     > > > > >     > > >     >     > to have the
> > > >     > > > > >     > > >     >     >         >> > > > tombstone
> marker.
> > > The
> > > > log
> > > >     > > > > compaction
> > > >     > > > > >     > thread
> > > >     > > > > >     > > > does log
> > > >     > > > > >     > > >     >     > compaction
> > > >     > > > > >     > > >     >     >         >> based
> > > >     > > > > >     > > >     >     >         >> > on
> > > >     > > > > >     > > >     >     >         >> > > > both null and
> > > > tombstone
> > > >     > > marker.
> > > >     > > > > > This is
> > > >     > > > > >     > our
> > > >     > > > > >     > > >     > transition
> > > >     > > > > >     > > >     >     > period.
> > > >     > > > > >     > > >     >     >         >> > > > Stage 2) The
> next
> > > > release we
> > > >     > > > only
> > > >     > > > > > say that
> > > >     > > > > >     > > log
> > > >     > > > > >     > > >     > compaction
> > > >     > > > > >     > > >     >     > is based
> > > >     > > > > >     > > >     >     >         >> on
> > > >     > > > > >     > > >     >     >         >> > > > tombstone
> marker.
> > > > (Open
> > > >     > source
> > > >     > > > > > kafka makes
> > > >     > > > > >     > > > this as a
> > > >     > > > > >     > > >     >     > policy). By
> > > >     > > > > >     > > >     >     >         >> this
> > > >     > > > > >     > > >     >     >         >> > > time,
> > > >     > > > > >     > > >     >     >         >> > > > the organization
> > > > which is
> > > >     > > moving
> > > >     > > > > to
> > > >     > > > > > this
> > > >     > > > > >     > > > release
> > > >     > > > > >     > > >     > will be
> > > >     > > > > >     > > >     >     > sure that
> > > >     > > > > >     > > >     >     >         >> they
> > > >     > > > > >     > > >     >     >         >> > > > have gone
> through
> > > the
> > > > entire
> > > >     > > > > > transition
> > > >     > > > > >     > > > period.
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > My only goal of
> > > doing
> > > > this
> > > >     > is
> > > >     > > > that
> > > >     > > > > > Kafka
> > > >     > > > > >     > > > clearly
> > > >     > > > > >     > > >     >     > specifies the end
> > > >     > > > > >     > > >     >     >         >> > state
> > > >     > > > > >     > > >     >     >         >> > > > about what log
> > > > compaction
> > > >     > > means
> > > >     > > > > (is
> > > >     > > > > > it
> > > >     > > > > >     > null
> > > >     > > > > >     > > > value
> > > >     > > > > >     > > >     > or a
> > > >     > > > > >     > > >     >     > tombstone
> > > >     > > > > >     > > >     >     >         >> > marker,
> > > >     > > > > >     > > >     >     >         >> > > > but not both).
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > What do you
> think?
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > Thanks,
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh
> > > >     > > > > >     > > >     >     >         >> > > > .
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16,
> > 2016
> > > > at 9:17
> > > >     > > AM,
> > > >     > > > > > Ismael
> > > >     > > > > >     > > Juma <
> > > >     > > > > >     > > >     >     > ismael@juma.me.uk>
> > > >     > > > > >     > > >     >     >         >> > wrote:
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > >> One comment
> > below.
> > > >     > > > > >     > > >     >     >         >> > > >>
> > > >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16,
> > > 2016
> > > > at
> > > >     > 5:08
> > > >     > > > PM,
> > > >     > > > > > Mayuresh
> > > >     > > > > >     > > > Gharat <
> > > >     > > > > >     > > >     >     >         >> > > >>
> > > > gharatmayuresh15@gmail.com
> > > >     > > > > >     > > >     >     >         >> > > >> > wrote:
> > > >     > > > > >     > > >     >     >         >> > > >>
> > > >     > > > > >     > > >     >     >         >> > > >> >    - If we
> > don't
> > > > bump up
> > > >     > > the
> > > >     > > > > > magic
> > > >     > > > > >     > byte,
> > > >     > > > > >     > > > on the
> > > >     > > > > >     > > >     > broker
> > > >     > > > > >     > > >     >     > side, the
> > > >     > > > > >     > > >     >     >         >> > > broker
> > > >     > > > > >     > > >     >     >         >> > > >> >    will
> always
> > > > have to
> > > >     > look
> > > >     > > > at
> > > >     > > > > > both
> > > >     > > > > >     > > > tombstone
> > > >     > > > > >     > > >     > bit and
> > > >     > > > > >     > > >     >     > the value
> > > >     > > > > >     > > >     >     >         >> when
> > > >     > > > > >     > > >     >     >         >> > > do
> > > >     > > > > >     > > >     >     >         >> > > >> the
> > > >     > > > > >     > > >     >     >         >> > > >> >
> compaction.
> > > > Assuming
> > > >     > we
> > > >     > > do
> > > >     > > > > > not bump
> > > >     > > > > >     > up
> > > >     > > > > >     > > > the
> > > >     > > > > >     > > >     > magic
> > > >     > > > > >     > > >     >     > byte,
> > > >     > > > > >     > > >     >     >         >> > > >> >    imagine
> the
> > > > broker
> > > >     > sees
> > > >     > > a
> > > >     > > > > > message
> > > >     > > > > >     > > which
> > > >     > > > > >     > > > does
> > > >     > > > > >     > > >     > not
> > > >     > > > > >     > > >     >     > have a
> > > >     > > > > >     > > >     >     >         >> tombstone
> > > >     > > > > >     > > >     >     >         >> > > bit
> > > >     > > > > >     > > >     >     >         >> > > >> >    set. The
> > > broker
> > > > does
> > > >     > not
> > > >     > > > > know
> > > >     > > > > > when
> > > >     > > > > >     > the
> > > >     > > > > >     > > >     > message was
> > > >     > > > > >     > > >     >     > produced
> > > >     > > > > >     > > >     >     >         >> (i.e.
> > > >     > > > > >     > > >     >     >         >> > > >> > whether
> > > >     > > > > >     > > >     >     >         >> > > >> >    the
> message
> > > has
> > > > been
> > > >     > up
> > > >     > > > > > converted or
> > > >     > > > > >     > > > not), it
> > > >     > > > > >     > > >     > has
> > > >     > > > > >     > > >     >     > to take a
> > > >     > > > > >     > > >     >     >         >> > further
> > > >     > > > > >     > > >     >     >         >> > > >> > look at
> > > >     > > > > >     > > >     >     >         >> > > >> >    the value
> to
> > > > see if it
> > > >     > > is
> > > >     > > > > > null or
> > > >     > > > > >     > not
> > > >     > > > > >     > > in
> > > >     > > > > >     > > >     > order to
> > > >     > > > > >     > > >     >     > determine
> > > >     > > > > >     > > >     >     >         >> if it
> > > >     > > > > >     > > >     >     >         >> > > is
> > > >     > > > > >     > > >     >     >         >> > > >> a
> > > >     > > > > >     > > >     >     >         >> > > >> >    tombstone.
> > The
> > > > same
> > > >     > > logic
> > > >     > > > > has
> > > >     > > > > > to be
> > > >     > > > > >     > > put
> > > >     > > > > >     > > > on the
> > > >     > > > > >     > > >     >     > consumer as
> > > >     > > > > >     > > >     >     >         >> well
> > > >     > > > > >     > > >     >     >         >> > > >> because
> > > >     > > > > >     > > >     >     >         >> > > >> > the
> > > >     > > > > >     > > >     >     >         >> > > >> >    consumer
> > does
> > > > not know
> > > >     > > if
> > > >     > > > > the
> > > >     > > > > >     > message
> > > >     > > > > >     > > > has
> > > >     > > > > >     > > >     > been up
> > > >     > > > > >     > > >     >     > converted or
> > > >     > > > > >     > > >     >     >         >> > not.
> > > >     > > > > >     > > >     >     >         >> > > >> >       - If we
> > > > upconvert
> > > >     > > while
> > > >     > > > > >     > appending,
> > > >     > > > > >     > > > this is
> > > >     > > > > >     > > >     > not
> > > >     > > > > >     > > >     >     > the case,
> > > >     > > > > >     > > >     >     >         >> > right?
> > > >     > > > > >     > > >     >     >         >> > > >>
> > > >     > > > > >     > > >     >     >         >> > > >>
> > > >     > > > > >     > > >     >     >         >> > > >> If I understand
> > you
> > > >     > > correctly,
> > > >     > > > > > this is
> > > >     > > > > >     > not
> > > >     > > > > >     > > >     > sufficient
> > > >     > > > > >     > > >     >     > because the
> > > >     > > > > >     > > >     >     >         >> log
> > > >     > > > > >     > > >     >     >         >> > > may
> > > >     > > > > >     > > >     >     >         >> > > >> have messages
> > > > appended
> > > >     > before
> > > >     > > > it
> > > >     > > > > > was
> > > >     > > > > >     > > > upgraded to
> > > >     > > > > >     > > >     > include
> > > >     > > > > >     > > >     >     > KIP-87.
> > > >     > > > > >     > > >     >     >         >> > > >>
> > > >     > > > > >     > > >     >     >         >> > > >> Ismael
> > > >     > > > > >     > > >     >     >         >> > > >>
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > > > --
> > > >     > > > > >     > > >     >     >         >> > > > -Regards,
> > > >     > > > > >     > > >     >     >         >> > > > Mayuresh R.
> Gharat
> > > >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
> > > >     > > > > >     > > >     >     >         >> > > >
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > > --
> > > >     > > > > >     > > >     >     >         >> > > -Regards,
> > > >     > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > > >     > > > > >     > > >     >     >         >> > > (862) 250-7125
> > > >     > > > > >     > > >     >     >         >> > >
> > > >     > > > > >     > > >     >     >         >> > The information
> > > contained
> > > > in
> > > >     > this
> > > >     > > > > email
> > > >     > > > > > is
> > > >     > > > > >     > > > strictly
> > > >     > > > > >     > > >     >     > confidential and for
> > > >     > > > > >     > > >     >     >         >> > the use of the
> > addressee
> > > > only,
> > > >     > > > unless
> > > >     > > > > >     > otherwise
> > > >     > > > > >     > > >     > indicated. If
> > > >     > > > > >     > > >     >     > you are
> > > >     > > > > >     > > >     >     >         >> not
> > > >     > > > > >     > > >     >     >         >> > the intended
> > recipient,
> > > > please
> > > >     > do
> > > >     > > > not
> > > >     > > > > > read,
> > > >     > > > > >     > > copy,
> > > >     > > > > >     > > > use or
> > > >     > > > > >     > > >     >     > disclose to
> > > >     > > > > >     > > >     >     >         >> others
> > > >     > > > > >     > > >     >     >         >> > this message or any
> > > > attachment.
> > > >     > > > Please
> > > >     > > > > > also
> > > >     > > > > >     > > > notify the
> > > >     > > > > >     > > >     > sender
> > > >     > > > > >     > > >     >     > by
> > > >     > > > > >     > > >     >     >         >> replying
> > > >     > > > > >     > > >     >     >         >> > to this email or by
> > > > telephone
> > > >     > > > (+44(020
> > > >     > > > > > 7896
> > > >     > > > > >     > > 0011)
> > > >     > > > > >     > > > and
> > > >     > > > > >     > > >     > then
> > > >     > > > > >     > > >     >     > delete the
> > > >     > > > > >     > > >     >     >         >> email
> > > >     > > > > >     > > >     >     >         >> > and any copies of
> it.
> > > > Opinions,
> > > >     > > > > > conclusion
> > > >     > > > > >     > (etc)
> > > >     > > > > >     > > > that
> > > >     > > > > >     > > >     > do not
> > > >     > > > > >     > > >     >     > relate to
> > > >     > > > > >     > > >     >     >         >> the
> > > >     > > > > >     > > >     >     >         >> > official business of
> > > this
> > > >     > company
> > > >     > > > > shall
> > > >     > > > > > be
> > > >     > > > > >     > > > understood as
> > > >     > > > > >     > > >     >     > neither given
> > > >     > > > > >     > > >     >     >         >> nor
> > > >     > > > > >     > > >     >     >         >> > endorsed by it. IG
> is
> > a
> > > > trading
> > > >     > > name
> > > >     > > > > of
> > > >     > > > > > IG
> > > >     > > > > >     > > Markets
> > > >     > > > > >     > > >     > Limited (a
> > > >     > > > > >     > > >     >     > company
> > > >     > > > > >     > > >     >     >         >> > registered in
> England
> > > and
> > > > Wales,
> > > >     > > > > company
> > > >     > > > > >     > number
> > > >     > > > > >     > > >     > 04008957) and
> > > >     > > > > >     > > >     >     > IG Index
> > > >     > > > > >     > > >     >     >         >> > Limited (a company
> > > > registered in
> > > >     > > > > > England and
> > > >     > > > > >     > > > Wales,
> > > >     > > > > >     > > >     > company
> > > >     > > > > >     > > >     >     > number
> > > >     > > > > >     > > >     >     >         >> > 01190902).
> Registered
> > > > address at
> > > >     > > > > Cannon
> > > >     > > > > > Bridge
> > > >     > > > > >     > > > House, 25
> > > >     > > > > >     > > >     >     > Dowgate Hill,
> > > >     > > > > >     > > >     >     >         >> > London EC4R 2YA.
> Both
> > IG
> > > > Markets
> > > >     > > > > Limited
> > > >     > > > > >     > > (register
> > > >     > > > > >     > > >     > number
> > > >     > > > > >     > > >     >     > 195355) and IG
> > > >     > > > > >     > > >     >     >         >> > Index Limited
> > (register
> > > > number
> > > >     > > > 114059)
> > > >     > > > > > are
> > > >     > > > > >     > > > authorised
> > > >     > > > > >     > > >     > and
> > > >     > > > > >     > > >     >     > regulated by
> > > >     > > > > >     > > >     >     >         >> the
> > > >     > > > > >     > > >     >     >         >> > Financial Conduct
> > > > Authority.
> > > >     > > > > >     > > >     >     >         >> >
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >> --
> > > >     > > > > >     > > >     >     >         >> -Regards,
> > > >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
> > > >     > > > > >     > > >     >     >         >> (862) 250-7125
> > > >     > > > > >     > > >     >     >         >> The information
> > contained
> > > > in this
> > > >     > > > email
> > > >     > > > > is
> > > >     > > > > >     > > strictly
> > > >     > > > > >     > > >     >     > confidential and for
> > > >     > > > > >     > > >     >     >         >> the use of the
> addressee
> > > > only,
> > > >     > > unless
> > > >     > > > > > otherwise
> > > >     > > > > >     > > >     > indicated. If
> > > >     > > > > >     > > >     >     > you are not
> > > >     > > > > >     > > >     >     >         >> the intended
> recipient,
> > > > please do
> > > >     > > not
> > > >     > > > > > read,
> > > >     > > > > >     > copy,
> > > >     > > > > >     > > > use or
> > > >     > > > > >     > > >     >     > disclose to others
> > > >     > > > > >     > > >     >     >         >> this message or any
> > > > attachment.
> > > >     > > Please
> > > >     > > > > > also
> > > >     > > > > >     > notify
> > > >     > > > > >     > > > the
> > > >     > > > > >     > > >     > sender
> > > >     > > > > >     > > >     >     > by replying
> > > >     > > > > >     > > >     >     >         >> to this email or by
> > > > telephone
> > > >     > > (+44(020
> > > >     > > > > > 7896
> > > >     > > > > >     > 0011)
> > > >     > > > > >     > > > and then
> > > >     > > > > >     > > >     >     > delete the email
> > > >     > > > > >     > > >     >     >         >> and any copies of it.
> > > > Opinions,
> > > >     > > > > > conclusion (etc)
> > > >     > > > > >     > > > that do
> > > >     > > > > >     > > >     > not
> > > >     > > > > >     > > >     >     > relate to the
> > > >     > > > > >     > > >     >     >         >> official business of
> > this
> > > > company
> > > >     > > > shall
> > > >     > > > > be
> > > >     > > > > >     > > > understood as
> > > >     > > > > >     > > >     >     > neither given nor
> > > >     > > > > >     > > >     >     >         >> endorsed by it. IG is
> a
> > > > trading
> > > >     > name
> > > >     > > > of
> > > >     > > > > IG
> > > >     > > > > >     > Markets
> > > >     > > > > >     > > >     > Limited (a
> > > >     > > > > >     > > >     >     > company
> > > >     > > > > >     > > >     >     >         >> registered in England
> > and
> > > > Wales,
> > > >     > > > company
> > > >     > > > > > number
> > > >     > > > > >     > > > 04008957)
> > > >     > > > > >     > > >     > and
> > > >     > > > > >     > > >     >     > IG Index
> > > >     > > > > >     > > >     >     >         >> Limited (a company
> > > > registered in
> > > >     > > > England
> > > >     > > > > > and
> > > >     > > > > >     > > Wales,
> > > >     > > > > >     > > >     > company
> > > >     > > > > >     > > >     >     > number
> > > >     > > > > >     > > >     >     >         >> 01190902). Registered
> > > > address at
> > > >     > > > Cannon
> > > >     > > > > > Bridge
> > > >     > > > > >     > > > House, 25
> > > >     > > > > >     > > >     >     > Dowgate Hill,
> > > >     > > > > >     > > >     >     >         >> London EC4R 2YA. Both
> IG
> > > > Markets
> > > >     > > > Limited
> > > >     > > > > >     > (register
> > > >     > > > > >     > > > number
> > > >     > > > > >     > > >     >     > 195355) and IG
> > > >     > > > > >     > > >     >     >         >> Index Limited
> (register
> > > > number
> > > >     > > 114059)
> > > >     > > > > are
> > > >     > > > > >     > > > authorised and
> > > >     > > > > >     > > >     >     > regulated by the
> > > >     > > > > >     > > >     >     >         >> Financial Conduct
> > > Authority.
> > > >     > > > > >     > > >     >     >         >>
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >         > --
> > > >     > > > > >     > > >     >     >         > -Regards,
> > > >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
> > > >     > > > > >     > > >     >     >         > (862) 250-7125
> > > >     > > > > >     > > >     >     >         >
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >         --
> > > >     > > > > >     > > >     >     >         -Regards,
> > > >     > > > > >     > > >     >     >         Mayuresh R. Gharat
> > > >     > > > > >     > > >     >     >         (862) 250-7125
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >     The information contained in
> > this
> > > > email
> > > >     > is
> > > >     > > > > > strictly
> > > >     > > > > >     > > > confidential
> > > >     > > > > >     > > >     > and
> > > >     > > > > >     > > >     >     > for the use of the addressee
> only,
> > > > unless
> > > >     > > > otherwise
> > > >     > > > > >     > > indicated.
> > > >     > > > > >     > > > If
> > > >     > > > > >     > > >     > you are
> > > >     > > > > >     > > >     >     > not the intended recipient,
> please
> > do
> > > > not
> > > >     > read,
> > > >     > > > > > copy, use
> > > >     > > > > >     > or
> > > >     > > > > >     > > >     > disclose to
> > > >     > > > > >     > > >     >     > others this message or any
> > > attachment.
> > > > Please
> > > >     > > > also
> > > >     > > > > > notify
> > > >     > > > > >     > the
> > > >     > > > > >     > > > sender
> > > >     > > > > >     > > >     > by
> > > >     > > > > >     > > >     >     > replying to this email or by
> > > telephone
> > > >     > (+44(020
> > > >     > > > > 7896
> > > >     > > > > > 0011)
> > > >     > > > > >     > > and
> > > >     > > > > >     > > > then
> > > >     > > > > >     > > >     > delete
> > > >     > > > > >     > > >     >     > the email and any copies of it.
> > > > Opinions,
> > > >     > > > > conclusion
> > > >     > > > > > (etc)
> > > >     > > > > >     > > > that do
> > > >     > > > > >     > > >     > not
> > > >     > > > > >     > > >     >     > relate to the official business
> of
> > > this
> > > >     > company
> > > >     > > > > > shall be
> > > >     > > > > >     > > > understood
> > > >     > > > > >     > > >     > as
> > > >     > > > > >     > > >     >     > neither given nor endorsed by it.
> > IG
> > > > is a
> > > >     > > trading
> > > >     > > > > > name of
> > > >     > > > > >     > IG
> > > >     > > > > >     > > > Markets
> > > >     > > > > >     > > >     >     > Limited (a company registered in
> > > > England and
> > > >     > > > Wales,
> > > >     > > > > > company
> > > >     > > > > >     > > > number
> > > >     > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> > > > company
> > > >     > > > > registered
> > > >     > > > > > in
> > > >     > > > > >     > > > England and
> > > >     > > > > >     > > >     > Wales,
> > > >     > > > > >     > > >     >     > company number 01190902).
> > Registered
> > > > address
> > > >     > at
> > > >     > > > > > Cannon
> > > >     > > > > >     > Bridge
> > > >     > > > > >     > > > House,
> > > >     > > > > >     > > >     > 25
> > > >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA.
> Both
> > > IG
> > > >     > Markets
> > > >     > > > > > Limited
> > > >     > > > > >     > > > (register
> > > >     > > > > >     > > >     > number
> > > >     > > > > >     > > >     >     > 195355) and IG Index Limited
> > > (register
> > > > number
> > > >     > > > > > 114059) are
> > > >     > > > > >     > > > authorised
> > > >     > > > > >     > > >     > and
> > > >     > > > > >     > > >     >     > regulated by the Financial
> Conduct
> > > > Authority.
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >     >
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >     > The information contained in this email
> > is
> > > > strictly
> > > >     > > > > > confidential
> > > >     > > > > >     > > and
> > > >     > > > > >     > > > for
> > > >     > > > > >     > > >     > the use of the addressee only, unless
> > > > otherwise
> > > >     > > > > indicated.
> > > >     > > > > > If you
> > > >     > > > > >     > > > are not
> > > >     > > > > >     > > >     > the intended recipient, please do not
> > read,
> > > > copy,
> > > >     > use
> > > >     > > > or
> > > >     > > > > > disclose
> > > >     > > > > >     > > to
> > > >     > > > > >     > > > others
> > > >     > > > > >     > > >     > this message or any attachment. Please
> > also
> > > > notify
> > > >     > > the
> > > >     > > > > > sender by
> > > >     > > > > >     > > > replying
> > > >     > > > > >     > > >     > to this email or by telephone (+44(020
> > 7896
> > > > 0011
> > > >     > <020%207896%200011>) and
> > > >     > > > > then
> > > >     > > > > > delete
> > > >     > > > > >     > > > the email
> > > >     > > > > >     > > >     > and any copies of it. Opinions,
> > conclusion
> > > > (etc)
> > > >     > that
> > > >     > > > do
> > > >     > > > > > not
> > > >     > > > > >     > relate
> > > >     > > > > >     > > > to the
> > > >     > > > > >     > > >     > official business of this company shall
> > be
> > > >     > understood
> > > >     > > > as
> > > >     > > > > > neither
> > > >     > > > > >     > > > given nor
> > > >     > > > > >     > > >     > endorsed by it. IG is a trading name of
> > IG
> > > > Markets
> > > >     > > > > Limited
> > > >     > > > > > (a
> > > >     > > > > >     > > company
> > > >     > > > > >     > > >     > registered in England and Wales,
> company
> > > > number
> > > >     > > > 04008957)
> > > >     > > > > > and IG
> > > >     > > > > >     > > > Index
> > > >     > > > > >     > > >     > Limited (a company registered in
> England
> > > and
> > > > Wales,
> > > >     > > > > company
> > > >     > > > > >     > number
> > > >     > > > > >     > > >     > 01190902). Registered address at Cannon
> > > > Bridge
> > > >     > House,
> > > >     > > > 25
> > > >     > > > > > Dowgate
> > > >     > > > > >     > > > Hill,
> > > >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets
> Limited
> > > > (register
> > > >     > > > number
> > > >     > > > > > 195355)
> > > >     > > > > >     > > > and IG
> > > >     > > > > >     > > >     > Index Limited (register number 114059)
> > are
> > > >     > authorised
> > > >     > > > and
> > > >     > > > > >     > regulated
> > > >     > > > > >     > > > by the
> > > >     > > > > >     > > >     > Financial Conduct Authority.
> > > >     > > > > >     > > >     >
> > > >     > > > > >     > > >
> > > >     > > > > >     > > >
> > > >     > > > > >     > > > The information contained in this email is
> > > strictly
> > > >     > > > > confidential
> > > >     > > > > > and
> > > >     > > > > >     > for
> > > >     > > > > >     > > > the use of the addressee only, unless
> otherwise
> > > >     > indicated.
> > > >     > > If
> > > >     > > > > > you are
> > > >     > > > > >     > not
> > > >     > > > > >     > > > the intended recipient, please do not read,
> > copy,
> > > > use or
> > > >     > > > > > disclose to
> > > >     > > > > >     > > others
> > > >     > > > > >     > > > this message or any attachment. Please also
> > > notify
> > > > the
> > > >     > > sender
> > > >     > > > > by
> > > >     > > > > >     > replying
> > > >     > > > > >     > > > to this email or by telephone (+44(020 7896
> > 0011
> > > >     > <020%207896%200011>) and then
> > > >     > > > > > delete the
> > > >     > > > > >     > > email
> > > >     > > > > >     > > > and any copies of it. Opinions, conclusion
> > (etc)
> > > > that do
> > > >     > > not
> > > >     > > > > > relate to
> > > >     > > > > >     > > the
> > > >     > > > > >     > > > official business of this company shall be
> > > > understood as
> > > >     > > > > neither
> > > >     > > > > > given
> > > >     > > > > >     > > nor
> > > >     > > > > >     > > > endorsed by it. IG is a trading name of IG
> > > Markets
> > > >     > Limited
> > > >     > > (a
> > > >     > > > > > company
> > > >     > > > > >     > > > registered in England and Wales, company
> number
> > > > 04008957)
> > > >     > > and
> > > >     > > > > IG
> > > >     > > > > > Index
> > > >     > > > > >     > > > Limited (a company registered in England and
> > > Wales,
> > > >     > company
> > > >     > > > > > number
> > > >     > > > > >     > > > 01190902). Registered address at Cannon
> Bridge
> > > > House, 25
> > > >     > > > > Dowgate
> > > >     > > > > > Hill,
> > > >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
> > > (register
> > > > number
> > > >     > > > > > 195355) and
> > > >     > > > > >     > IG
> > > >     > > > > >     > > > Index Limited (register number 114059) are
> > > > authorised and
> > > >     > > > > > regulated by
> > > >     > > > > >     > > the
> > > >     > > > > >     > > > Financial Conduct Authority.
> > > >     > > > > >     > > >
> > > >     > > > > >     > > The information contained in this email is
> > strictly
> > > >     > > > confidential
> > > >     > > > > > and for
> > > >     > > > > >     > > the use of the addressee only, unless otherwise
> > > > indicated.
> > > >     > If
> > > >     > > > you
> > > >     > > > > > are not
> > > >     > > > > >     > > the intended recipient, please do not read,
> copy,
> > > > use or
> > > >     > > > disclose
> > > >     > > > > > to
> > > >     > > > > >     > others
> > > >     > > > > >     > > this message or any attachment. Please also
> > notify
> > > > the
> > > >     > sender
> > > >     > > > by
> > > >     > > > > > replying
> > > >     > > > > >     > > to this email or by telephone (+44(020 7896
> 0011
> > > >     > <020%207896%200011>) and then
> > > >     > > > delete
> > > >     > > > > > the
> > > >     > > > > >     > email
> > > >     > > > > >     > > and any copies of it. Opinions, conclusion
> (etc)
> > > > that do
> > > >     > not
> > > >     > > > > > relate to
> > > >     > > > > >     > the
> > > >     > > > > >     > > official business of this company shall be
> > > > understood as
> > > >     > > > neither
> > > >     > > > > > given
> > > >     > > > > >     > nor
> > > >     > > > > >     > > endorsed by it. IG is a trading name of IG
> > Markets
> > > > Limited
> > > >     > (a
> > > >     > > > > > company
> > > >     > > > > >     > > registered in England and Wales, company number
> > > > 04008957)
> > > >     > and
> > > >     > > > IG
> > > >     > > > > > Index
> > > >     > > > > >     > > Limited (a company registered in England and
> > Wales,
> > > > company
> > > >     > > > > number
> > > >     > > > > >     > > 01190902). Registered address at Cannon Bridge
> > > > House, 25
> > > >     > > > Dowgate
> > > >     > > > > > Hill,
> > > >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > > number
> > > >     > > > 195355)
> > > >     > > > > > and IG
> > > >     > > > > >     > > Index Limited (register number 114059) are
> > > > authorised and
> > > >     > > > > > regulated by
> > > >     > > > > >     > the
> > > >     > > > > >     > > Financial Conduct Authority.
> > > >     > > > > >     > >
> > > >     > > > > >     > >
> > > >     > > > > >     > The information contained in this email is
> strictly
> > > >     > > confidential
> > > >     > > > > and
> > > >     > > > > > for
> > > >     > > > > >     > the use of the addressee only, unless otherwise
> > > > indicated. If
> > > >     > > you
> > > >     > > > > > are not
> > > >     > > > > >     > the intended recipient, please do not read, copy,
> > use
> > > > or
> > > >     > > disclose
> > > >     > > > > to
> > > >     > > > > > others
> > > >     > > > > >     > this message or any attachment. Please also
> notify
> > > the
> > > > sender
> > > >     > > by
> > > >     > > > > > replying
> > > >     > > > > >     > to this email or by telephone (+44(020 7896 0011
> > > >     > <020%207896%200011>) and then
> > > >     > > delete
> > > >     > > > > > the email
> > > >     > > > > >     > and any copies of it. Opinions, conclusion (etc)
> > that
> > > > do not
> > > >     > > > relate
> > > >     > > > > > to the
> > > >     > > > > >     > official business of this company shall be
> > understood
> > > > as
> > > >     > > neither
> > > >     > > > > > given nor
> > > >     > > > > >     > endorsed by it. IG is a trading name of IG
> Markets
> > > > Limited (a
> > > >     > > > > company
> > > >     > > > > >     > registered in England and Wales, company number
> > > > 04008957) and
> > > >     > > IG
> > > >     > > > > > Index
> > > >     > > > > >     > Limited (a company registered in England and
> Wales,
> > > > company
> > > >     > > > number
> > > >     > > > > >     > 01190902). Registered address at Cannon Bridge
> > House,
> > > > 25
> > > >     > > Dowgate
> > > >     > > > > > Hill,
> > > >     > > > > >     > London EC4R 2YA. Both IG Markets Limited
> (register
> > > > number
> > > >     > > 195355)
> > > >     > > > > > and IG
> > > >     > > > > >     > Index Limited (register number 114059) are
> > authorised
> > > > and
> > > >     > > > regulated
> > > >     > > > > > by the
> > > >     > > > > >     > Financial Conduct Authority.
> > > >     > > > > >     >
> > > >     > > > > >     >
> > > >     > > > > >
> > > >     > > > > >
> > > >     > > > > > The information contained in this email is strictly
> > > > confidential
> > > >     > and
> > > >     > > > for
> > > >     > > > > > the use of the addressee only, unless otherwise
> > indicated.
> > > > If you
> > > >     > are
> > > >     > > > not
> > > >     > > > > > the intended recipient, please do not read, copy, use
> or
> > > > disclose
> > > >     > to
> > > >     > > > > others
> > > >     > > > > > this message or any attachment. Please also notify the
> > > > sender by
> > > >     > > > replying
> > > >     > > > > > to this email or by telephone (+44(020 7896 0011
> > > >     > <020%207896%200011>) and then delete the
> > > >     > > > > email
> > > >     > > > > > and any copies of it. Opinions, conclusion (etc) that
> do
> > > not
> > > > relate
> > > >     > > to
> > > >     > > > > the
> > > >     > > > > > official business of this company shall be understood
> as
> > > > neither
> > > >     > > given
> > > >     > > > > nor
> > > >     > > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited
> > > (a
> > > >     > company
> > > >     > > > > > registered in England and Wales, company number
> 04008957)
> > > > and IG
> > > >     > > Index
> > > >     > > > > > Limited (a company registered in England and Wales,
> > company
> > > > number
> > > >     > > > > > 01190902). Registered address at Cannon Bridge House,
> 25
> > > > Dowgate
> > > >     > > Hill,
> > > >     > > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > > 195355)
> > > >     > and
> > > >     > > > IG
> > > >     > > > > > Index Limited (register number 114059) are authorised
> and
> > > > regulated
> > > >     > > by
> > > >     > > > > the
> > > >     > > > > > Financial Conduct Authority.
> > > >     > > > > >
> > > >     > > > > The information contained in this email is strictly
> > > > confidential and
> > > >     > > for
> > > >     > > > > the use of the addressee only, unless otherwise
> indicated.
> > If
> > > > you are
> > > >     > > not
> > > >     > > > > the intended recipient, please do not read, copy, use or
> > > > disclose to
> > > >     > > > others
> > > >     > > > > this message or any attachment. Please also notify the
> > sender
> > > > by
> > > >     > > replying
> > > >     > > > > to this email or by telephone (+44(020 7896 0011
> > > > <020%207896%200011>)
> > > >     > and then delete the
> > > >     > > > email
> > > >     > > > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > > relate
> > > >     > to
> > > >     > > > the
> > > >     > > > > official business of this company shall be understood as
> > > > neither
> > > >     > given
> > > >     > > > nor
> > > >     > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > > company
> > > >     > > > > registered in England and Wales, company number 04008957)
> > and
> > > > IG
> > > >     > Index
> > > >     > > > > Limited (a company registered in England and Wales,
> company
> > > > number
> > > >     > > > > 01190902). Registered address at Cannon Bridge House, 25
> > > > Dowgate
> > > >     > Hill,
> > > >     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > > 195355) and
> > > >     > > IG
> > > >     > > > > Index Limited (register number 114059) are authorised and
> > > > regulated
> > > >     > by
> > > >     > > > the
> > > >     > > > > Financial Conduct Authority.
> > > >     > > > >
> > > >     > > > The information contained in this email is strictly
> > > confidential
> > > > and
> > > >     > for
> > > >     > > > the use of the addressee only, unless otherwise indicated.
> If
> > > > you are
> > > >     > not
> > > >     > > > the intended recipient, please do not read, copy, use or
> > > > disclose to
> > > >     > > others
> > > >     > > > this message or any attachment. Please also notify the
> sender
> > > by
> > > >     > replying
> > > >     > > > to this email or by telephone (+44(020 7896 0011
> > > > <020%207896%200011>)
> > > >     > and then delete the
> > > >     > > email
> > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > > relate to
> > > >     > > the
> > > >     > > > official business of this company shall be understood as
> > > neither
> > > > given
> > > >     > > nor
> > > >     > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > > company
> > > >     > > > registered in England and Wales, company number 04008957)
> and
> > > IG
> > > > Index
> > > >     > > > Limited (a company registered in England and Wales, company
> > > > number
> > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > > Hill,
> > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > > 195355) and
> > > >     > IG
> > > >     > > > Index Limited (register number 114059) are authorised and
> > > > regulated by
> > > >     > > the
> > > >     > > > Financial Conduct Authority.
> > > >     > > >
> > > >     > > The information contained in this email is strictly
> > confidential
> > > > and for
> > > >     > > the use of the addressee only, unless otherwise indicated. If
> > you
> > > > are not
> > > >     > > the intended recipient, please do not read, copy, use or
> > disclose
> > > > to
> > > >     > others
> > > >     > > this message or any attachment. Please also notify the sender
> > by
> > > > replying
> > > >     > > to this email or by telephone (+44(020 7896 0011
> > > > <020%207896%200011>)
> > > >     > and then delete the email
> > > >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > > > relate to
> > > >     > the
> > > >     > > official business of this company shall be understood as
> > neither
> > > > given
> > > >     > nor
> > > >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > > company
> > > >     > > registered in England and Wales, company number 04008957) and
> > IG
> > > > Index
> > > >     > > Limited (a company registered in England and Wales, company
> > > number
> > > >     > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > Hill,
> > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > and IG
> > > >     > > Index Limited (register number 114059) are authorised and
> > > > regulated by
> > > >     > the
> > > >     > > Financial Conduct Authority.
> > > >     > >
> > > >     >
> > > >
> > > >
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jay Kreps <ja...@confluent.io>]

Makes sense!

-Jay

On Mon, Dec 19, 2016 at 2:40 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Wow just read that def over tired. Hopefully it makes sense. Or you get
> the gist at least.
>
> ________________________________________
> From: Michael Pearce <Mi...@ig.com>
> Sent: Monday, December 19, 2016 9:19:02 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hi Jay,
>
> Agreed this stemmed as offshoot from KIP-82.
>
> Which our main driver for was to be able to have some headers for a null
> value as such for our routing, audit, tracing and a few other bits which
> currently we are forced to do with a message wrapper, if we all agreed on
> KIP-82 that we need native headers and look to implement that the push for
> this would dissipate.
>
> This KIP would allow for though one use case that comes to mind we could
> see which is to have business data with a delete. Though as said this isn't
> something we are pushing for think really we would have.
>
> As such in summary yes, if you want to fully support KIP-82 and we can get
> that agreed in principle and a target release version, I think quite a few
> guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Monday, December 19, 2016 8:51:23 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hey Michael,
>
> Here is the compatibility concern I have:
>
>    1. You have a consumer app that relies on value == null to indicate a
>    delete (current semantics).
>    2. You upgrade Kafka and your clients.
>    3. Some producer starts using the tombstone field in combination with
>    non-null.
>
> I share Ismael's dislike of setting tombstones on records with null values.
> This makes sense as a transitional state, but as an end state its a bit
> weird. You'd expect to be able to mix null values and tombstones, and have
> the null values remain and the tombstones get compacted. However what will
> happen is both will be compacted and upon debugging this you'll learn that
> we sometimes use null in the value to indicate tombstone. Ismael's solution
> is a bigger compatibility break, though, so not sure if that is better.
>
> My other question is the relationship to KIP-82. My read is that this KIP
> solves some but not all of the problems KIP-82 is intended for. KIP-82, on
> the other hand, seems to address most of the motivating uses for this KIP.
> The exception is maybe item (5) on the list where you want to simultaneous
> delete and convey some information to subscribers, but I couldn't construct
> a concrete examples for that one. Do we need to rationalize these two KIPs?
> That is, do you still advocate this proposal if we do KIP-82 and vice
> versa? As you may have noticed I'm somewhat emotionally invested in the
> simplicity of the core data model, so my default position is let's try to
> avoid stuffing more stuff in, but if we have to add stuff I like each of
> these individually more than doing both. :-)
>
> -Jay
>
>
>
>
> On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jay
> >
> > I disagree here that we are breaking any compatibility, we went through
> > this on the discussion thread in fact with the help of that thread is how
> > the kip came to the solution.
> >
> > Also on the supported combinations front you mention, we are not taking
> > anything away afaik.
> >
> > Currently supported are only are:
> > Null value = delete
> > Non-null value = non delete
> >
> > With this kip we would support
> > Null value + tombstone = delete
> > Non null value + tombstone = delete
> > Non null value + no tombstone = non delete
> >
> > As for the alternative idea, this is simply a new policy, how can there
> be
> > confusion here? For this policy it would be explicit that tombstone
> marker
> > would need to be set for a delete.
> >
> > I'm going to vent a little now as starting to get quite frustrated.
> >
> > We are going round in circles on kip-82 as per use cases there is now
> many
> > use cases, how many more are needed? just because confluent don't see
> these
> > doesn't mean they aren't real use cases other have, this is the point of
> > the Apache foundation, it shouldn't be the view of just one organisation.
> > It really is getting a feeling of the NIH syndrome. Rather than it being
> > constructive on discussion of the implementation detail.
> >
> > kip-87 spawned from as on the kip call we all agreed this was needed. And
> > would at least allow a custom wrapper be supported in a compacted topic,
> > allowing meta data. Which again now I feel we are spinning wheels, and
> > simply finding reasons not support it.
> >
> > Cheers
> > Mike
> >
> >
> >
> > Sent using OWA for iPad
> > ________________________________________
> > From: Jay Kreps <ja...@confluent.io>
> > Sent: Friday, December 16, 2016 7:09:23 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hey Michael,
> >
> > I do think it might have been better had we started with a separate
> concept
> > of null vs delete. Given that we are where we are, I'm not sure that the
> > possible cures we explored so far are better than the disease.
> >
> > I apologize for coming to this late, but I didn't really understand the
> > implications of the KIP and that we'd be breaking compatibility with
> > existing apps until the vote had begun, and in my defense I'm not sure
> the
> > other folks voting did either.
> >
> > I think we all agree there are many existing apps that are built with the
> > assumption of "null value non-tombstone" and it isn't possible to
> > disambiguate these from tombstones on the producer. It isn't that anyone
> is
> > saying we have to support all four possibilities at once, it's that we
> > simply can't orphan one of the existing combinations or our users will
> eat
> > us!
> >
> > If I've understood your alternate solution of adding another setting for
> > compaction, I think this does fix the compatibility problem, but it adds
> an
> > odd mode the user has to add on all their topics. While the current state
> > is easily explainable, the resulting state where the meaning of tombstone
> > and null are overlapping and ambiguous and dependent on a compaction
> > setting that could change out of band or not be in sync with your code in
> > some environment seems worse to me then where we currently are. I think
> the
> > question is how would this combination be explained to users and does it
> > make sense?
> >
> > -Jay
> >
> >
> >
> > On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Chaps,
> > >
> > > Can we either get one more +1 binding (we have 2 already) on the
> > existing?
> > >
> > > Or have some response on the below possible alternative. We are keen to
> > > get working on this, so we make next feature release.
> > >
> > > Cheers
> > > Mike
> > >
> > >
> > > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com> wrote:
> > >
> > >     Hi Ismael
> > >
> > >     Did you see our email this morning, what's your thoughts on this
> > > approach to instead we simply have a brand new policy?
> > >
> > >     Cheers
> > >     Mike
> > >
> > >
> > >     Sent using OWA for iPhone
> > >     ________________________________________
> > >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
> > Juma <
> > > ismael@juma.me.uk>
> > >     Sent: Tuesday, December 13, 2016 11:30:05 AM
> > >     To: dev@kafka.apache.org
> > >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > >     Yes, this is actually tricky to do in a way where we both have the
> > > desired
> > >     semantics and maintain compatibility. When someone creates a
> > >     `ProducerRecord` with a `null` value today, the producer doesn't
> know
> > > if
> > >     it's meant to be a tombstone or not. For V3 messages, it's easy
> when
> > > the
> > >     constructor that takes a tombstone is used. However, if any other
> > >     constructor is used, it's not clear. A couple of options I can
> think
> > > of,
> > >     none of them particularly nice:
> > >
> > >     1. Have a third state where tombstone = unknown and the broker
> would
> > > set
> > >     the tombstone bit if the value was null and the topic was
> compacted.
> > > People
> > >     that wanted to pass a non-null value for the tombstone would have
> to
> > > use
> > >     the constructor that takes a tombstone. The drawbacks: third state
> > for
> > >     tombstone in message format, message conversion at the broker for a
> > > common
> > >     case.
> > >
> > >     2. Extend MetadataResponse to optionally include topic configs,
> which
> > > would
> > >     make it possible for the producer to be smarter about setting the
> > >     tombstone. It would only do it if a tombstone was not passed
> > > explicitly,
> > >     the value was null and the topic was compacted. The main drawback
> is
> > > that
> > >     the producer would be getting a bit more data for each topic even
> > > though it
> > >     probably won't use it most of the time. Extending MetadataResponse
> to
> > >     return topic configs would be useful for other reasons as well, so
> > that
> > >     part seems OK.
> > >
> > >     In addition, for both proposals, we could consider adding warnings
> to
> > > the
> > >     documentation that the behaviour of the constructors that don't
> take
> > a
> > >     tombstone would change in the next major release so that tombstone
> =
> > > false.
> > >     Not sure if this would be worth it though.
> > >
> > >     Ismael
> > >
> > >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> > > ewen@confluent.io>
> > >     wrote:
> > >
> > >     > Michael,
> > >     >
> > >     > It kind of depends on how you want to interpret the tombstone
> flag.
> > > If it's
> > >     > purely a producer-facing Kafka-level thing that we treat as
> > internal
> > > to the
> > >     > broker and log cleaner once the record is sent, then your
> approach
> > > makes
> > >     > sense. You're just moving copying the null-indicates-delete
> > behavior
> > > of the
> > >     > old constructor into the tombstone flag.
> > >     >
> > >     > However, if you want this change to more generally decouple the
> > idea
> > > of
> > >     > deletion and null values, then you are sometimes converting what
> > > might be a
> > >     > completely valid null value that doesn't indicate deletion into a
> > >     > tombstone. Downstream applications could potentially handle these
> > > cases
> > >     > differently given the separation of deletion from value.
> > >     >
> > >     > I guess the question is if we want to try to support the latter
> > even
> > > for
> > >     > topics where we have older produce requests. An example where
> this
> > > could
> > >     > come up is in something like a CDC Connector. If we try to
> support
> > > the
> > >     > semantic difference, a connector might write changes to Kafka
> using
> > > the
> > >     > tombstone flag to indicate when a row was truly deleted (vs an
> > > update that
> > >     > sets it to null but still present; this probably makes more sense
> > > for CDC
> > >     > from document stores or extracting single columns). There are
> > various
> > >     > reasons we might want to maintain the full log and not turn
> > > compaction on
> > >     > (or just use a time-based retention policy), but downstream
> > > applications
> > >     > might care to know the difference between a delete and a null
> > value.
> > > In
> > >     > fact both versions of the same log (compacted and time-retention)
> > > could be
> > >     > useful and I don't think it'll be uncommon to maintain both or
> use
> > > KIP-71
> > >     > to maintain a hybrid compacted/retention topic.
> > >     >
> > >     > -Ewen
> > >     >
> > >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > >     > wrote:
> > >     >
> > >     > > Hi Jay,
> > >     > >
> > >     > > Why wouldn't that work, the tombstone value is only looked at
> by
> > > the
> > >     > > broker, on a topic configured for compaction as such is benign
> on
> > > non
> > >     > > compacted topics. This is as much as sending a null value
> > currently
> > >     > >
> > >     > >
> > >     > > Regards
> > >     > > Mike
> > >     > >
> > >     > >
> > >     > >
> > >     > > Sent using OWA for iPhone
> > >     > > ________________________________________
> > >     > > From: Jay Kreps <ja...@confluent.io>
> > >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
> > >     > > To: dev@kafka.apache.org
> > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     > >
> > >     > > Hey Michael,
> > >     > >
> > >     > > I'm not quite sure that works as that would translate ALL null
> > > values to
> > >     > > tombstones, even for non-compacted topics that use null as an
> > > acceptable
> > >     > > value sent by the producer and expected by the consumer.
> > >     > >
> > >     > > -Jay
> > >     > >
> > >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > >     > > wrote:
> > >     > >
> > >     > > > Hi Ewen,
> > >     > > >
> > >     > > > I think the easiest way to show this is with code.
> > >     > > >
> > >     > > > As you can see we keep the existing behaviour for
> code/binaries
> > > calling
> > >     > > > the pre-existing constructors, whereby if the value is null
> the
> > >     > tombstone
> > >     > > > is set to true.
> > >     > > >
> > >     > > > Regards
> > >     > > > Mike
> > >     > > >
> > >     > > >
> > >     > > >
> > >     > > >     /**
> > >     > > >      * Creates a record with a specified timestamp to be sent
> > to
> > > a
> > >     > > > specified topic and partition
> > >     > > >      *
> > >     > > >      * @param topic The topic the record will be appended to
> > >     > > >      * @param partition The partition to which the record
> > should
> > > be
> > >     > sent
> > >     > > >      * @param timestamp The timestamp of the record
> > >     > > >      * @param tombstone if the record should be treated as a
> > > tombstone
> > >     > if
> > >     > > > the topic is compacted
> > >     > > >      * @param key The key that will be included in the record
> > >     > > >      * @param value The record contents
> > >     > > >      */
> > >     > > >     public ProducerRecord(String topic, Integer partition,
> > > Boolean
> > >     > > > tombstone, Long timestamp, K key, V value) {
> > >     > > >         if (topic == null)
> > >     > > >             throw new IllegalArgumentException("Topic cannot
> > be
> > >     > null.");
> > >     > > >         if (timestamp != null && timestamp < 0)
> > >     > > >             throw new IllegalArgumentException(
> > >     > > >                     String.format("Invalid timestamp: %d.
> > > Timestamp
> > >     > > should
> > >     > > > always be non-negative or null.", timestamp));
> > >     > > >         if (partition != null && partition < 0)
> > >     > > >             throw new IllegalArgumentException(
> > >     > > >                     String.format("Invalid partition: %d.
> > > Partition
> > >     > > number
> > >     > > > should always be non-negative or null.", partition));
> > >     > > >         if (!tombstone && value == null){
> > >     > > >             throw new IllegalArgumentException(
> > >     > > >                     String.format("Tombstone must be true if
> > null
> > >     > > value"));
> > >     > > >         }
> > >     > > >         this.topic = topic;
> > >     > > >         this.partition = partition;
> > >     > > >         this.tombstone = tombstone;
> > >     > > >         this.key = key;
> > >     > > >         this.value = value;
> > >     > > >         this.timestamp = timestamp;
> > >     > > >     }
> > >     > > >
> > >     > > >     public ProducerRecord(String topic, Integer partition,
> Long
> > >     > > timestamp,
> > >     > > > K key, V value) {
> > >     > > >         this(topic, partition, value == null, timestamp, key,
> > > value);
> > >     > > >     }
> > >     > > > ________________________________________
> > >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> > >     > > > Sent: Sunday, December 11, 2016 5:45 AM
> > >     > > > To: dev@kafka.apache.org
> > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     > > >
> > >     > > > It seemed like this was addressed in the migration section,
> > > wasn't it?
> > >     > > The
> > >     > > > V2 vs V3 requests and the fact that the broker will downgrade
> > the
> > >     > message
> > >     > > > format (losing zero copy) if you issues a V2 request to a
> > broker
> > > using
> > >     > V3
> > >     > > > format handles compatibility, does it not? The existing
> > > consumers won't
> > >     > > see
> > >     > > > the extra metadata in the value, but they will get a null
> > > instead and
> > >     > > treat
> > >     > > > it as a tombstone. Certainly there is a performance impact,
> but
> > > it
> > >     > seemed
> > >     > > > compatible.
> > >     > > >
> > >     > > > I'm worried about this though:
> > >     > > >
> > >     > > >
> > >     > > >    - *NOTE *: With the new version of producer client using
> > >     > > ProduceRequest
> > >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not
> set)
> > > and
> > >     > null
> > >     > > > value
> > >     > > >    should not be allowed as the older version of consumer
> using
> > >     > > > FetchRequest
> > >     > > >    V2 will think of this as a tombstone when its actually
> not.
> > >     > > >
> > >     > > >
> > >     > > > Unless I'm misunderstanding, this ends up breaking binary
> > > compatibility
> > >     > > for
> > >     > > > the Java API. It sounds like this suggests that passing a
> null
> > > value to
> > >     > > the
> > >     > > > existing ProducerRecord constructors would generate an
> > exception
> > > since
> > >     > > you
> > >     > > > didn't explicitly enable the tombstone (via whatever new
> > > constructor is
> > >     > > > provided). But that means you can't swap in a newer client
> jar
> > > without
> > >     > > > recompiling and get the same behavior if your app deletes
> keys
> > > using
> > >     > the
> > >     > > > current approach because your app will start throwing
> > > exceptions. Maybe
> > >     > > > this is a tradeoff we're ok with, but we've tried pretty hard
> > to
> > > avoid
> > >     > > > breaking compatibility like this so far -- it makes picking
> up
> > > bug
> > >     > fixes
> > >     > > in
> > >     > > > the clients harder for users of frameworks that have to pin
> to
> > > earlier
> > >     > > > default versions for compatibility.
> > >     > > >
> > >     > > > But then later the KIP says:
> > >     > > >
> > >     > > >
> > >     > > >    - The old constructors for ProducerRecord without this
> > > parameter
> > >     > will
> > >     > > be
> > >     > > >    kept but updated so that their default behaviour if
> setting
> > > null
> > >     > value
> > >     > > > will
> > >     > > >    be the tombstone will be set to true to keep existing
> > > behaviour.
> > >     > > >
> > >     > > >
> > >     > > > So maybe I am misinterpreting something.
> > >     > > >
> > >     > > > And just a nit re: motivation section, item 6 would be
> clearer
> > > for a
> > >     > > union
> > >     > > > schema with null (or optional schemas in other formats), e.g.
> > > [null,
> > >     > > > string], in which case losing the schema truly is losing
> > > information
> > >     > > > (whereas null is already the only valid value for a pure null
> > > schema).
> > >     > > >
> > >     > > > -Ewen
> > >     > > >
> > >     > > >
> > >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > >     > >
> > >     > > > wrote:
> > >     > > >
> > >     > > > > Hi Jay,
> > >     > > > >
> > >     > > > > Good point this detail is missing in the KIP write up. Ive
> > > added this
> > >     > > > now.
> > >     > > > >
> > >     > > > > Essentially simply just upgrading the clients we do not
> > expect
> > > any
> > >     > > client
> > >     > > > > app code change needed.
> > >     > > > >
> > >     > > > > Cheers
> > >     > > > > Mike
> > >     > > > > ________________________________________
> > >     > > > > From: Jay Kreps <ja...@confluent.io>
> > >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
> > >     > > > > To: dev@kafka.apache.org
> > >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     > > > >
> > >     > > > > Michael,
> > >     > > > >
> > >     > > > > The compatibility section goes through the migration path,
> > but
> > > isn't
> > >     > > the
> > >     > > > > bigger compatibility issue with existing apps? There are
> many
> > >     > (probably
> > >     > > > > thousands) of apps in production that use this feature and
> > > send null
> > >     > to
> > >     > > > > mean delete. It seems like this would break compatibility
> > with
> > > them,
> > >     > > and
> > >     > > > > they would have to be rewritten to right?
> > >     > > > >
> > >     > > > > -Jay
> > >     > > > >
> > >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> > >     > Michael.Pearce@ig.com
> > >     > > >
> > >     > > > > wrote:
> > >     > > > >
> > >     > > > > > Hi Jun,
> > >     > > > > >
> > >     > > > > > 4) On v3 we honour the tombstone. As such we expect it to
> > be
> > > set
> > >     > > > > correctly
> > >     > > > > > as per the KIP.
> > >     > > > > >
> > >     > > > > > 4.1) why would we want to produce an error when its v3?
> > This
> > > is the
> > >     > > > exact
> > >     > > > > > purpose to support non-null tombstone's
> > >     > > > > > 4.2) again here im unclear on the question on the v3,
> > produce
> > >     > request
> > >     > > > we
> > >     > > > > > expect the tombstone flag to be set correctly.
> > >     > > > > >
> > >     > > > > > 4.4) compaction only occurs on compacted topics, the bit
> > > makes no
> > >     > > > > > difference and not looked at on un-compacted (time/size
> > based
> > >     > > > eviction).
> > >     > > > > >
> > >     > > > > >
> > >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io>
> wrote:
> > >     > > > > >
> > >     > > > > >     Hi, Michael,
> > >     > > > > >
> > >     > > > > >     4. Then, I think I misunderstood this point. Could
> you
> > > document
> > >     > > the
> > >     > > > > >     following points in the wiki?
> > >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
> > > non-null
> > >     > value,
> > >     > > > > does
> > >     > > > > > the
> > >     > > > > >     send() get an error or does the producer
> automatically
> > > set the
> > >     > > > value
> > >     > > > > to
> > >     > > > > >     null?
> > >     > > > > >     4.2 If producer V3 doesn't set tombstone, but
> provides
> > a
> > > null
> > >     > > > value,
> > >     > > > > > does
> > >     > > > > >     the send() get an error or does the producer
> > > automatically sets
> > >     > > the
> > >     > > > > >     tombstone?
> > >     > > > > >     4.3 Does the broker only expect messages that (a)
> have
> > no
> > >     > > tombstone
> > >     > > > > and
> > >     > > > > >     non-null value; (b) have tombstone and null value and
> > > reject
> > >     > the
> > >     > > > > > messages
> > >     > > > > >     with an error code otherwise?
> > >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
> > > compacted
> > >     > on
> > >     > > > > not?
> > >     > > > > >
> > >     > > > > >     Thanks,
> > >     > > > > >
> > >     > > > > >     Jun
> > >     > > > > >
> > >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > >     > > > > Michael.Pearce@ig.com
> > >     > > > > > >
> > >     > > > > >     wrote:
> > >     > > > > >
> > >     > > > > >     > Not at all.  This only acts on compacted topics
> just
> > > as what
> > >     > > > occurs
> > >     > > > > > today
> > >     > > > > >     >
> > >     > > > > >     > Sent using OWA for iPhone
> > >     > > > > >     > ________________________________________
> > >     > > > > >     > From: Jun Rao <ju...@confluent.io>
> > >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > >     > > > > >     > To: dev@kafka.apache.org
> > >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone
> > > Flag
> > >     > > > > >     >
> > >     > > > > >     > Hi, Michael,
> > >     > > > > >     >
> > >     > > > > >     > 4. Hmm, does that mean the new client library can
> > > never send
> > >     > a
> > >     > > > null
> > >     > > > > > message
> > >     > > > > >     > even to a regular topic? This seems like a change
> of
> > > the
> > >     > > existing
> > >     > > > > > behavior.
> > >     > > > > >     >
> > >     > > > > >     > Thanks,
> > >     > > > > >     >
> > >     > > > > >     > Jun
> > >     > > > > >     >
> > >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > >     > > > > > Michael.Pearce@ig.com>
> > >     > > > > >     > wrote:
> > >     > > > > >     >
> > >     > > > > >     > > Hi Jun,
> > >     > > > > >     > >
> > >     > > > > >     > > Re 4) That's because we expect the tombstone
> value
> > > to be
> > >     > set
> > >     > > > > > correctly if
> > >     > > > > >     > > message bit is 2, as such if an older client
> sends
> > > in on
> > >     > old
> > >     > > > > > message the
> > >     > > > > >     > > message is upcast and the bit is set correctly.
> And
> > > such no
> > >     > > > > longer
> > >     > > > > > need
> > >     > > > > >     > to
> > >     > > > > >     > > check the value. Mayuresh can you confirm my
> > > thinking and
> > >     > > > > > understanding
> > >     > > > > >     > of
> > >     > > > > >     > > what we've discussed?
> > >     > > > > >     > >
> > >     > > > > >     > > The second point I understand what you're getting
> > at
> > > now my
> > >     > > > > > apologies.
> > >     > > > > >     > Yes
> > >     > > > > >     > > this makes sense to save on touching the message,
> > if
> > > we're
> > >     > > the
> > >     > > > > > only kip
> > >     > > > > >     > > going in, in this release.
> > >     > > > > >     > >
> > >     > > > > >     > > Cheers
> > >     > > > > >     > > Mike
> > >     > > > > >     > >
> > >     > > > > >     > > Sent using OWA for iPhone
> > >     > > > > >     > > ________________________________________
> > >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
> > >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > >     > > > > >     > > To: dev@kafka.apache.org
> > >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > > Tombstone Flag
> > >     > > > > >     > >
> > >     > > > > >     > > Hi, Michael,
> > >     > > > > >     > >
> > >     > > > > >     > > 4. Is this updated in the wiki? The text "If the
> > > magic byte
> > >     > > on
> > >     > > > > > message is
> > >     > > > > >     > > 2, the broker should use the tombstone bit for
> log
> > >     > > compaction."
> > >     > > > > > doesn't
> > >     > > > > >     > > seem to have changed.
> > >     > > > > >     > >
> > >     > > > > >     > > 2. My point is that if we change the message
> format
> > > just
> > >     > for
> > >     > > > this
> > >     > > > > > KIP, we
> > >     > > > > >     > > should consider whether it's worth optimizing the
> > > down
> > >     > > > conversion
> > >     > > > > > path
> > >     > > > > >     > > (i.e., decide whether a conversion is needed by
> > just
> > >     > looking
> > >     > > at
> > >     > > > > the
> > >     > > > > >     > > tombstone bit in the wrapper message) since
> > > tombstone will
> > >     > be
> > >     > > > > used
> > >     > > > > >     > rarely.
> > >     > > > > >     > > However, if the message format change here is
> > > combined with
> > >     > > > other
> > >     > > > > > KIPs,
> > >     > > > > >     > > then this optimization likely won't be needed.
> The
> > > latter
> > >     > > > > probably
> > >     > > > > > makes
> > >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> > > think?
> > >     > > > > >     > >
> > >     > > > > >     > > Other than those, +1 from me,
> > >     > > > > >     > >
> > >     > > > > >     > > Thanks,
> > >     > > > > >     > >
> > >     > > > > >     > > Jun
> > >     > > > > >     > >
> > >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > >     > > > > > Michael.Pearce@ig.com>
> > >     > > > > >     > > wrote:
> > >     > > > > >     > >
> > >     > > > > >     > > > Hi Jun
> > >     > > > > >     > > >
> > >     > > > > >     > > > do we have your vote on this now?
> > >     > > > > >     > > >
> > >     > > > > >     > > > Any other concerns?
> > >     > > > > >     > > >
> > >     > > > > >     > > > Cheers
> > >     > > > > >     > > > Mike
> > >     > > > > >     > > >
> > >     > > > > >     > > > Sent using OWA for iPhone
> > >     > > > > >     > > > ________________________________________
> > >     > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> > >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > >     > > > > >     > > > To: dev@kafka.apache.org
> > >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > > Tombstone
> > >     > Flag
> > >     > > > > >     > > >
> > >     > > > > >     > > > Hi Jun,
> > >     > > > > >     > > >
> > >     > > > > >     > > > Have updated. Thanks again for the feedback.
> > >     > > > > >     > > >
> > >     > > > > >     > > > Agree yes we should align up when it gets to
> > that,
> > > I
> > >     > assume
> > >     > > > > > you've
> > >     > > > > >     > > flagged
> > >     > > > > >     > > > the same in the other KIP?
> > >     > > > > >     > > >
> > >     > > > > >     > > > I think for now let's get this KIP approved,
> then
> > > we can
> > >     > > > start
> > >     > > > > > the work
> > >     > > > > >     > > to
> > >     > > > > >     > > > get an initial PR, then we can discuss how to
> > > align the
> > >     > two
> > >     > > > if
> > >     > > > > > needed.
> > >     > > > > >     > > >
> > >     > > > > >     > > > Cheers,
> > >     > > > > >     > > > Mike
> > >     > > > > >     > > >
> > >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
> > jun@confluent.io>
> > >     > wrote:
> > >     > > > > >     > > >
> > >     > > > > >     > > >     Hi, Michael,
> > >     > > > > >     > > >
> > >     > > > > >     > > >     For 2), this is fine. Could you update the
> > KIP
> > > wiki
> > >     > to
> > >     > > > make
> > >     > > > > > this
> > >     > > > > >     > and
> > >     > > > > >     > > > other
> > >     > > > > >     > > >     points clearer? Other than that, the KIP
> > looks
> > > good
> > >     > to
> > >     > > > me.
> > >     > > > > >     > > >
> > >     > > > > >     > > >     An orthogonal thing is that there are other
> > > KIPs such
> > >     > > as
> > >     > > > > > KIP-98
> > >     > > > > >     > that
> > >     > > > > >     > > > also
> > >     > > > > >     > > >     intend to change the message format. If
> they
> > > all get
> > >     > > > > > approved, we
> > >     > > > > >     > > > should
> > >     > > > > >     > > >     think about whether it's better to just
> bump
> > > up the
> > >     > > magic
> > >     > > > > > byte once
> > >     > > > > >     > > to
> > >     > > > > >     > > >     incorporate multiple format changes like we
> > > did in
> > >     > > > > > KIP-31/KIP-32.
> > >     > > > > >     > > >
> > >     > > > > >     > > >     Thanks,
> > >     > > > > >     > > >
> > >     > > > > >     > > >     Jun
> > >     > > > > >     > > >
> > >     > > > > >     > > >
> > >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
> > Pearce
> > > <
> > >     > > > > >     > > Michael.Pearce@ig.com>
> > >     > > > > >     > > >     wrote:
> > >     > > > > >     > > >
> > >     > > > > >     > > >     > Hi Jao
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     > Thanks for the response. Sorry for slow
> > > reply, both
> > >     > > > with
> > >     > > > > > personal
> > >     > > > > >     > > > sickness
> > >     > > > > >     > > >     > and also battling some critical issues
> > > encountered
> > >     > > > since
> > >     > > > > >     > upgrading
> > >     > > > > >     > > to
> > >     > > > > >     > > >     > 0.10.1.0
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     > 1) Thans for spotting, Document error
> where
> > > we
> > >     > > branched
> > >     > > > > > this KIP
> > >     > > > > >     > > from
> > >     > > > > >     > > >     > KIP-82, will get that removed.
> > >     > > > > >     > > >     > 2) Intent is to do this just at the
> record
> > > message
> > >     > > > level.
> > >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure this
> is
> > >     > > corrected.
> > >     > > > > >     > > >     > 4) As per discussion thread we will
> support
> > >     > > tombstone +
> > >     > > > > > null
> > >     > > > > >     > value,
> > >     > > > > >     > > >     > tombstone + non null value, no tombstone
> +
> > > null
> > >     > > value.
> > >     > > > > >     > > >     > 5) I believe this was in the discussion
> > > thread,
> > >     > > > @Mayuresh
> > >     > > > > > is this
> > >     > > > > >     > > >     > something we've overlooked? I thought we
> > > would down
> > >     > > > > > convert and
> > >     > > > > >     > > > remove the
> > >     > > > > >     > > >     > value so the old consumer had existing
> > > behavior, or
> > >     > > is
> > >     > > > > > there
> > >     > > > > >     > > > something we
> > >     > > > > >     > > >     > haven't thought about?
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     > Cheers
> > >     > > > > >     > > >     > Mike
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> > > jun@confluent.io>
> > >     > > > > wrote:
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     Hi, Michael,
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     Thanks for the KIP. A few comments
> > below.
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     1. The message format change contains
> > >     > > > "HeadersLength
> > >     > > > > >     > Headers".
> > >     > > > > >     > > > Is that
> > >     > > > > >     > > >     >     intended?
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     2. For compressed messageset, is the
> > > tombstone
> > >     > > bit
> > >     > > > > > only set
> > >     > > > > >     > at
> > >     > > > > >     > > > the
> > >     > > > > >     > > >     > shallow
> > >     > > > > >     > > >     >     level? Do we always leave that bit in
> > the
> > >     > wrapper
> > >     > > > > > message
> > >     > > > > >     > > unset?
> > >     > > > > >     > > > An
> > >     > > > > >     > > >     >     alternative is to set the tombstone
> bit
> > > in the
> > >     > > > > wrapper
> > >     > > > > > if at
> > >     > > > > >     > > > least one
> > >     > > > > >     > > >     >     inner message has the tombstone bit
> > set.
> > > This
> > >     > > makes
> > >     > > > > > things a
> > >     > > > > >     > > bit
> > >     > > > > >     > > > more
> > >     > > > > >     > > >     >     complicated, but we could potentially
> > > exploit
> > >     > > that
> > >     > > > > for
> > >     > > > > >     > > > optimizing down
> > >     > > > > >     > > >     >     conversion. For example, we only need
> > to
> > >     > convert
> > >     > > > > > messages
> > >     > > > > >     > with
> > >     > > > > >     > > > magic 2
> > >     > > > > >     > > >     > to
> > >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone
> bit
> > > is set
> > >     > > > > > (conversion is
> > >     > > > > >     > > > always
> > >     > > > > >     > > >     > needed
> > >     > > > > >     > > >     >     from magic 2 to magic 0). Not sure if
> > the
> > >     > > > > optimization
> > >     > > > > > is
> > >     > > > > >     > worth
> > >     > > > > >     > > > the
> > >     > > > > >     > > >     >     complexity though.
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     3. The referencing of the new version
> > of
> > >     > > > > >     > > > ProducerRequest/FetchRequest
> > >     > > > > >     > > >     > is
> > >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> > > convention
> > >     > > > starts
> > >     > > > > at
> > >     > > > > >     > version
> > >     > > > > >     > > > at 0, I
> > >     > > > > >     > > >     >     think the new version would be 3.
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     4. "If the magic byte on message is
> 2,
> > > the
> > >     > broker
> > >     > > > > > should use
> > >     > > > > >     > > the
> > >     > > > > >     > > >     > tombstone
> > >     > > > > >     > > >     >     bit for log compaction." What about
> > null
> > > value?
> > >     > > My
> > >     > > > > >     > > understanding
> > >     > > > > >     > > > is
> > >     > > > > >     > > >     > that
> > >     > > > > >     > > >     >     null value will be treated the same
> as
> > > setting
> > >     > > the
> > >     > > > > > tombstone
> > >     > > > > >     > > bit.
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     5. For the migration path, it would
> be
> > > useful
> > >     > to
> > >     > > > > > describe the
> > >     > > > > >     > > > down
> > >     > > > > >     > > >     >     conversion path to consumers (i.e.,
> > > brokers on
> > >     > > > > message
> > >     > > > > > format
> > >     > > > > >     > > > 0.10.2
> > >     > > > > >     > > >     > and
> > >     > > > > >     > > >     >     consumers on older version).
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     Thanks,
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     Jun
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
> > Michael
> > >     > Pearce <
> > >     > > > > >     > > > Michael.Pearce@ig.com
> > >     > > > > >     > > >     > >
> > >     > > > > >     > > >     >     wrote:
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >     > Hi All,
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     > We have been discussing in the
> below
> > > thread
> > >     > and
> > >     > > > > final
> > >     > > > > >     > changes
> > >     > > > > >     > > > have
> > >     > > > > >     > > >     > been
> > >     > > > > >     > > >     >     > made to the KIP wiki based on these
> > >     > > discussions.
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     > We would now like to put to the
> vote
> > > the
> > >     > > > following
> > >     > > > > > KIP:
> > >     > > > > >     > > >     >     > https://cwiki.apache.org/
> > >     > > > > > confluence/display/KAFKA/KIP-
> > >     > > > > >     > 87+-+
> > >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     > This kip is for having a distinct
> > > compaction
> > >     > > > > > attribute
> > >     > > > > >     > > > "tombstone"
> > >     > > > > >     > > >     > flag
> > >     > > > > >     > > >     >     > instead of relying on null value,
> > > allowing
> > >     > > > non-null
> > >     > > > > > value
> > >     > > > > >     > > > delete
> > >     > > > > >     > > >     > messages.
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     > Many thanks,
> > >     > > > > >     > > >     >     > Michael
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
> > Pearce"
> > > <
> > >     > > > > >     > > Michael.Pearce@ig.com>
> > >     > > > > >     > > >     > wrote:
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >     Hi Mayuresh,
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >     LGTM. Ive just made one small
> > > adjustment
> > >     > > > > > updating the
> > >     > > > > >     > > wire
> > >     > > > > >     > > >     > protocol to
> > >     > > > > >     > > >     >     > show the magic byte bump.
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >     Do we think we're good to put
> to
> > a
> > > vote?
> > >     > Is
> > >     > > > > > there any
> > >     > > > > >     > > > other bits
> > >     > > > > >     > > >     >     > needing discussion?
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >     Cheers
> > >     > > > > >     > > >     >     >     Mike
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
> > > Gharat" <
> > >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
> > >     > > > > >     > > >     >     > wrote:
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >         Hi Michael,
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >         I have updated the
> migration
> > > section
> > >     > of
> > >     > > > the
> > >     > > > > > KIP.
> > >     > > > > >     > Can
> > >     > > > > >     > > > you
> > >     > > > > >     > > >     > please
> > >     > > > > >     > > >     >     > take a look?
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >         Thanks,
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >         Mayuresh
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at
> 9:07
> > > AM,
> > >     > > Mayuresh
> > >     > > > > > Gharat <
> > >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
> > >     > > > > >     > > >     >     >         > wrote:
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >         > Hi Michael,
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         > That whilst sending
> > > tombstone and
> > >     > non
> > >     > > > > null
> > >     > > > > > value,
> > >     > > > > >     > > the
> > >     > > > > >     > > >     > consumer
> > >     > > > > >     > > >     >     > can expect
> > >     > > > > >     > > >     >     >         > only to receive the
> > non-null
> > >     > message
> > >     > > > only
> > >     > > > > > in step
> > >     > > > > >     > > > (3) is
> > >     > > > > >     > > >     > this
> > >     > > > > >     > > >     >     > correct?
> > >     > > > > >     > > >     >     >         > ---> I do agree with you
> > > here.
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         > Becket, Ismael : can you
> > guys
> > >     > review
> > >     > > > the
> > >     > > > > >     > migration
> > >     > > > > >     > > > plan
> > >     > > > > >     > > >     > listed
> > >     > > > > >     > > >     >     > above using
> > >     > > > > >     > > >     >     >         > magic byte?
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         > Thanks,
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         > Mayuresh
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at
> > 8:58
> > > AM,
> > >     > > > Michael
> > >     > > > > > Pearce <
> > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> > >     > > > > >     > > >     >     >         > wrote:
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         >> Many thanks for this
> > > Mayuresh. I
> > >     > > don't
> > >     > > > > > have any
> > >     > > > > >     > > >     > objections.
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> I assume we should
> state:
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> That whilst sending
> > > tombstone and
> > >     > > non
> > >     > > > > null
> > >     > > > > >     > value,
> > >     > > > > >     > > > the
> > >     > > > > >     > > >     > consumer
> > >     > > > > >     > > >     >     > can expect
> > >     > > > > >     > > >     >     >         >> only to receive the
> > non-null
> > >     > message
> > >     > > > > only
> > >     > > > > > in
> > >     > > > > >     > step
> > >     > > > > >     > > > (3) is
> > >     > > > > >     > > >     > this
> > >     > > > > >     > > >     >     > correct?
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Cheers
> > >     > > > > >     > > >     >     >         >> Mike
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Sent using OWA for
> iPhone
> > >     > > > > >     > > >     >     >         >>
> > > ______________________________
> > >     > > > > __________
> > >     > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
> > >     > > > > >     > gharatmayuresh15@gmail.com
> > >     > > > > >     > > >
> > >     > > > > >     > > >     >     >         >> Sent: Thursday, November
> > > 17, 2016
> > >     > > > > 5:18:41
> > >     > > > > > PM
> > >     > > > > >     > > >     >     >         >> To:
> dev@kafka.apache.org
> > >     > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> > > KIP-87 -
> > >     > Add
> > >     > > > > > Compaction
> > >     > > > > >     > > > Tombstone
> > >     > > > > >     > > >     > Flag
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Hi Ismael,
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Thanks for the
> > explanation.
> > >     > > > > >     > > >     >     >         >> Specially I like this
> part
> > > where
> > >     > in
> > >     > > > you
> > >     > > > > >     > mentioned
> > >     > > > > >     > > > we can
> > >     > > > > >     > > >     > get
> > >     > > > > >     > > >     >     > rid of the
> > >     > > > > >     > > >     >     >         >> older null value support
> > > for log
> > >     > > > > > compaction
> > >     > > > > >     > later
> > >     > > > > >     > > > on,
> > >     > > > > >     > > >     > here :
> > >     > > > > >     > > >     >     >         >> We can't change
> semantics
> > > of the
> > >     > > > message
> > >     > > > > > format
> > >     > > > > >     > > > without
> > >     > > > > >     > > >     > having
> > >     > > > > >     > > >     >     > a long
> > >     > > > > >     > > >     >     >         >> transition period. And
> we
> > > can't
> > >     > rely
> > >     > > > > >     > > >     >     >         >> on people reading
> > > documentation or
> > >     > > > > acting
> > >     > > > > > on a
> > >     > > > > >     > > > warning for
> > >     > > > > >     > > >     >     > something so
> > >     > > > > >     > > >     >     >         >> fundamental. As such, my
> > > take is
> > >     > > that
> > >     > > > we
> > >     > > > > > need to
> > >     > > > > >     > > > bump the
> > >     > > > > >     > > >     > magic
> > >     > > > > >     > > >     >     > byte. The
> > >     > > > > >     > > >     >     >         >> good news is
> > >     > > > > >     > > >     >     >         >> that we don't have to
> > > support all
> > >     > > > > versions
> > >     > > > > >     > > forever.
> > >     > > > > >     > > > We
> > >     > > > > >     > > >     > have
> > >     > > > > >     > > >     >     > said that we
> > >     > > > > >     > > >     >     >         >> will support direct
> > > upgrades for 2
> > >     > > > > years.
> > >     > > > > > That
> > >     > > > > >     > > > means that
> > >     > > > > >     > > >     >     > message format
> > >     > > > > >     > > >     >     >         >> version n could, in
> > theory,
> > > be
> > >     > > > removed 2
> > >     > > > > > years
> > >     > > > > >     > > > after the
> > >     > > > > >     > > >     > it's
> > >     > > > > >     > > >     >     > introduced.
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Just a heads up, I would
> > > like to
> > >     > > > mention
> > >     > > > > > that
> > >     > > > > >     > even
> > >     > > > > >     > > > without
> > >     > > > > >     > > >     >     > bumping magic
> > >     > > > > >     > > >     >     >         >> byte, we will *NOT*
> loose
> > > zero
> > >     > copy
> > >     > > as
> > >     > > > > in
> > >     > > > > > the
> > >     > > > > >     > > > client(x+1)
> > >     > > > > >     > > >     > in my
> > >     > > > > >     > > >     >     >         >> explanation
> > >     > > > > >     > > >     >     >         >> above will convert
> > > internally a
> > >     > null
> > >     > > > > > value to
> > >     > > > > >     > > have a
> > >     > > > > >     > > >     > tombstone
> > >     > > > > >     > > >     >     > bit set and
> > >     > > > > >     > > >     >     >         >> a tombstone bit set to
> > have
> > > a null
> > >     > > > value
> > >     > > > > >     > > > automatically
> > >     > > > > >     > > >     >     > internally and by
> > >     > > > > >     > > >     >     >         >> the time we move to
> > version
> > > (x+2),
> > >     > > the
> > >     > > > > > clients
> > >     > > > > >     > > > would have
> > >     > > > > >     > > >     >     > upgraded.
> > >     > > > > >     > > >     >     >         >> Obviously if we support
> a
> > > request
> > >     > > from
> > >     > > > > >     > > consumer(x),
> > >     > > > > >     > > > we
> > >     > > > > >     > > >     > will
> > >     > > > > >     > > >     >     > loose zero
> > >     > > > > >     > > >     >     >         >> copy
> > >     > > > > >     > > >     >     >         >> but that is the same
> case
> > > with
> > >     > magic
> > >     > > > > byte.
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> But if magic byte bump
> > > makes life
> > >     > > > easier
> > >     > > > > > for
> > >     > > > > >     > > > transition
> > >     > > > > >     > > >     > for the
> > >     > > > > >     > > >     >     > above
> > >     > > > > >     > > >     >     >         >> reasons that you
> > explained,
> > > I am
> > >     > OK
> > >     > > > with
> > >     > > > > > it
> > >     > > > > >     > since
> > >     > > > > >     > > > we are
> > >     > > > > >     > > >     > going
> > >     > > > > >     > > >     >     > to meet the
> > >     > > > > >     > > >     >     >         >> end goal down the road
> :)
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> On a side note can we
> > > update the
> > >     > doc
> > >     > > > > here
> > >     > > > > > on
> > >     > > > > >     > magic
> > >     > > > > >     > > > byte
> > >     > > > > >     > > >     > to say
> > >     > > > > >     > > >     >     > that "*it
> > >     > > > > >     > > >     >     >         >> should be bumped
> whenever
> > > the
> > >     > > message
> > >     > > > > > format is
> > >     > > > > >     > > > changed
> > >     > > > > >     > > >     > or the
> > >     > > > > >     > > >     >     >         >> interpretation of
> message
> > > format
> > >     > > > (usage
> > >     > > > > > of the
> > >     > > > > >     > > > reserved
> > >     > > > > >     > > >     > bits as
> > >     > > > > >     > > >     >     > well) is
> > >     > > > > >     > > >     >     >         >> changed*".
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Hi Michael,
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Here is the update plan
> > > that we
> > >     > > > > discussed
> > >     > > > > >     > offline
> > >     > > > > >     > > >     > yesterday :
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Currently the magic-byte
> > > which
> > >     > > > > > corresponds to
> > >     > > > > >     > the
> > >     > > > > >     > > >     >     > "message.format.version"
> > >     > > > > >     > > >     >     >         >> is set to 1.
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> 1) On broker it will be
> > set
> > > to 1
> > >     > > > > > initially.
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> 2) When a producer
> client
> > > sends a
> > >     > > > > message
> > >     > > > > > with
> > >     > > > > >     > > > magic-byte
> > >     > > > > >     > > >     > = 2,
> > >     > > > > >     > > >     >     > since the
> > >     > > > > >     > > >     >     >         >> broker is on magic-byte
> =
> > > 1, we
> > >     > will
> > >     > > > > down
> > >     > > > > >     > convert
> > >     > > > > >     > > > it,
> > >     > > > > >     > > >     > which
> > >     > > > > >     > > >     >     > means if the
> > >     > > > > >     > > >     >     >         >> tombstone bit is set,
> the
> > > value
> > >     > will
> > >     > > > be
> > >     > > > > > set to
> > >     > > > > >     > > > null. A
> > >     > > > > >     > > >     > consumer
> > >     > > > > >     > > >     >     >         >> understanding
> magic-byte =
> > > 1, will
> > >     > > > still
> > >     > > > > > work
> > >     > > > > >     > with
> > >     > > > > >     > > > this. A
> > >     > > > > >     > > >     >     > consumer
> > >     > > > > >     > > >     >     >         >> working
> > >     > > > > >     > > >     >     >         >> with magic-byte =2 will
> > > also be
> > >     > able
> > >     > > > to
> > >     > > > > >     > understand
> > >     > > > > >     > > > this,
> > >     > > > > >     > > >     > since
> > >     > > > > >     > > >     >     > it
> > >     > > > > >     > > >     >     >         >> understands the
> tombstone.
> > >     > > > > >     > > >     >     >         >> Now there is still the
> > > question of
> > >     > > > > > supporting a
> > >     > > > > >     > > >     > non-tombstone
> > >     > > > > >     > > >     >     > and null
> > >     > > > > >     > > >     >     >         >> value from producer
> client
> > > with
> > >     > > > > > magic-byte = 2.*
> > >     > > > > >     > > (I
> > >     > > > > >     > > > am
> > >     > > > > >     > > >     > not sure
> > >     > > > > >     > > >     >     > if we
> > >     > > > > >     > > >     >     >         >> should support this.
> > > Ismael/Becket
> > >     > > can
> > >     > > > > > comment
> > >     > > > > >     > > > here)*
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> 3) When almost all the
> > > clients
> > >     > have
> > >     > > > > > upgraded,
> > >     > > > > >     > the
> > >     > > > > >     > > >     >     > message.format.version
> > >     > > > > >     > > >     >     >         >> on
> > >     > > > > >     > > >     >     >         >> the broker can be
> changed
> > > to 2,
> > >     > > where
> > >     > > > in
> > >     > > > > > the
> > >     > > > > >     > down
> > >     > > > > >     > > >     > conversion in
> > >     > > > > >     > > >     >     > the above
> > >     > > > > >     > > >     >     >         >> step will not happen. If
> > at
> > > this
> > >     > > point
> > >     > > > > we
> > >     > > > > > get a
> > >     > > > > >     > > > consumer
> > >     > > > > >     > > >     >     > request from a
> > >     > > > > >     > > >     >     >         >> older consumer, we might
> > > have to
> > >     > > down
> > >     > > > > > convert
> > >     > > > > >     > > where
> > >     > > > > >     > > > in we
> > >     > > > > >     > > >     > loose
> > >     > > > > >     > > >     >     > zero copy,
> > >     > > > > >     > > >     >     >         >> but these cases should
> be
> > > rare.
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Becket can you review
> this
> > > plan
> > >     > and
> > >     > > > add
> > >     > > > > > more
> > >     > > > > >     > > > details if I
> > >     > > > > >     > > >     > have
> > >     > > > > >     > > >     >     >         >> missed/wronged
> something,
> > > before
> > >     > we
> > >     > > > put
> > >     > > > > > it on
> > >     > > > > >     > KIP.
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Thanks,
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> Mayuresh
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> > > 11:07 PM,
> > >     > > > > Michael
> > >     > > > > >     > Pearce <
> > >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> > >     > > > > >     > > >     >     >         >> wrote:
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> > Thanks guys, for
> > > discussing this
> > >     > > > > > offline and
> > >     > > > > >     > > > getting
> > >     > > > > >     > > >     > some
> > >     > > > > >     > > >     >     > consensus.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > So its clear for
> myself
> > > and
> > >     > others
> > >     > > > > what
> > >     > > > > > is
> > >     > > > > >     > > > proposed now
> > >     > > > > >     > > >     > (i
> > >     > > > > >     > > >     >     > think i
> > >     > > > > >     > > >     >     >         >> > understand, but want
> to
> > > make
> > >     > sure)
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > Could i ask either
> > > directly
> > >     > update
> > >     > > > the
> > >     > > > > > kip to
> > >     > > > > >     > > > detail the
> > >     > > > > >     > > >     >     > migration
> > >     > > > > >     > > >     >     >         >> > strategy, or
> (re-)state
> > > your
> > >     > > offline
> > >     > > > > > discussed
> > >     > > > > >     > > and
> > >     > > > > >     > > >     > agreed
> > >     > > > > >     > > >     >     > migration
> > >     > > > > >     > > >     >     >         >> > strategy based on a
> > magic
> > > byte
> > >     > is
> > >     > > in
> > >     > > > > > this
> > >     > > > > >     > > thread.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > The main original
> driver
> > > for the
> > >     > > KIP
> > >     > > > > > was to
> > >     > > > > >     > > > support
> > >     > > > > >     > > >     >     > compaction where
> > >     > > > > >     > > >     >     >         >> value
> > >     > > > > >     > > >     >     >         >> > isn't null, based off
> > the
> > >     > > > discussions
> > >     > > > > on
> > >     > > > > >     > KIP-82
> > >     > > > > >     > > > thread.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > We should be able to
> > > support
> > >     > > > > > non-tombstone +
> > >     > > > > >     > > null
> > >     > > > > >     > > > value
> > >     > > > > >     > > >     > by the
> > >     > > > > >     > > >     >     >         >> completion
> > >     > > > > >     > > >     >     >         >> > of the KIP, as we
> noted
> > > when
> > >     > > > > discussing
> > >     > > > > > this
> > >     > > > > >     > > kip,
> > >     > > > > >     > > > having
> > >     > > > > >     > > >     >     > logic based on
> > >     > > > > >     > > >     >     >         >> a
> > >     > > > > >     > > >     >     >         >> > null value isn't very
> > > clean and
> > >     > > also
> > >     > > > > > separates
> > >     > > > > >     > > the
> > >     > > > > >     > > >     > concerns.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > As discussed already
> > > though we
> > >     > can
> > >     > > > > > split this
> > >     > > > > >     > > into
> > >     > > > > >     > > >     > KIP-87a
> > >     > > > > >     > > >     >     > and KIP-87b
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > Where we look to
> deliver
> > > KIP-87a
> > >     > > on
> > >     > > > a
> > >     > > > > >     > compacted
> > >     > > > > >     > > > topic
> > >     > > > > >     > > >     > (to
> > >     > > > > >     > > >     >     > address the
> > >     > > > > >     > > >     >     >         >> > immediate issues)
> > >     > > > > >     > > >     >     >         >> > * tombstone + null
> value
> > >     > > > > >     > > >     >     >         >> > * tombstone + non-null
> > > value
> > >     > > > > >     > > >     >     >         >> > * non-tombstone +
> > > non-null value
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > Then we can discuss
> once
> > > KIP-87a
> > >     > > is
> > >     > > > > > completed
> > >     > > > > >     > > > options
> > >     > > > > >     > > >     > later
> > >     > > > > >     > > >     >     > and how we
> > >     > > > > >     > > >     >     >         >> > support the second
> part
> > > KIP-87b
> > >     > to
> > >     > > > > > deliver:
> > >     > > > > >     > > >     >     >         >> > * non-tombstone + null
> > > value
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > Cheers
> > >     > > > > >     > > >     >     >         >> > Mike
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> >
> > > ______________________________
> > >     > > > > > __________
> > >     > > > > >     > > >     >     >         >> > From: Becket Qin <
> > >     > > > > becket.qin@gmail.com>
> > >     > > > > >     > > >     >     >         >> > Sent: Thursday,
> November
> > > 17,
> > >     > 2016
> > >     > > > 1:43
> > >     > > > > > AM
> > >     > > > > >     > > >     >     >         >> > To:
> > dev@kafka.apache.org
> > >     > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> > > KIP-87 -
> > >     > > Add
> > >     > > > > > Compaction
> > >     > > > > >     > > >     > Tombstone Flag
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and I
> had
> > > an
> > >     > > offline
> > >     > > > > >     > discussion,
> > >     > > > > >     > > > and
> > >     > > > > >     > > >     > following
> > >     > > > > >     > > >     >     > is a brief
> > >     > > > > >     > > >     >     >         >> > summary.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > 1. We agreed that not
> > > bumping up
> > >     > > > magic
> > >     > > > > > value
> > >     > > > > >     > may
> > >     > > > > >     > > > result
> > >     > > > > >     > > >     > in
> > >     > > > > >     > > >     >     > losing zero
> > >     > > > > >     > > >     >     >         >> copy
> > >     > > > > >     > > >     >     >         >> > during migration.
> > >     > > > > >     > > >     >     >         >> > 2. Given that bumping
> up
> > > magic
> > >     > > value
> > >     > > > > is
> > >     > > > > > almost
> > >     > > > > >     > > > free and
> > >     > > > > >     > > >     > has
> > >     > > > > >     > > >     >     > benefit of
> > >     > > > > >     > > >     >     >         >> > avoiding potential
> > > performance
> > >     > > > issue.
> > >     > > > > > It is
> > >     > > > > >     > > > probably
> > >     > > > > >     > > >     > worth
> > >     > > > > >     > > >     >     > doing.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > One issue we still
> need
> > > to think
> > >     > > > about
> > >     > > > > > is
> > >     > > > > >     > > whether
> > >     > > > > >     > > > we
> > >     > > > > >     > > >     > want to
> > >     > > > > >     > > >     >     > support a
> > >     > > > > >     > > >     >     >         >> > non-tombstone message
> > > with null
> > >     > > > value.
> > >     > > > > >     > > >     >     >         >> > Currently it is not
> > > supported by
> > >     > > > > Kafka.
> > >     > > > > > If we
> > >     > > > > >     > > > allow a
> > >     > > > > >     > > >     >     > non-tombstone null
> > >     > > > > >     > > >     >     >         >> > value message to exist
> > > after
> > >     > > KIP-87.
> > >     > > > > The
> > >     > > > > >     > problem
> > >     > > > > >     > > > is
> > >     > > > > >     > > >     > that such
> > >     > > > > >     > > >     >     > message
> > >     > > > > >     > > >     >     >         >> will
> > >     > > > > >     > > >     >     >         >> > not be supported by
> the
> > >     > consumers
> > >     > > > > prior
> > >     > > > > > to
> > >     > > > > >     > > KIP-87.
> > >     > > > > >     > > >     > Because a
> > >     > > > > >     > > >     >     > null value
> > >     > > > > >     > > >     >     >         >> > will always be
> > > interpreted to a
> > >     > > > > > tombstone.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > One option is that we
> > > keep the
> > >     > > > current
> > >     > > > > > way,
> > >     > > > > >     > i.e.
> > >     > > > > >     > > > do not
> > >     > > > > >     > > >     >     > support such
> > >     > > > > >     > > >     >     >         >> > message. It would be
> > good
> > > to
> > >     > know
> > >     > > if
> > >     > > > > > there is
> > >     > > > > >     > a
> > >     > > > > >     > > >     > concrete use
> > >     > > > > >     > > >     >     > case for
> > >     > > > > >     > > >     >     >         >> such
> > >     > > > > >     > > >     >     >         >> > message. If there is
> > not,
> > > we can
> > >     > > > > > probably just
> > >     > > > > >     > > not
> > >     > > > > >     > > >     > support it.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > Thanks,
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016
> at
> > > 1:28 PM,
> > >     > > > > > Mayuresh
> > >     > > > > >     > > Gharat <
> > >     > > > > >     > > >     >     >         >> >
> > > gharatmayuresh15@gmail.com
> > >     > > > > >     > > >     >     >         >> > > wrote:
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >> > > Hi Ismael,
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > > This is something I
> > can
> > > think
> > >     > of
> > >     > > > for
> > >     > > > > >     > migration
> > >     > > > > >     > > > plan:
> > >     > > > > >     > > >     >     >         >> > > So the migration
> plan
> > > can look
> > >     > > > > > something
> > >     > > > > >     > like
> > >     > > > > >     > > > this,
> > >     > > > > >     > > >     > with up
> > >     > > > > >     > > >     >     >         >> conversion :
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > > 1) Currently lets
> say
> > > we have
> > >     > > > Broker
> > >     > > > > > at
> > >     > > > > >     > > version
> > >     > > > > >     > > > x.
> > >     > > > > >     > > >     >     >         >> > > 2) Currently we have
> > > clients
> > >     > at
> > >     > > > > > version x.
> > >     > > > > >     > > >     >     >         >> > > 3) a) We move the
> > > version to
> > >     > > > > > Broker(x+1) :
> > >     > > > > >     > > > supports
> > >     > > > > >     > > >     > both
> > >     > > > > >     > > >     >     > tombstone and
> > >     > > > > >     > > >     >     >         >> > null
> > >     > > > > >     > > >     >     >         >> > > for log compaction.
> > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
> the
> > > client
> > >     > to
> > >     > > > > > version
> > >     > > > > >     > > > client(x+1) :
> > >     > > > > >     > > >     > if in
> > >     > > > > >     > > >     >     > the
> > >     > > > > >     > > >     >     >         >> producer
> > >     > > > > >     > > >     >     >         >> > > client(x+1) the
> value
> > > is set
> > >     > to
> > >     > > > > null,
> > >     > > > > > we
> > >     > > > > >     > will
> > >     > > > > >     > > >     > automatically
> > >     > > > > >     > > >     >     > set the
> > >     > > > > >     > > >     >     >         >> > > Tombstone bit
> > > internally. If
> > >     > the
> > >     > > > > > producer
> > >     > > > > >     > > > client(x+1)
> > >     > > > > >     > > >     > sets
> > >     > > > > >     > > >     >     > the
> > >     > > > > >     > > >     >     >         >> tombstone
> > >     > > > > >     > > >     >     >         >> > > itself, well and
> good.
> > > For
> > >     > > > producer
> > >     > > > > >     > client(x),
> > >     > > > > >     > > > the
> > >     > > > > >     > > >     > broker
> > >     > > > > >     > > >     >     > will up
> > >     > > > > >     > > >     >     >         >> convert
> > >     > > > > >     > > >     >     >         >> > > to have the
> tombstone
> > > bit.
> > >     > > > > > Broker(x+1) is
> > >     > > > > >     > > > supporting
> > >     > > > > >     > > >     > both.
> > >     > > > > >     > > >     >     > Consumer
> > >     > > > > >     > > >     >     >         >> > > client(x+1) will be
> > > aware of
> > >     > > this
> > >     > > > > and
> > >     > > > > > should
> > >     > > > > >     > > be
> > >     > > > > >     > > > able
> > >     > > > > >     > > >     > to
> > >     > > > > >     > > >     >     > handle this.
> > >     > > > > >     > > >     >     >         >> For
> > >     > > > > >     > > >     >     >         >> > > consumer client(x)
> we
> > > will
> > >     > down
> > >     > > > > > convert the
> > >     > > > > >     > > > message
> > >     > > > > >     > > >     > on the
> > >     > > > > >     > > >     >     > broker
> > >     > > > > >     > > >     >     >         >> side.
> > >     > > > > >     > > >     >     >         >> > >     c) At this point
> > we
> > > will
> > >     > > have
> > >     > > > to
> > >     > > > > >     > specify a
> > >     > > > > >     > > >     > warning or
> > >     > > > > >     > > >     >     > clearly
> > >     > > > > >     > > >     >     >         >> specify
> > >     > > > > >     > > >     >     >         >> > > in docs that this
> > > behavior is
> > >     > > > about
> > >     > > > > > to be
> > >     > > > > >     > > > changed for
> > >     > > > > >     > > >     > log
> > >     > > > > >     > > >     >     > compaction.
> > >     > > > > >     > > >     >     >         >> > > 4) a) In next
> release
> > > of the
> > >     > > > > > Broker(x+2), we
> > >     > > > > >     > > > say that
> > >     > > > > >     > > >     > only
> > >     > > > > >     > > >     >     > Tombstone
> > >     > > > > >     > > >     >     >         >> is
> > >     > > > > >     > > >     >     >         >> > > used for log
> > compaction
> > > on the
> > >     > > > > Broker
> > >     > > > > > side.
> > >     > > > > >     > > >     > Clients(x+1)
> > >     > > > > >     > > >     >     > still is
> > >     > > > > >     > > >     >     >         >> > > supported.
> > >     > > > > >     > > >     >     >         >> > >     b) We upgrade
> the
> > > client
> > >     > to
> > >     > > > > > version
> > >     > > > > >     > > > client(x+2) :
> > >     > > > > >     > > >     > if
> > >     > > > > >     > > >     >     > value is set
> > >     > > > > >     > > >     >     >         >> to
> > >     > > > > >     > > >     >     >         >> > > null, tombstone will
> > > not be
> > >     > set
> > >     > > > > >     > automatically.
> > >     > > > > >     > > > The
> > >     > > > > >     > > >     > client
> > >     > > > > >     > > >     >     > will have to
> > >     > > > > >     > > >     >     >         >> > call
> > >     > > > > >     > > >     >     >         >> > > setTombstone() to
> > > actually set
> > >     > > the
> > >     > > > > >     > tombstone.
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > > We should compare
> this
> > >     > migration
> > >     > > > > plan
> > >     > > > > > with
> > >     > > > > >     > the
> > >     > > > > >     > > >     > migration
> > >     > > > > >     > > >     >     > plan for
> > >     > > > > >     > > >     >     >         >> magic
> > >     > > > > >     > > >     >     >         >> > > byte bump and do
> > > whatever
> > >     > looks
> > >     > > > > good.
> > >     > > > > >     > > >     >     >         >> > > I am just worried
> that
> > > if we
> > >     > go
> > >     > > > down
> > >     > > > > > magic
> > >     > > > > >     > > byte
> > >     > > > > >     > > > route,
> > >     > > > > >     > > >     >     > unless I am
> > >     > > > > >     > > >     >     >         >> > missing
> > >     > > > > >     > > >     >     >         >> > > something, it sounds
> > > like
> > >     > kafka
> > >     > > > will
> > >     > > > > > be
> > >     > > > > >     > stuck
> > >     > > > > >     > > > with
> > >     > > > > >     > > >     >     > supporting both
> > >     > > > > >     > > >     >     >         >> null
> > >     > > > > >     > > >     >     >         >> > > value and tombstone
> > bit
> > > for
> > >     > log
> > >     > > > > > compaction
> > >     > > > > >     > for
> > >     > > > > >     > > > life
> > >     > > > > >     > > >     > long,
> > >     > > > > >     > > >     >     > which does
> > >     > > > > >     > > >     >     >         >> not
> > >     > > > > >     > > >     >     >         >> > > look like a good end
> > > state.
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > > Thanks,
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > > Mayuresh
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016
> > at
> > > 9:32
> > >     > AM,
> > >     > > > > > Mayuresh
> > >     > > > > >     > > > Gharat <
> > >     > > > > >     > > >     >     >         >> > >
> > > gharatmayuresh15@gmail.com
> > >     > > > > >     > > >     >     >         >> > > > wrote:
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > That's a very good
> > > point
> > >     > > which I
> > >     > > > > > might
> > >     > > > > >     > have
> > >     > > > > >     > > > not
> > >     > > > > >     > > >     >     > considered earlier.
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > Here is a plan
> that
> > I
> > > can
> > >     > > think
> > >     > > > > of:
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > Stage 1) The
> broker
> > > from now
> > >     > > on,
> > >     > > > > up
> > >     > > > > >     > converts
> > >     > > > > >     > > > the
> > >     > > > > >     > > >     > message
> > >     > > > > >     > > >     >     > to have the
> > >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> > The
> > > log
> > >     > > > > compaction
> > >     > > > > >     > thread
> > >     > > > > >     > > > does log
> > >     > > > > >     > > >     >     > compaction
> > >     > > > > >     > > >     >     >         >> based
> > >     > > > > >     > > >     >     >         >> > on
> > >     > > > > >     > > >     >     >         >> > > > both null and
> > > tombstone
> > >     > > marker.
> > >     > > > > > This is
> > >     > > > > >     > our
> > >     > > > > >     > > >     > transition
> > >     > > > > >     > > >     >     > period.
> > >     > > > > >     > > >     >     >         >> > > > Stage 2) The next
> > > release we
> > >     > > > only
> > >     > > > > > say that
> > >     > > > > >     > > log
> > >     > > > > >     > > >     > compaction
> > >     > > > > >     > > >     >     > is based
> > >     > > > > >     > > >     >     >         >> on
> > >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> > > (Open
> > >     > source
> > >     > > > > > kafka makes
> > >     > > > > >     > > > this as a
> > >     > > > > >     > > >     >     > policy). By
> > >     > > > > >     > > >     >     >         >> this
> > >     > > > > >     > > >     >     >         >> > > time,
> > >     > > > > >     > > >     >     >         >> > > > the organization
> > > which is
> > >     > > moving
> > >     > > > > to
> > >     > > > > > this
> > >     > > > > >     > > > release
> > >     > > > > >     > > >     > will be
> > >     > > > > >     > > >     >     > sure that
> > >     > > > > >     > > >     >     >         >> they
> > >     > > > > >     > > >     >     >         >> > > > have gone through
> > the
> > > entire
> > >     > > > > > transition
> > >     > > > > >     > > > period.
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > My only goal of
> > doing
> > > this
> > >     > is
> > >     > > > that
> > >     > > > > > Kafka
> > >     > > > > >     > > > clearly
> > >     > > > > >     > > >     >     > specifies the end
> > >     > > > > >     > > >     >     >         >> > state
> > >     > > > > >     > > >     >     >         >> > > > about what log
> > > compaction
> > >     > > means
> > >     > > > > (is
> > >     > > > > > it
> > >     > > > > >     > null
> > >     > > > > >     > > > value
> > >     > > > > >     > > >     > or a
> > >     > > > > >     > > >     >     > tombstone
> > >     > > > > >     > > >     >     >         >> > marker,
> > >     > > > > >     > > >     >     >         >> > > > but not both).
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > What do you think?
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > Thanks,
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > Mayuresh
> > >     > > > > >     > > >     >     >         >> > > > .
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16,
> 2016
> > > at 9:17
> > >     > > AM,
> > >     > > > > > Ismael
> > >     > > > > >     > > Juma <
> > >     > > > > >     > > >     >     > ismael@juma.me.uk>
> > >     > > > > >     > > >     >     >         >> > wrote:
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > >> One comment
> below.
> > >     > > > > >     > > >     >     >         >> > > >>
> > >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16,
> > 2016
> > > at
> > >     > 5:08
> > >     > > > PM,
> > >     > > > > > Mayuresh
> > >     > > > > >     > > > Gharat <
> > >     > > > > >     > > >     >     >         >> > > >>
> > > gharatmayuresh15@gmail.com
> > >     > > > > >     > > >     >     >         >> > > >> > wrote:
> > >     > > > > >     > > >     >     >         >> > > >>
> > >     > > > > >     > > >     >     >         >> > > >> >    - If we
> don't
> > > bump up
> > >     > > the
> > >     > > > > > magic
> > >     > > > > >     > byte,
> > >     > > > > >     > > > on the
> > >     > > > > >     > > >     > broker
> > >     > > > > >     > > >     >     > side, the
> > >     > > > > >     > > >     >     >         >> > > broker
> > >     > > > > >     > > >     >     >         >> > > >> >    will always
> > > have to
> > >     > look
> > >     > > > at
> > >     > > > > > both
> > >     > > > > >     > > > tombstone
> > >     > > > > >     > > >     > bit and
> > >     > > > > >     > > >     >     > the value
> > >     > > > > >     > > >     >     >         >> when
> > >     > > > > >     > > >     >     >         >> > > do
> > >     > > > > >     > > >     >     >         >> > > >> the
> > >     > > > > >     > > >     >     >         >> > > >> >    compaction.
> > > Assuming
> > >     > we
> > >     > > do
> > >     > > > > > not bump
> > >     > > > > >     > up
> > >     > > > > >     > > > the
> > >     > > > > >     > > >     > magic
> > >     > > > > >     > > >     >     > byte,
> > >     > > > > >     > > >     >     >         >> > > >> >    imagine the
> > > broker
> > >     > sees
> > >     > > a
> > >     > > > > > message
> > >     > > > > >     > > which
> > >     > > > > >     > > > does
> > >     > > > > >     > > >     > not
> > >     > > > > >     > > >     >     > have a
> > >     > > > > >     > > >     >     >         >> tombstone
> > >     > > > > >     > > >     >     >         >> > > bit
> > >     > > > > >     > > >     >     >         >> > > >> >    set. The
> > broker
> > > does
> > >     > not
> > >     > > > > know
> > >     > > > > > when
> > >     > > > > >     > the
> > >     > > > > >     > > >     > message was
> > >     > > > > >     > > >     >     > produced
> > >     > > > > >     > > >     >     >         >> (i.e.
> > >     > > > > >     > > >     >     >         >> > > >> > whether
> > >     > > > > >     > > >     >     >         >> > > >> >    the message
> > has
> > > been
> > >     > up
> > >     > > > > > converted or
> > >     > > > > >     > > > not), it
> > >     > > > > >     > > >     > has
> > >     > > > > >     > > >     >     > to take a
> > >     > > > > >     > > >     >     >         >> > further
> > >     > > > > >     > > >     >     >         >> > > >> > look at
> > >     > > > > >     > > >     >     >         >> > > >> >    the value to
> > > see if it
> > >     > > is
> > >     > > > > > null or
> > >     > > > > >     > not
> > >     > > > > >     > > in
> > >     > > > > >     > > >     > order to
> > >     > > > > >     > > >     >     > determine
> > >     > > > > >     > > >     >     >         >> if it
> > >     > > > > >     > > >     >     >         >> > > is
> > >     > > > > >     > > >     >     >         >> > > >> a
> > >     > > > > >     > > >     >     >         >> > > >> >    tombstone.
> The
> > > same
> > >     > > logic
> > >     > > > > has
> > >     > > > > > to be
> > >     > > > > >     > > put
> > >     > > > > >     > > > on the
> > >     > > > > >     > > >     >     > consumer as
> > >     > > > > >     > > >     >     >         >> well
> > >     > > > > >     > > >     >     >         >> > > >> because
> > >     > > > > >     > > >     >     >         >> > > >> > the
> > >     > > > > >     > > >     >     >         >> > > >> >    consumer
> does
> > > not know
> > >     > > if
> > >     > > > > the
> > >     > > > > >     > message
> > >     > > > > >     > > > has
> > >     > > > > >     > > >     > been up
> > >     > > > > >     > > >     >     > converted or
> > >     > > > > >     > > >     >     >         >> > not.
> > >     > > > > >     > > >     >     >         >> > > >> >       - If we
> > > upconvert
> > >     > > while
> > >     > > > > >     > appending,
> > >     > > > > >     > > > this is
> > >     > > > > >     > > >     > not
> > >     > > > > >     > > >     >     > the case,
> > >     > > > > >     > > >     >     >         >> > right?
> > >     > > > > >     > > >     >     >         >> > > >>
> > >     > > > > >     > > >     >     >         >> > > >>
> > >     > > > > >     > > >     >     >         >> > > >> If I understand
> you
> > >     > > correctly,
> > >     > > > > > this is
> > >     > > > > >     > not
> > >     > > > > >     > > >     > sufficient
> > >     > > > > >     > > >     >     > because the
> > >     > > > > >     > > >     >     >         >> log
> > >     > > > > >     > > >     >     >         >> > > may
> > >     > > > > >     > > >     >     >         >> > > >> have messages
> > > appended
> > >     > before
> > >     > > > it
> > >     > > > > > was
> > >     > > > > >     > > > upgraded to
> > >     > > > > >     > > >     > include
> > >     > > > > >     > > >     >     > KIP-87.
> > >     > > > > >     > > >     >     >         >> > > >>
> > >     > > > > >     > > >     >     >         >> > > >> Ismael
> > >     > > > > >     > > >     >     >         >> > > >>
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > > > --
> > >     > > > > >     > > >     >     >         >> > > > -Regards,
> > >     > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> > >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
> > >     > > > > >     > > >     >     >         >> > > >
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > > --
> > >     > > > > >     > > >     >     >         >> > > -Regards,
> > >     > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > >     > > > > >     > > >     >     >         >> > > (862) 250-7125
> > >     > > > > >     > > >     >     >         >> > >
> > >     > > > > >     > > >     >     >         >> > The information
> > contained
> > > in
> > >     > this
> > >     > > > > email
> > >     > > > > > is
> > >     > > > > >     > > > strictly
> > >     > > > > >     > > >     >     > confidential and for
> > >     > > > > >     > > >     >     >         >> > the use of the
> addressee
> > > only,
> > >     > > > unless
> > >     > > > > >     > otherwise
> > >     > > > > >     > > >     > indicated. If
> > >     > > > > >     > > >     >     > you are
> > >     > > > > >     > > >     >     >         >> not
> > >     > > > > >     > > >     >     >         >> > the intended
> recipient,
> > > please
> > >     > do
> > >     > > > not
> > >     > > > > > read,
> > >     > > > > >     > > copy,
> > >     > > > > >     > > > use or
> > >     > > > > >     > > >     >     > disclose to
> > >     > > > > >     > > >     >     >         >> others
> > >     > > > > >     > > >     >     >         >> > this message or any
> > > attachment.
> > >     > > > Please
> > >     > > > > > also
> > >     > > > > >     > > > notify the
> > >     > > > > >     > > >     > sender
> > >     > > > > >     > > >     >     > by
> > >     > > > > >     > > >     >     >         >> replying
> > >     > > > > >     > > >     >     >         >> > to this email or by
> > > telephone
> > >     > > > (+44(020
> > >     > > > > > 7896
> > >     > > > > >     > > 0011)
> > >     > > > > >     > > > and
> > >     > > > > >     > > >     > then
> > >     > > > > >     > > >     >     > delete the
> > >     > > > > >     > > >     >     >         >> email
> > >     > > > > >     > > >     >     >         >> > and any copies of it.
> > > Opinions,
> > >     > > > > > conclusion
> > >     > > > > >     > (etc)
> > >     > > > > >     > > > that
> > >     > > > > >     > > >     > do not
> > >     > > > > >     > > >     >     > relate to
> > >     > > > > >     > > >     >     >         >> the
> > >     > > > > >     > > >     >     >         >> > official business of
> > this
> > >     > company
> > >     > > > > shall
> > >     > > > > > be
> > >     > > > > >     > > > understood as
> > >     > > > > >     > > >     >     > neither given
> > >     > > > > >     > > >     >     >         >> nor
> > >     > > > > >     > > >     >     >         >> > endorsed by it. IG is
> a
> > > trading
> > >     > > name
> > >     > > > > of
> > >     > > > > > IG
> > >     > > > > >     > > Markets
> > >     > > > > >     > > >     > Limited (a
> > >     > > > > >     > > >     >     > company
> > >     > > > > >     > > >     >     >         >> > registered in England
> > and
> > > Wales,
> > >     > > > > company
> > >     > > > > >     > number
> > >     > > > > >     > > >     > 04008957) and
> > >     > > > > >     > > >     >     > IG Index
> > >     > > > > >     > > >     >     >         >> > Limited (a company
> > > registered in
> > >     > > > > > England and
> > >     > > > > >     > > > Wales,
> > >     > > > > >     > > >     > company
> > >     > > > > >     > > >     >     > number
> > >     > > > > >     > > >     >     >         >> > 01190902). Registered
> > > address at
> > >     > > > > Cannon
> > >     > > > > > Bridge
> > >     > > > > >     > > > House, 25
> > >     > > > > >     > > >     >     > Dowgate Hill,
> > >     > > > > >     > > >     >     >         >> > London EC4R 2YA. Both
> IG
> > > Markets
> > >     > > > > Limited
> > >     > > > > >     > > (register
> > >     > > > > >     > > >     > number
> > >     > > > > >     > > >     >     > 195355) and IG
> > >     > > > > >     > > >     >     >         >> > Index Limited
> (register
> > > number
> > >     > > > 114059)
> > >     > > > > > are
> > >     > > > > >     > > > authorised
> > >     > > > > >     > > >     > and
> > >     > > > > >     > > >     >     > regulated by
> > >     > > > > >     > > >     >     >         >> the
> > >     > > > > >     > > >     >     >         >> > Financial Conduct
> > > Authority.
> > >     > > > > >     > > >     >     >         >> >
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >> --
> > >     > > > > >     > > >     >     >         >> -Regards,
> > >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
> > >     > > > > >     > > >     >     >         >> (862) 250-7125
> > >     > > > > >     > > >     >     >         >> The information
> contained
> > > in this
> > >     > > > email
> > >     > > > > is
> > >     > > > > >     > > strictly
> > >     > > > > >     > > >     >     > confidential and for
> > >     > > > > >     > > >     >     >         >> the use of the addressee
> > > only,
> > >     > > unless
> > >     > > > > > otherwise
> > >     > > > > >     > > >     > indicated. If
> > >     > > > > >     > > >     >     > you are not
> > >     > > > > >     > > >     >     >         >> the intended recipient,
> > > please do
> > >     > > not
> > >     > > > > > read,
> > >     > > > > >     > copy,
> > >     > > > > >     > > > use or
> > >     > > > > >     > > >     >     > disclose to others
> > >     > > > > >     > > >     >     >         >> this message or any
> > > attachment.
> > >     > > Please
> > >     > > > > > also
> > >     > > > > >     > notify
> > >     > > > > >     > > > the
> > >     > > > > >     > > >     > sender
> > >     > > > > >     > > >     >     > by replying
> > >     > > > > >     > > >     >     >         >> to this email or by
> > > telephone
> > >     > > (+44(020
> > >     > > > > > 7896
> > >     > > > > >     > 0011)
> > >     > > > > >     > > > and then
> > >     > > > > >     > > >     >     > delete the email
> > >     > > > > >     > > >     >     >         >> and any copies of it.
> > > Opinions,
> > >     > > > > > conclusion (etc)
> > >     > > > > >     > > > that do
> > >     > > > > >     > > >     > not
> > >     > > > > >     > > >     >     > relate to the
> > >     > > > > >     > > >     >     >         >> official business of
> this
> > > company
> > >     > > > shall
> > >     > > > > be
> > >     > > > > >     > > > understood as
> > >     > > > > >     > > >     >     > neither given nor
> > >     > > > > >     > > >     >     >         >> endorsed by it. IG is a
> > > trading
> > >     > name
> > >     > > > of
> > >     > > > > IG
> > >     > > > > >     > Markets
> > >     > > > > >     > > >     > Limited (a
> > >     > > > > >     > > >     >     > company
> > >     > > > > >     > > >     >     >         >> registered in England
> and
> > > Wales,
> > >     > > > company
> > >     > > > > > number
> > >     > > > > >     > > > 04008957)
> > >     > > > > >     > > >     > and
> > >     > > > > >     > > >     >     > IG Index
> > >     > > > > >     > > >     >     >         >> Limited (a company
> > > registered in
> > >     > > > England
> > >     > > > > > and
> > >     > > > > >     > > Wales,
> > >     > > > > >     > > >     > company
> > >     > > > > >     > > >     >     > number
> > >     > > > > >     > > >     >     >         >> 01190902). Registered
> > > address at
> > >     > > > Cannon
> > >     > > > > > Bridge
> > >     > > > > >     > > > House, 25
> > >     > > > > >     > > >     >     > Dowgate Hill,
> > >     > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> > > Markets
> > >     > > > Limited
> > >     > > > > >     > (register
> > >     > > > > >     > > > number
> > >     > > > > >     > > >     >     > 195355) and IG
> > >     > > > > >     > > >     >     >         >> Index Limited (register
> > > number
> > >     > > 114059)
> > >     > > > > are
> > >     > > > > >     > > > authorised and
> > >     > > > > >     > > >     >     > regulated by the
> > >     > > > > >     > > >     >     >         >> Financial Conduct
> > Authority.
> > >     > > > > >     > > >     >     >         >>
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >         > --
> > >     > > > > >     > > >     >     >         > -Regards,
> > >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
> > >     > > > > >     > > >     >     >         > (862) 250-7125
> > >     > > > > >     > > >     >     >         >
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >         --
> > >     > > > > >     > > >     >     >         -Regards,
> > >     > > > > >     > > >     >     >         Mayuresh R. Gharat
> > >     > > > > >     > > >     >     >         (862) 250-7125
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >     The information contained in
> this
> > > email
> > >     > is
> > >     > > > > > strictly
> > >     > > > > >     > > > confidential
> > >     > > > > >     > > >     > and
> > >     > > > > >     > > >     >     > for the use of the addressee only,
> > > unless
> > >     > > > otherwise
> > >     > > > > >     > > indicated.
> > >     > > > > >     > > > If
> > >     > > > > >     > > >     > you are
> > >     > > > > >     > > >     >     > not the intended recipient, please
> do
> > > not
> > >     > read,
> > >     > > > > > copy, use
> > >     > > > > >     > or
> > >     > > > > >     > > >     > disclose to
> > >     > > > > >     > > >     >     > others this message or any
> > attachment.
> > > Please
> > >     > > > also
> > >     > > > > > notify
> > >     > > > > >     > the
> > >     > > > > >     > > > sender
> > >     > > > > >     > > >     > by
> > >     > > > > >     > > >     >     > replying to this email or by
> > telephone
> > >     > (+44(020
> > >     > > > > 7896
> > >     > > > > > 0011)
> > >     > > > > >     > > and
> > >     > > > > >     > > > then
> > >     > > > > >     > > >     > delete
> > >     > > > > >     > > >     >     > the email and any copies of it.
> > > Opinions,
> > >     > > > > conclusion
> > >     > > > > > (etc)
> > >     > > > > >     > > > that do
> > >     > > > > >     > > >     > not
> > >     > > > > >     > > >     >     > relate to the official business of
> > this
> > >     > company
> > >     > > > > > shall be
> > >     > > > > >     > > > understood
> > >     > > > > >     > > >     > as
> > >     > > > > >     > > >     >     > neither given nor endorsed by it.
> IG
> > > is a
> > >     > > trading
> > >     > > > > > name of
> > >     > > > > >     > IG
> > >     > > > > >     > > > Markets
> > >     > > > > >     > > >     >     > Limited (a company registered in
> > > England and
> > >     > > > Wales,
> > >     > > > > > company
> > >     > > > > >     > > > number
> > >     > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> > > company
> > >     > > > > registered
> > >     > > > > > in
> > >     > > > > >     > > > England and
> > >     > > > > >     > > >     > Wales,
> > >     > > > > >     > > >     >     > company number 01190902).
> Registered
> > > address
> > >     > at
> > >     > > > > > Cannon
> > >     > > > > >     > Bridge
> > >     > > > > >     > > > House,
> > >     > > > > >     > > >     > 25
> > >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both
> > IG
> > >     > Markets
> > >     > > > > > Limited
> > >     > > > > >     > > > (register
> > >     > > > > >     > > >     > number
> > >     > > > > >     > > >     >     > 195355) and IG Index Limited
> > (register
> > > number
> > >     > > > > > 114059) are
> > >     > > > > >     > > > authorised
> > >     > > > > >     > > >     > and
> > >     > > > > >     > > >     >     > regulated by the Financial Conduct
> > > Authority.
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >     >
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >     > The information contained in this email
> is
> > > strictly
> > >     > > > > > confidential
> > >     > > > > >     > > and
> > >     > > > > >     > > > for
> > >     > > > > >     > > >     > the use of the addressee only, unless
> > > otherwise
> > >     > > > > indicated.
> > >     > > > > > If you
> > >     > > > > >     > > > are not
> > >     > > > > >     > > >     > the intended recipient, please do not
> read,
> > > copy,
> > >     > use
> > >     > > > or
> > >     > > > > > disclose
> > >     > > > > >     > > to
> > >     > > > > >     > > > others
> > >     > > > > >     > > >     > this message or any attachment. Please
> also
> > > notify
> > >     > > the
> > >     > > > > > sender by
> > >     > > > > >     > > > replying
> > >     > > > > >     > > >     > to this email or by telephone (+44(020
> 7896
> > > 0011
> > >     > <020%207896%200011>) and
> > >     > > > > then
> > >     > > > > > delete
> > >     > > > > >     > > > the email
> > >     > > > > >     > > >     > and any copies of it. Opinions,
> conclusion
> > > (etc)
> > >     > that
> > >     > > > do
> > >     > > > > > not
> > >     > > > > >     > relate
> > >     > > > > >     > > > to the
> > >     > > > > >     > > >     > official business of this company shall
> be
> > >     > understood
> > >     > > > as
> > >     > > > > > neither
> > >     > > > > >     > > > given nor
> > >     > > > > >     > > >     > endorsed by it. IG is a trading name of
> IG
> > > Markets
> > >     > > > > Limited
> > >     > > > > > (a
> > >     > > > > >     > > company
> > >     > > > > >     > > >     > registered in England and Wales, company
> > > number
> > >     > > > 04008957)
> > >     > > > > > and IG
> > >     > > > > >     > > > Index
> > >     > > > > >     > > >     > Limited (a company registered in England
> > and
> > > Wales,
> > >     > > > > company
> > >     > > > > >     > number
> > >     > > > > >     > > >     > 01190902). Registered address at Cannon
> > > Bridge
> > >     > House,
> > >     > > > 25
> > >     > > > > > Dowgate
> > >     > > > > >     > > > Hill,
> > >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
> > > (register
> > >     > > > number
> > >     > > > > > 195355)
> > >     > > > > >     > > > and IG
> > >     > > > > >     > > >     > Index Limited (register number 114059)
> are
> > >     > authorised
> > >     > > > and
> > >     > > > > >     > regulated
> > >     > > > > >     > > > by the
> > >     > > > > >     > > >     > Financial Conduct Authority.
> > >     > > > > >     > > >     >
> > >     > > > > >     > > >
> > >     > > > > >     > > >
> > >     > > > > >     > > > The information contained in this email is
> > strictly
> > >     > > > > confidential
> > >     > > > > > and
> > >     > > > > >     > for
> > >     > > > > >     > > > the use of the addressee only, unless otherwise
> > >     > indicated.
> > >     > > If
> > >     > > > > > you are
> > >     > > > > >     > not
> > >     > > > > >     > > > the intended recipient, please do not read,
> copy,
> > > use or
> > >     > > > > > disclose to
> > >     > > > > >     > > others
> > >     > > > > >     > > > this message or any attachment. Please also
> > notify
> > > the
> > >     > > sender
> > >     > > > > by
> > >     > > > > >     > replying
> > >     > > > > >     > > > to this email or by telephone (+44(020 7896
> 0011
> > >     > <020%207896%200011>) and then
> > >     > > > > > delete the
> > >     > > > > >     > > email
> > >     > > > > >     > > > and any copies of it. Opinions, conclusion
> (etc)
> > > that do
> > >     > > not
> > >     > > > > > relate to
> > >     > > > > >     > > the
> > >     > > > > >     > > > official business of this company shall be
> > > understood as
> > >     > > > > neither
> > >     > > > > > given
> > >     > > > > >     > > nor
> > >     > > > > >     > > > endorsed by it. IG is a trading name of IG
> > Markets
> > >     > Limited
> > >     > > (a
> > >     > > > > > company
> > >     > > > > >     > > > registered in England and Wales, company number
> > > 04008957)
> > >     > > and
> > >     > > > > IG
> > >     > > > > > Index
> > >     > > > > >     > > > Limited (a company registered in England and
> > Wales,
> > >     > company
> > >     > > > > > number
> > >     > > > > >     > > > 01190902). Registered address at Cannon Bridge
> > > House, 25
> > >     > > > > Dowgate
> > >     > > > > > Hill,
> > >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > >     > > > > > 195355) and
> > >     > > > > >     > IG
> > >     > > > > >     > > > Index Limited (register number 114059) are
> > > authorised and
> > >     > > > > > regulated by
> > >     > > > > >     > > the
> > >     > > > > >     > > > Financial Conduct Authority.
> > >     > > > > >     > > >
> > >     > > > > >     > > The information contained in this email is
> strictly
> > >     > > > confidential
> > >     > > > > > and for
> > >     > > > > >     > > the use of the addressee only, unless otherwise
> > > indicated.
> > >     > If
> > >     > > > you
> > >     > > > > > are not
> > >     > > > > >     > > the intended recipient, please do not read, copy,
> > > use or
> > >     > > > disclose
> > >     > > > > > to
> > >     > > > > >     > others
> > >     > > > > >     > > this message or any attachment. Please also
> notify
> > > the
> > >     > sender
> > >     > > > by
> > >     > > > > > replying
> > >     > > > > >     > > to this email or by telephone (+44(020 7896 0011
> > >     > <020%207896%200011>) and then
> > >     > > > delete
> > >     > > > > > the
> > >     > > > > >     > email
> > >     > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> > > that do
> > >     > not
> > >     > > > > > relate to
> > >     > > > > >     > the
> > >     > > > > >     > > official business of this company shall be
> > > understood as
> > >     > > > neither
> > >     > > > > > given
> > >     > > > > >     > nor
> > >     > > > > >     > > endorsed by it. IG is a trading name of IG
> Markets
> > > Limited
> > >     > (a
> > >     > > > > > company
> > >     > > > > >     > > registered in England and Wales, company number
> > > 04008957)
> > >     > and
> > >     > > > IG
> > >     > > > > > Index
> > >     > > > > >     > > Limited (a company registered in England and
> Wales,
> > > company
> > >     > > > > number
> > >     > > > > >     > > 01190902). Registered address at Cannon Bridge
> > > House, 25
> > >     > > > Dowgate
> > >     > > > > > Hill,
> > >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited
> (register
> > > number
> > >     > > > 195355)
> > >     > > > > > and IG
> > >     > > > > >     > > Index Limited (register number 114059) are
> > > authorised and
> > >     > > > > > regulated by
> > >     > > > > >     > the
> > >     > > > > >     > > Financial Conduct Authority.
> > >     > > > > >     > >
> > >     > > > > >     > >
> > >     > > > > >     > The information contained in this email is strictly
> > >     > > confidential
> > >     > > > > and
> > >     > > > > > for
> > >     > > > > >     > the use of the addressee only, unless otherwise
> > > indicated. If
> > >     > > you
> > >     > > > > > are not
> > >     > > > > >     > the intended recipient, please do not read, copy,
> use
> > > or
> > >     > > disclose
> > >     > > > > to
> > >     > > > > > others
> > >     > > > > >     > this message or any attachment. Please also notify
> > the
> > > sender
> > >     > > by
> > >     > > > > > replying
> > >     > > > > >     > to this email or by telephone (+44(020 7896 0011
> > >     > <020%207896%200011>) and then
> > >     > > delete
> > >     > > > > > the email
> > >     > > > > >     > and any copies of it. Opinions, conclusion (etc)
> that
> > > do not
> > >     > > > relate
> > >     > > > > > to the
> > >     > > > > >     > official business of this company shall be
> understood
> > > as
> > >     > > neither
> > >     > > > > > given nor
> > >     > > > > >     > endorsed by it. IG is a trading name of IG Markets
> > > Limited (a
> > >     > > > > company
> > >     > > > > >     > registered in England and Wales, company number
> > > 04008957) and
> > >     > > IG
> > >     > > > > > Index
> > >     > > > > >     > Limited (a company registered in England and Wales,
> > > company
> > >     > > > number
> > >     > > > > >     > 01190902). Registered address at Cannon Bridge
> House,
> > > 25
> > >     > > Dowgate
> > >     > > > > > Hill,
> > >     > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > > number
> > >     > > 195355)
> > >     > > > > > and IG
> > >     > > > > >     > Index Limited (register number 114059) are
> authorised
> > > and
> > >     > > > regulated
> > >     > > > > > by the
> > >     > > > > >     > Financial Conduct Authority.
> > >     > > > > >     >
> > >     > > > > >     >
> > >     > > > > >
> > >     > > > > >
> > >     > > > > > The information contained in this email is strictly
> > > confidential
> > >     > and
> > >     > > > for
> > >     > > > > > the use of the addressee only, unless otherwise
> indicated.
> > > If you
> > >     > are
> > >     > > > not
> > >     > > > > > the intended recipient, please do not read, copy, use or
> > > disclose
> > >     > to
> > >     > > > > others
> > >     > > > > > this message or any attachment. Please also notify the
> > > sender by
> > >     > > > replying
> > >     > > > > > to this email or by telephone (+44(020 7896 0011
> > >     > <020%207896%200011>) and then delete the
> > >     > > > > email
> > >     > > > > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > relate
> > >     > > to
> > >     > > > > the
> > >     > > > > > official business of this company shall be understood as
> > > neither
> > >     > > given
> > >     > > > > nor
> > >     > > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > >     > company
> > >     > > > > > registered in England and Wales, company number 04008957)
> > > and IG
> > >     > > Index
> > >     > > > > > Limited (a company registered in England and Wales,
> company
> > > number
> > >     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > >     > > Hill,
> > >     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355)
> > >     > and
> > >     > > > IG
> > >     > > > > > Index Limited (register number 114059) are authorised and
> > > regulated
> > >     > > by
> > >     > > > > the
> > >     > > > > > Financial Conduct Authority.
> > >     > > > > >
> > >     > > > > The information contained in this email is strictly
> > > confidential and
> > >     > > for
> > >     > > > > the use of the addressee only, unless otherwise indicated.
> If
> > > you are
> > >     > > not
> > >     > > > > the intended recipient, please do not read, copy, use or
> > > disclose to
> > >     > > > others
> > >     > > > > this message or any attachment. Please also notify the
> sender
> > > by
> > >     > > replying
> > >     > > > > to this email or by telephone (+44(020 7896 0011
> > > <020%207896%200011>)
> > >     > and then delete the
> > >     > > > email
> > >     > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > relate
> > >     > to
> > >     > > > the
> > >     > > > > official business of this company shall be understood as
> > > neither
> > >     > given
> > >     > > > nor
> > >     > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > company
> > >     > > > > registered in England and Wales, company number 04008957)
> and
> > > IG
> > >     > Index
> > >     > > > > Limited (a company registered in England and Wales, company
> > > number
> > >     > > > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > >     > Hill,
> > >     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355) and
> > >     > > IG
> > >     > > > > Index Limited (register number 114059) are authorised and
> > > regulated
> > >     > by
> > >     > > > the
> > >     > > > > Financial Conduct Authority.
> > >     > > > >
> > >     > > > The information contained in this email is strictly
> > confidential
> > > and
> > >     > for
> > >     > > > the use of the addressee only, unless otherwise indicated. If
> > > you are
> > >     > not
> > >     > > > the intended recipient, please do not read, copy, use or
> > > disclose to
> > >     > > others
> > >     > > > this message or any attachment. Please also notify the sender
> > by
> > >     > replying
> > >     > > > to this email or by telephone (+44(020 7896 0011
> > > <020%207896%200011>)
> > >     > and then delete the
> > >     > > email
> > >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to
> > >     > > the
> > >     > > > official business of this company shall be understood as
> > neither
> > > given
> > >     > > nor
> > >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > >     > > > registered in England and Wales, company number 04008957) and
> > IG
> > > Index
> > >     > > > Limited (a company registered in England and Wales, company
> > > number
> > >     > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355) and
> > >     > IG
> > >     > > > Index Limited (register number 114059) are authorised and
> > > regulated by
> > >     > > the
> > >     > > > Financial Conduct Authority.
> > >     > > >
> > >     > > The information contained in this email is strictly
> confidential
> > > and for
> > >     > > the use of the addressee only, unless otherwise indicated. If
> you
> > > are not
> > >     > > the intended recipient, please do not read, copy, use or
> disclose
> > > to
> > >     > others
> > >     > > this message or any attachment. Please also notify the sender
> by
> > > replying
> > >     > > to this email or by telephone (+44(020 7896 0011
> > > <020%207896%200011>)
> > >     > and then delete the email
> > >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to
> > >     > the
> > >     > > official business of this company shall be understood as
> neither
> > > given
> > >     > nor
> > >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > >     > > registered in England and Wales, company number 04008957) and
> IG
> > > Index
> > >     > > Limited (a company registered in England and Wales, company
> > number
> > >     > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > > Hill,
> > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > and IG
> > >     > > Index Limited (register number 114059) are authorised and
> > > regulated by
> > >     > the
> > >     > > Financial Conduct Authority.
> > >     > >
> > >     >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Wow just read that def over tired. Hopefully it makes sense. Or you get the gist at least.

________________________________________
From: Michael Pearce <Mi...@ig.com>
Sent: Monday, December 19, 2016 9:19:02 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Jay,

Agreed this stemmed as offshoot from KIP-82.

Which our main driver for was to be able to have some headers for a null value as such for our routing, audit, tracing and a few other bits which currently we are forced to do with a message wrapper, if we all agreed on KIP-82 that we need native headers and look to implement that the push for this would dissipate.

This KIP would allow for though one use case that comes to mind we could see which is to have business data with a delete. Though as said this isn't something we are pushing for think really we would have.

As such in summary yes, if you want to fully support KIP-82 and we can get that agreed in principle and a target release version, I think quite a few guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.

Cheers
Mike

Sent using OWA for iPhone
________________________________________
From: Jay Kreps <ja...@confluent.io>
Sent: Monday, December 19, 2016 8:51:23 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hey Michael,

Here is the compatibility concern I have:

   1. You have a consumer app that relies on value == null to indicate a
   delete (current semantics).
   2. You upgrade Kafka and your clients.
   3. Some producer starts using the tombstone field in combination with
   non-null.

I share Ismael's dislike of setting tombstones on records with null values.
This makes sense as a transitional state, but as an end state its a bit
weird. You'd expect to be able to mix null values and tombstones, and have
the null values remain and the tombstones get compacted. However what will
happen is both will be compacted and upon debugging this you'll learn that
we sometimes use null in the value to indicate tombstone. Ismael's solution
is a bigger compatibility break, though, so not sure if that is better.

My other question is the relationship to KIP-82. My read is that this KIP
solves some but not all of the problems KIP-82 is intended for. KIP-82, on
the other hand, seems to address most of the motivating uses for this KIP.
The exception is maybe item (5) on the list where you want to simultaneous
delete and convey some information to subscribers, but I couldn't construct
a concrete examples for that one. Do we need to rationalize these two KIPs?
That is, do you still advocate this proposal if we do KIP-82 and vice
versa? As you may have noticed I'm somewhat emotionally invested in the
simplicity of the core data model, so my default position is let's try to
avoid stuffing more stuff in, but if we have to add stuff I like each of
these individually more than doing both. :-)

-Jay




On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jay
>
> I disagree here that we are breaking any compatibility, we went through
> this on the discussion thread in fact with the help of that thread is how
> the kip came to the solution.
>
> Also on the supported combinations front you mention, we are not taking
> anything away afaik.
>
> Currently supported are only are:
> Null value = delete
> Non-null value = non delete
>
> With this kip we would support
> Null value + tombstone = delete
> Non null value + tombstone = delete
> Non null value + no tombstone = non delete
>
> As for the alternative idea, this is simply a new policy, how can there be
> confusion here? For this policy it would be explicit that tombstone marker
> would need to be set for a delete.
>
> I'm going to vent a little now as starting to get quite frustrated.
>
> We are going round in circles on kip-82 as per use cases there is now many
> use cases, how many more are needed? just because confluent don't see these
> doesn't mean they aren't real use cases other have, this is the point of
> the Apache foundation, it shouldn't be the view of just one organisation.
> It really is getting a feeling of the NIH syndrome. Rather than it being
> constructive on discussion of the implementation detail.
>
> kip-87 spawned from as on the kip call we all agreed this was needed. And
> would at least allow a custom wrapper be supported in a compacted topic,
> allowing meta data. Which again now I feel we are spinning wheels, and
> simply finding reasons not support it.
>
> Cheers
> Mike
>
>
>
> Sent using OWA for iPad
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Friday, December 16, 2016 7:09:23 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hey Michael,
>
> I do think it might have been better had we started with a separate concept
> of null vs delete. Given that we are where we are, I'm not sure that the
> possible cures we explored so far are better than the disease.
>
> I apologize for coming to this late, but I didn't really understand the
> implications of the KIP and that we'd be breaking compatibility with
> existing apps until the vote had begun, and in my defense I'm not sure the
> other folks voting did either.
>
> I think we all agree there are many existing apps that are built with the
> assumption of "null value non-tombstone" and it isn't possible to
> disambiguate these from tombstones on the producer. It isn't that anyone is
> saying we have to support all four possibilities at once, it's that we
> simply can't orphan one of the existing combinations or our users will eat
> us!
>
> If I've understood your alternate solution of adding another setting for
> compaction, I think this does fix the compatibility problem, but it adds an
> odd mode the user has to add on all their topics. While the current state
> is easily explainable, the resulting state where the meaning of tombstone
> and null are overlapping and ambiguous and dependent on a compaction
> setting that could change out of band or not be in sync with your code in
> some environment seems worse to me then where we currently are. I think the
> question is how would this combination be explained to users and does it
> make sense?
>
> -Jay
>
>
>
> On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Chaps,
> >
> > Can we either get one more +1 binding (we have 2 already) on the
> existing?
> >
> > Or have some response on the below possible alternative. We are keen to
> > get working on this, so we make next feature release.
> >
> > Cheers
> > Mike
> >
> >
> > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> >     Hi Ismael
> >
> >     Did you see our email this morning, what's your thoughts on this
> > approach to instead we simply have a brand new policy?
> >
> >     Cheers
> >     Mike
> >
> >
> >     Sent using OWA for iPhone
> >     ________________________________________
> >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
> Juma <
> > ismael@juma.me.uk>
> >     Sent: Tuesday, December 13, 2016 11:30:05 AM
> >     To: dev@kafka.apache.org
> >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> >     Yes, this is actually tricky to do in a way where we both have the
> > desired
> >     semantics and maintain compatibility. When someone creates a
> >     `ProducerRecord` with a `null` value today, the producer doesn't know
> > if
> >     it's meant to be a tombstone or not. For V3 messages, it's easy when
> > the
> >     constructor that takes a tombstone is used. However, if any other
> >     constructor is used, it's not clear. A couple of options I can think
> > of,
> >     none of them particularly nice:
> >
> >     1. Have a third state where tombstone = unknown and the broker would
> > set
> >     the tombstone bit if the value was null and the topic was compacted.
> > People
> >     that wanted to pass a non-null value for the tombstone would have to
> > use
> >     the constructor that takes a tombstone. The drawbacks: third state
> for
> >     tombstone in message format, message conversion at the broker for a
> > common
> >     case.
> >
> >     2. Extend MetadataResponse to optionally include topic configs, which
> > would
> >     make it possible for the producer to be smarter about setting the
> >     tombstone. It would only do it if a tombstone was not passed
> > explicitly,
> >     the value was null and the topic was compacted. The main drawback is
> > that
> >     the producer would be getting a bit more data for each topic even
> > though it
> >     probably won't use it most of the time. Extending MetadataResponse to
> >     return topic configs would be useful for other reasons as well, so
> that
> >     part seems OK.
> >
> >     In addition, for both proposals, we could consider adding warnings to
> > the
> >     documentation that the behaviour of the constructors that don't take
> a
> >     tombstone would change in the next major release so that tombstone =
> > false.
> >     Not sure if this would be worth it though.
> >
> >     Ismael
> >
> >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> > ewen@confluent.io>
> >     wrote:
> >
> >     > Michael,
> >     >
> >     > It kind of depends on how you want to interpret the tombstone flag.
> > If it's
> >     > purely a producer-facing Kafka-level thing that we treat as
> internal
> > to the
> >     > broker and log cleaner once the record is sent, then your approach
> > makes
> >     > sense. You're just moving copying the null-indicates-delete
> behavior
> > of the
> >     > old constructor into the tombstone flag.
> >     >
> >     > However, if you want this change to more generally decouple the
> idea
> > of
> >     > deletion and null values, then you are sometimes converting what
> > might be a
> >     > completely valid null value that doesn't indicate deletion into a
> >     > tombstone. Downstream applications could potentially handle these
> > cases
> >     > differently given the separation of deletion from value.
> >     >
> >     > I guess the question is if we want to try to support the latter
> even
> > for
> >     > topics where we have older produce requests. An example where this
> > could
> >     > come up is in something like a CDC Connector. If we try to support
> > the
> >     > semantic difference, a connector might write changes to Kafka using
> > the
> >     > tombstone flag to indicate when a row was truly deleted (vs an
> > update that
> >     > sets it to null but still present; this probably makes more sense
> > for CDC
> >     > from document stores or extracting single columns). There are
> various
> >     > reasons we might want to maintain the full log and not turn
> > compaction on
> >     > (or just use a time-based retention policy), but downstream
> > applications
> >     > might care to know the difference between a delete and a null
> value.
> > In
> >     > fact both versions of the same log (compacted and time-retention)
> > could be
> >     > useful and I don't think it'll be uncommon to maintain both or use
> > KIP-71
> >     > to maintain a hybrid compacted/retention topic.
> >     >
> >     > -Ewen
> >     >
> >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > wrote:
> >     >
> >     > > Hi Jay,
> >     > >
> >     > > Why wouldn't that work, the tombstone value is only looked at by
> > the
> >     > > broker, on a topic configured for compaction as such is benign on
> > non
> >     > > compacted topics. This is as much as sending a null value
> currently
> >     > >
> >     > >
> >     > > Regards
> >     > > Mike
> >     > >
> >     > >
> >     > >
> >     > > Sent using OWA for iPhone
> >     > > ________________________________________
> >     > > From: Jay Kreps <ja...@confluent.io>
> >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
> >     > > To: dev@kafka.apache.org
> >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > >
> >     > > Hey Michael,
> >     > >
> >     > > I'm not quite sure that works as that would translate ALL null
> > values to
> >     > > tombstones, even for non-compacted topics that use null as an
> > acceptable
> >     > > value sent by the producer and expected by the consumer.
> >     > >
> >     > > -Jay
> >     > >
> >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > > wrote:
> >     > >
> >     > > > Hi Ewen,
> >     > > >
> >     > > > I think the easiest way to show this is with code.
> >     > > >
> >     > > > As you can see we keep the existing behaviour for code/binaries
> > calling
> >     > > > the pre-existing constructors, whereby if the value is null the
> >     > tombstone
> >     > > > is set to true.
> >     > > >
> >     > > > Regards
> >     > > > Mike
> >     > > >
> >     > > >
> >     > > >
> >     > > >     /**
> >     > > >      * Creates a record with a specified timestamp to be sent
> to
> > a
> >     > > > specified topic and partition
> >     > > >      *
> >     > > >      * @param topic The topic the record will be appended to
> >     > > >      * @param partition The partition to which the record
> should
> > be
> >     > sent
> >     > > >      * @param timestamp The timestamp of the record
> >     > > >      * @param tombstone if the record should be treated as a
> > tombstone
> >     > if
> >     > > > the topic is compacted
> >     > > >      * @param key The key that will be included in the record
> >     > > >      * @param value The record contents
> >     > > >      */
> >     > > >     public ProducerRecord(String topic, Integer partition,
> > Boolean
> >     > > > tombstone, Long timestamp, K key, V value) {
> >     > > >         if (topic == null)
> >     > > >             throw new IllegalArgumentException("Topic cannot
> be
> >     > null.");
> >     > > >         if (timestamp != null && timestamp < 0)
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Invalid timestamp: %d.
> > Timestamp
> >     > > should
> >     > > > always be non-negative or null.", timestamp));
> >     > > >         if (partition != null && partition < 0)
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Invalid partition: %d.
> > Partition
> >     > > number
> >     > > > should always be non-negative or null.", partition));
> >     > > >         if (!tombstone && value == null){
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Tombstone must be true if
> null
> >     > > value"));
> >     > > >         }
> >     > > >         this.topic = topic;
> >     > > >         this.partition = partition;
> >     > > >         this.tombstone = tombstone;
> >     > > >         this.key = key;
> >     > > >         this.value = value;
> >     > > >         this.timestamp = timestamp;
> >     > > >     }
> >     > > >
> >     > > >     public ProducerRecord(String topic, Integer partition, Long
> >     > > timestamp,
> >     > > > K key, V value) {
> >     > > >         this(topic, partition, value == null, timestamp, key,
> > value);
> >     > > >     }
> >     > > > ________________________________________
> >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> >     > > > Sent: Sunday, December 11, 2016 5:45 AM
> >     > > > To: dev@kafka.apache.org
> >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > >
> >     > > > It seemed like this was addressed in the migration section,
> > wasn't it?
> >     > > The
> >     > > > V2 vs V3 requests and the fact that the broker will downgrade
> the
> >     > message
> >     > > > format (losing zero copy) if you issues a V2 request to a
> broker
> > using
> >     > V3
> >     > > > format handles compatibility, does it not? The existing
> > consumers won't
> >     > > see
> >     > > > the extra metadata in the value, but they will get a null
> > instead and
> >     > > treat
> >     > > > it as a tombstone. Certainly there is a performance impact, but
> > it
> >     > seemed
> >     > > > compatible.
> >     > > >
> >     > > > I'm worried about this though:
> >     > > >
> >     > > >
> >     > > >    - *NOTE *: With the new version of producer client using
> >     > > ProduceRequest
> >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set)
> > and
> >     > null
> >     > > > value
> >     > > >    should not be allowed as the older version of consumer using
> >     > > > FetchRequest
> >     > > >    V2 will think of this as a tombstone when its actually not.
> >     > > >
> >     > > >
> >     > > > Unless I'm misunderstanding, this ends up breaking binary
> > compatibility
> >     > > for
> >     > > > the Java API. It sounds like this suggests that passing a null
> > value to
> >     > > the
> >     > > > existing ProducerRecord constructors would generate an
> exception
> > since
> >     > > you
> >     > > > didn't explicitly enable the tombstone (via whatever new
> > constructor is
> >     > > > provided). But that means you can't swap in a newer client jar
> > without
> >     > > > recompiling and get the same behavior if your app deletes keys
> > using
> >     > the
> >     > > > current approach because your app will start throwing
> > exceptions. Maybe
> >     > > > this is a tradeoff we're ok with, but we've tried pretty hard
> to
> > avoid
> >     > > > breaking compatibility like this so far -- it makes picking up
> > bug
> >     > fixes
> >     > > in
> >     > > > the clients harder for users of frameworks that have to pin to
> > earlier
> >     > > > default versions for compatibility.
> >     > > >
> >     > > > But then later the KIP says:
> >     > > >
> >     > > >
> >     > > >    - The old constructors for ProducerRecord without this
> > parameter
> >     > will
> >     > > be
> >     > > >    kept but updated so that their default behaviour if setting
> > null
> >     > value
> >     > > > will
> >     > > >    be the tombstone will be set to true to keep existing
> > behaviour.
> >     > > >
> >     > > >
> >     > > > So maybe I am misinterpreting something.
> >     > > >
> >     > > > And just a nit re: motivation section, item 6 would be clearer
> > for a
> >     > > union
> >     > > > schema with null (or optional schemas in other formats), e.g.
> > [null,
> >     > > > string], in which case losing the schema truly is losing
> > information
> >     > > > (whereas null is already the only valid value for a pure null
> > schema).
> >     > > >
> >     > > > -Ewen
> >     > > >
> >     > > >
> >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> >     > >
> >     > > > wrote:
> >     > > >
> >     > > > > Hi Jay,
> >     > > > >
> >     > > > > Good point this detail is missing in the KIP write up. Ive
> > added this
> >     > > > now.
> >     > > > >
> >     > > > > Essentially simply just upgrading the clients we do not
> expect
> > any
> >     > > client
> >     > > > > app code change needed.
> >     > > > >
> >     > > > > Cheers
> >     > > > > Mike
> >     > > > > ________________________________________
> >     > > > > From: Jay Kreps <ja...@confluent.io>
> >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
> >     > > > > To: dev@kafka.apache.org
> >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > > >
> >     > > > > Michael,
> >     > > > >
> >     > > > > The compatibility section goes through the migration path,
> but
> > isn't
> >     > > the
> >     > > > > bigger compatibility issue with existing apps? There are many
> >     > (probably
> >     > > > > thousands) of apps in production that use this feature and
> > send null
> >     > to
> >     > > > > mean delete. It seems like this would break compatibility
> with
> > them,
> >     > > and
> >     > > > > they would have to be rewritten to right?
> >     > > > >
> >     > > > > -Jay
> >     > > > >
> >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> >     > Michael.Pearce@ig.com
> >     > > >
> >     > > > > wrote:
> >     > > > >
> >     > > > > > Hi Jun,
> >     > > > > >
> >     > > > > > 4) On v3 we honour the tombstone. As such we expect it to
> be
> > set
> >     > > > > correctly
> >     > > > > > as per the KIP.
> >     > > > > >
> >     > > > > > 4.1) why would we want to produce an error when its v3?
> This
> > is the
> >     > > > exact
> >     > > > > > purpose to support non-null tombstone's
> >     > > > > > 4.2) again here im unclear on the question on the v3,
> produce
> >     > request
> >     > > > we
> >     > > > > > expect the tombstone flag to be set correctly.
> >     > > > > >
> >     > > > > > 4.4) compaction only occurs on compacted topics, the bit
> > makes no
> >     > > > > > difference and not looked at on un-compacted (time/size
> based
> >     > > > eviction).
> >     > > > > >
> >     > > > > >
> >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> >     > > > > >
> >     > > > > >     Hi, Michael,
> >     > > > > >
> >     > > > > >     4. Then, I think I misunderstood this point. Could you
> > document
> >     > > the
> >     > > > > >     following points in the wiki?
> >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
> > non-null
> >     > value,
> >     > > > > does
> >     > > > > > the
> >     > > > > >     send() get an error or does the producer automatically
> > set the
> >     > > > value
> >     > > > > to
> >     > > > > >     null?
> >     > > > > >     4.2 If producer V3 doesn't set tombstone, but provides
> a
> > null
> >     > > > value,
> >     > > > > > does
> >     > > > > >     the send() get an error or does the producer
> > automatically sets
> >     > > the
> >     > > > > >     tombstone?
> >     > > > > >     4.3 Does the broker only expect messages that (a) have
> no
> >     > > tombstone
> >     > > > > and
> >     > > > > >     non-null value; (b) have tombstone and null value and
> > reject
> >     > the
> >     > > > > > messages
> >     > > > > >     with an error code otherwise?
> >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
> > compacted
> >     > on
> >     > > > > not?
> >     > > > > >
> >     > > > > >     Thanks,
> >     > > > > >
> >     > > > > >     Jun
> >     > > > > >
> >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> >     > > > > Michael.Pearce@ig.com
> >     > > > > > >
> >     > > > > >     wrote:
> >     > > > > >
> >     > > > > >     > Not at all.  This only acts on compacted topics just
> > as what
> >     > > > occurs
> >     > > > > > today
> >     > > > > >     >
> >     > > > > >     > Sent using OWA for iPhone
> >     > > > > >     > ________________________________________
> >     > > > > >     > From: Jun Rao <ju...@confluent.io>
> >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> >     > > > > >     > To: dev@kafka.apache.org
> >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> > Flag
> >     > > > > >     >
> >     > > > > >     > Hi, Michael,
> >     > > > > >     >
> >     > > > > >     > 4. Hmm, does that mean the new client library can
> > never send
> >     > a
> >     > > > null
> >     > > > > > message
> >     > > > > >     > even to a regular topic? This seems like a change of
> > the
> >     > > existing
> >     > > > > > behavior.
> >     > > > > >     >
> >     > > > > >     > Thanks,
> >     > > > > >     >
> >     > > > > >     > Jun
> >     > > > > >     >
> >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> >     > > > > > Michael.Pearce@ig.com>
> >     > > > > >     > wrote:
> >     > > > > >     >
> >     > > > > >     > > Hi Jun,
> >     > > > > >     > >
> >     > > > > >     > > Re 4) That's because we expect the tombstone value
> > to be
> >     > set
> >     > > > > > correctly if
> >     > > > > >     > > message bit is 2, as such if an older client sends
> > in on
> >     > old
> >     > > > > > message the
> >     > > > > >     > > message is upcast and the bit is set correctly. And
> > such no
> >     > > > > longer
> >     > > > > > need
> >     > > > > >     > to
> >     > > > > >     > > check the value. Mayuresh can you confirm my
> > thinking and
> >     > > > > > understanding
> >     > > > > >     > of
> >     > > > > >     > > what we've discussed?
> >     > > > > >     > >
> >     > > > > >     > > The second point I understand what you're getting
> at
> > now my
> >     > > > > > apologies.
> >     > > > > >     > Yes
> >     > > > > >     > > this makes sense to save on touching the message,
> if
> > we're
> >     > > the
> >     > > > > > only kip
> >     > > > > >     > > going in, in this release.
> >     > > > > >     > >
> >     > > > > >     > > Cheers
> >     > > > > >     > > Mike
> >     > > > > >     > >
> >     > > > > >     > > Sent using OWA for iPhone
> >     > > > > >     > > ________________________________________
> >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
> >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> >     > > > > >     > > To: dev@kafka.apache.org
> >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone Flag
> >     > > > > >     > >
> >     > > > > >     > > Hi, Michael,
> >     > > > > >     > >
> >     > > > > >     > > 4. Is this updated in the wiki? The text "If the
> > magic byte
> >     > > on
> >     > > > > > message is
> >     > > > > >     > > 2, the broker should use the tombstone bit for log
> >     > > compaction."
> >     > > > > > doesn't
> >     > > > > >     > > seem to have changed.
> >     > > > > >     > >
> >     > > > > >     > > 2. My point is that if we change the message format
> > just
> >     > for
> >     > > > this
> >     > > > > > KIP, we
> >     > > > > >     > > should consider whether it's worth optimizing the
> > down
> >     > > > conversion
> >     > > > > > path
> >     > > > > >     > > (i.e., decide whether a conversion is needed by
> just
> >     > looking
> >     > > at
> >     > > > > the
> >     > > > > >     > > tombstone bit in the wrapper message) since
> > tombstone will
> >     > be
> >     > > > > used
> >     > > > > >     > rarely.
> >     > > > > >     > > However, if the message format change here is
> > combined with
> >     > > > other
> >     > > > > > KIPs,
> >     > > > > >     > > then this optimization likely won't be needed. The
> > latter
> >     > > > > probably
> >     > > > > > makes
> >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> > think?
> >     > > > > >     > >
> >     > > > > >     > > Other than those, +1 from me,
> >     > > > > >     > >
> >     > > > > >     > > Thanks,
> >     > > > > >     > >
> >     > > > > >     > > Jun
> >     > > > > >     > >
> >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> >     > > > > > Michael.Pearce@ig.com>
> >     > > > > >     > > wrote:
> >     > > > > >     > >
> >     > > > > >     > > > Hi Jun
> >     > > > > >     > > >
> >     > > > > >     > > > do we have your vote on this now?
> >     > > > > >     > > >
> >     > > > > >     > > > Any other concerns?
> >     > > > > >     > > >
> >     > > > > >     > > > Cheers
> >     > > > > >     > > > Mike
> >     > > > > >     > > >
> >     > > > > >     > > > Sent using OWA for iPhone
> >     > > > > >     > > > ________________________________________
> >     > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> >     > > > > >     > > > To: dev@kafka.apache.org
> >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone
> >     > Flag
> >     > > > > >     > > >
> >     > > > > >     > > > Hi Jun,
> >     > > > > >     > > >
> >     > > > > >     > > > Have updated. Thanks again for the feedback.
> >     > > > > >     > > >
> >     > > > > >     > > > Agree yes we should align up when it gets to
> that,
> > I
> >     > assume
> >     > > > > > you've
> >     > > > > >     > > flagged
> >     > > > > >     > > > the same in the other KIP?
> >     > > > > >     > > >
> >     > > > > >     > > > I think for now let's get this KIP approved, then
> > we can
> >     > > > start
> >     > > > > > the work
> >     > > > > >     > > to
> >     > > > > >     > > > get an initial PR, then we can discuss how to
> > align the
> >     > two
> >     > > > if
> >     > > > > > needed.
> >     > > > > >     > > >
> >     > > > > >     > > > Cheers,
> >     > > > > >     > > > Mike
> >     > > > > >     > > >
> >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
> jun@confluent.io>
> >     > wrote:
> >     > > > > >     > > >
> >     > > > > >     > > >     Hi, Michael,
> >     > > > > >     > > >
> >     > > > > >     > > >     For 2), this is fine. Could you update the
> KIP
> > wiki
> >     > to
> >     > > > make
> >     > > > > > this
> >     > > > > >     > and
> >     > > > > >     > > > other
> >     > > > > >     > > >     points clearer? Other than that, the KIP
> looks
> > good
> >     > to
> >     > > > me.
> >     > > > > >     > > >
> >     > > > > >     > > >     An orthogonal thing is that there are other
> > KIPs such
> >     > > as
> >     > > > > > KIP-98
> >     > > > > >     > that
> >     > > > > >     > > > also
> >     > > > > >     > > >     intend to change the message format. If they
> > all get
> >     > > > > > approved, we
> >     > > > > >     > > > should
> >     > > > > >     > > >     think about whether it's better to just bump
> > up the
> >     > > magic
> >     > > > > > byte once
> >     > > > > >     > > to
> >     > > > > >     > > >     incorporate multiple format changes like we
> > did in
> >     > > > > > KIP-31/KIP-32.
> >     > > > > >     > > >
> >     > > > > >     > > >     Thanks,
> >     > > > > >     > > >
> >     > > > > >     > > >     Jun
> >     > > > > >     > > >
> >     > > > > >     > > >
> >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
> Pearce
> > <
> >     > > > > >     > > Michael.Pearce@ig.com>
> >     > > > > >     > > >     wrote:
> >     > > > > >     > > >
> >     > > > > >     > > >     > Hi Jao
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > Thanks for the response. Sorry for slow
> > reply, both
> >     > > > with
> >     > > > > > personal
> >     > > > > >     > > > sickness
> >     > > > > >     > > >     > and also battling some critical issues
> > encountered
> >     > > > since
> >     > > > > >     > upgrading
> >     > > > > >     > > to
> >     > > > > >     > > >     > 0.10.1.0
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > 1) Thans for spotting, Document error where
> > we
> >     > > branched
> >     > > > > > this KIP
> >     > > > > >     > > from
> >     > > > > >     > > >     > KIP-82, will get that removed.
> >     > > > > >     > > >     > 2) Intent is to do this just at the record
> > message
> >     > > > level.
> >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> >     > > corrected.
> >     > > > > >     > > >     > 4) As per discussion thread we will support
> >     > > tombstone +
> >     > > > > > null
> >     > > > > >     > value,
> >     > > > > >     > > >     > tombstone + non null value, no tombstone +
> > null
> >     > > value.
> >     > > > > >     > > >     > 5) I believe this was in the discussion
> > thread,
> >     > > > @Mayuresh
> >     > > > > > is this
> >     > > > > >     > > >     > something we've overlooked? I thought we
> > would down
> >     > > > > > convert and
> >     > > > > >     > > > remove the
> >     > > > > >     > > >     > value so the old consumer had existing
> > behavior, or
> >     > > is
> >     > > > > > there
> >     > > > > >     > > > something we
> >     > > > > >     > > >     > haven't thought about?
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > Cheers
> >     > > > > >     > > >     > Mike
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> > jun@confluent.io>
> >     > > > > wrote:
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Hi, Michael,
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Thanks for the KIP. A few comments
> below.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     1. The message format change contains
> >     > > > "HeadersLength
> >     > > > > >     > Headers".
> >     > > > > >     > > > Is that
> >     > > > > >     > > >     >     intended?
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     2. For compressed messageset, is the
> > tombstone
> >     > > bit
> >     > > > > > only set
> >     > > > > >     > at
> >     > > > > >     > > > the
> >     > > > > >     > > >     > shallow
> >     > > > > >     > > >     >     level? Do we always leave that bit in
> the
> >     > wrapper
> >     > > > > > message
> >     > > > > >     > > unset?
> >     > > > > >     > > > An
> >     > > > > >     > > >     >     alternative is to set the tombstone bit
> > in the
> >     > > > > wrapper
> >     > > > > > if at
> >     > > > > >     > > > least one
> >     > > > > >     > > >     >     inner message has the tombstone bit
> set.
> > This
> >     > > makes
> >     > > > > > things a
> >     > > > > >     > > bit
> >     > > > > >     > > > more
> >     > > > > >     > > >     >     complicated, but we could potentially
> > exploit
> >     > > that
> >     > > > > for
> >     > > > > >     > > > optimizing down
> >     > > > > >     > > >     >     conversion. For example, we only need
> to
> >     > convert
> >     > > > > > messages
> >     > > > > >     > with
> >     > > > > >     > > > magic 2
> >     > > > > >     > > >     > to
> >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit
> > is set
> >     > > > > > (conversion is
> >     > > > > >     > > > always
> >     > > > > >     > > >     > needed
> >     > > > > >     > > >     >     from magic 2 to magic 0). Not sure if
> the
> >     > > > > optimization
> >     > > > > > is
> >     > > > > >     > worth
> >     > > > > >     > > > the
> >     > > > > >     > > >     >     complexity though.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     3. The referencing of the new version
> of
> >     > > > > >     > > > ProducerRequest/FetchRequest
> >     > > > > >     > > >     > is
> >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> > convention
> >     > > > starts
> >     > > > > at
> >     > > > > >     > version
> >     > > > > >     > > > at 0, I
> >     > > > > >     > > >     >     think the new version would be 3.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     4. "If the magic byte on message is 2,
> > the
> >     > broker
> >     > > > > > should use
> >     > > > > >     > > the
> >     > > > > >     > > >     > tombstone
> >     > > > > >     > > >     >     bit for log compaction." What about
> null
> > value?
> >     > > My
> >     > > > > >     > > understanding
> >     > > > > >     > > > is
> >     > > > > >     > > >     > that
> >     > > > > >     > > >     >     null value will be treated the same as
> > setting
> >     > > the
> >     > > > > > tombstone
> >     > > > > >     > > bit.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     5. For the migration path, it would be
> > useful
> >     > to
> >     > > > > > describe the
> >     > > > > >     > > > down
> >     > > > > >     > > >     >     conversion path to consumers (i.e.,
> > brokers on
> >     > > > > message
> >     > > > > > format
> >     > > > > >     > > > 0.10.2
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     consumers on older version).
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Thanks,
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Jun
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
> Michael
> >     > Pearce <
> >     > > > > >     > > > Michael.Pearce@ig.com
> >     > > > > >     > > >     > >
> >     > > > > >     > > >     >     wrote:
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     > Hi All,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > We have been discussing in the below
> > thread
> >     > and
> >     > > > > final
> >     > > > > >     > changes
> >     > > > > >     > > > have
> >     > > > > >     > > >     > been
> >     > > > > >     > > >     >     > made to the KIP wiki based on these
> >     > > discussions.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > We would now like to put to the vote
> > the
> >     > > > following
> >     > > > > > KIP:
> >     > > > > >     > > >     >     > https://cwiki.apache.org/
> >     > > > > > confluence/display/KAFKA/KIP-
> >     > > > > >     > 87+-+
> >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > This kip is for having a distinct
> > compaction
> >     > > > > > attribute
> >     > > > > >     > > > "tombstone"
> >     > > > > >     > > >     > flag
> >     > > > > >     > > >     >     > instead of relying on null value,
> > allowing
> >     > > > non-null
> >     > > > > > value
> >     > > > > >     > > > delete
> >     > > > > >     > > >     > messages.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > Many thanks,
> >     > > > > >     > > >     >     > Michael
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
> Pearce"
> > <
> >     > > > > >     > > Michael.Pearce@ig.com>
> >     > > > > >     > > >     > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Hi Mayuresh,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     LGTM. Ive just made one small
> > adjustment
> >     > > > > > updating the
> >     > > > > >     > > wire
> >     > > > > >     > > >     > protocol to
> >     > > > > >     > > >     >     > show the magic byte bump.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Do we think we're good to put to
> a
> > vote?
> >     > Is
> >     > > > > > there any
> >     > > > > >     > > > other bits
> >     > > > > >     > > >     >     > needing discussion?
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Cheers
> >     > > > > >     > > >     >     >     Mike
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
> > Gharat" <
> >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
> >     > > > > >     > > >     >     > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Hi Michael,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         I have updated the migration
> > section
> >     > of
> >     > > > the
> >     > > > > > KIP.
> >     > > > > >     > Can
> >     > > > > >     > > > you
> >     > > > > >     > > >     > please
> >     > > > > >     > > >     >     > take a look?
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Thanks,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Mayuresh
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07
> > AM,
> >     > > Mayuresh
> >     > > > > > Gharat <
> >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         > Hi Michael,
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > That whilst sending
> > tombstone and
> >     > non
> >     > > > > null
> >     > > > > > value,
> >     > > > > >     > > the
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     > can expect
> >     > > > > >     > > >     >     >         > only to receive the
> non-null
> >     > message
> >     > > > only
> >     > > > > > in step
> >     > > > > >     > > > (3) is
> >     > > > > >     > > >     > this
> >     > > > > >     > > >     >     > correct?
> >     > > > > >     > > >     >     >         > ---> I do agree with you
> > here.
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Becket, Ismael : can you
> guys
> >     > review
> >     > > > the
> >     > > > > >     > migration
> >     > > > > >     > > > plan
> >     > > > > >     > > >     > listed
> >     > > > > >     > > >     >     > above using
> >     > > > > >     > > >     >     >         > magic byte?
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Thanks,
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Mayuresh
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at
> 8:58
> > AM,
> >     > > > Michael
> >     > > > > > Pearce <
> >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> >     > > > > >     > > >     >     >         > wrote:
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >> Many thanks for this
> > Mayuresh. I
> >     > > don't
> >     > > > > > have any
> >     > > > > >     > > >     > objections.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> I assume we should state:
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> That whilst sending
> > tombstone and
> >     > > non
> >     > > > > null
> >     > > > > >     > value,
> >     > > > > >     > > > the
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     > can expect
> >     > > > > >     > > >     >     >         >> only to receive the
> non-null
> >     > message
> >     > > > > only
> >     > > > > > in
> >     > > > > >     > step
> >     > > > > >     > > > (3) is
> >     > > > > >     > > >     > this
> >     > > > > >     > > >     >     > correct?
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Cheers
> >     > > > > >     > > >     >     >         >> Mike
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Sent using OWA for iPhone
> >     > > > > >     > > >     >     >         >>
> > ______________________________
> >     > > > > __________
> >     > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
> >     > > > > >     > gharatmayuresh15@gmail.com
> >     > > > > >     > > >
> >     > > > > >     > > >     >     >         >> Sent: Thursday, November
> > 17, 2016
> >     > > > > 5:18:41
> >     > > > > > PM
> >     > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
> >     > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> > KIP-87 -
> >     > Add
> >     > > > > > Compaction
> >     > > > > >     > > > Tombstone
> >     > > > > >     > > >     > Flag
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Hi Ismael,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Thanks for the
> explanation.
> >     > > > > >     > > >     >     >         >> Specially I like this part
> > where
> >     > in
> >     > > > you
> >     > > > > >     > mentioned
> >     > > > > >     > > > we can
> >     > > > > >     > > >     > get
> >     > > > > >     > > >     >     > rid of the
> >     > > > > >     > > >     >     >         >> older null value support
> > for log
> >     > > > > > compaction
> >     > > > > >     > later
> >     > > > > >     > > > on,
> >     > > > > >     > > >     > here :
> >     > > > > >     > > >     >     >         >> We can't change semantics
> > of the
> >     > > > message
> >     > > > > > format
> >     > > > > >     > > > without
> >     > > > > >     > > >     > having
> >     > > > > >     > > >     >     > a long
> >     > > > > >     > > >     >     >         >> transition period. And we
> > can't
> >     > rely
> >     > > > > >     > > >     >     >         >> on people reading
> > documentation or
> >     > > > > acting
> >     > > > > > on a
> >     > > > > >     > > > warning for
> >     > > > > >     > > >     >     > something so
> >     > > > > >     > > >     >     >         >> fundamental. As such, my
> > take is
> >     > > that
> >     > > > we
> >     > > > > > need to
> >     > > > > >     > > > bump the
> >     > > > > >     > > >     > magic
> >     > > > > >     > > >     >     > byte. The
> >     > > > > >     > > >     >     >         >> good news is
> >     > > > > >     > > >     >     >         >> that we don't have to
> > support all
> >     > > > > versions
> >     > > > > >     > > forever.
> >     > > > > >     > > > We
> >     > > > > >     > > >     > have
> >     > > > > >     > > >     >     > said that we
> >     > > > > >     > > >     >     >         >> will support direct
> > upgrades for 2
> >     > > > > years.
> >     > > > > > That
> >     > > > > >     > > > means that
> >     > > > > >     > > >     >     > message format
> >     > > > > >     > > >     >     >         >> version n could, in
> theory,
> > be
> >     > > > removed 2
> >     > > > > > years
> >     > > > > >     > > > after the
> >     > > > > >     > > >     > it's
> >     > > > > >     > > >     >     > introduced.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Just a heads up, I would
> > like to
> >     > > > mention
> >     > > > > > that
> >     > > > > >     > even
> >     > > > > >     > > > without
> >     > > > > >     > > >     >     > bumping magic
> >     > > > > >     > > >     >     >         >> byte, we will *NOT* loose
> > zero
> >     > copy
> >     > > as
> >     > > > > in
> >     > > > > > the
> >     > > > > >     > > > client(x+1)
> >     > > > > >     > > >     > in my
> >     > > > > >     > > >     >     >         >> explanation
> >     > > > > >     > > >     >     >         >> above will convert
> > internally a
> >     > null
> >     > > > > > value to
> >     > > > > >     > > have a
> >     > > > > >     > > >     > tombstone
> >     > > > > >     > > >     >     > bit set and
> >     > > > > >     > > >     >     >         >> a tombstone bit set to
> have
> > a null
> >     > > > value
> >     > > > > >     > > > automatically
> >     > > > > >     > > >     >     > internally and by
> >     > > > > >     > > >     >     >         >> the time we move to
> version
> > (x+2),
> >     > > the
> >     > > > > > clients
> >     > > > > >     > > > would have
> >     > > > > >     > > >     >     > upgraded.
> >     > > > > >     > > >     >     >         >> Obviously if we support a
> > request
> >     > > from
> >     > > > > >     > > consumer(x),
> >     > > > > >     > > > we
> >     > > > > >     > > >     > will
> >     > > > > >     > > >     >     > loose zero
> >     > > > > >     > > >     >     >         >> copy
> >     > > > > >     > > >     >     >         >> but that is the same case
> > with
> >     > magic
> >     > > > > byte.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> But if magic byte bump
> > makes life
> >     > > > easier
> >     > > > > > for
> >     > > > > >     > > > transition
> >     > > > > >     > > >     > for the
> >     > > > > >     > > >     >     > above
> >     > > > > >     > > >     >     >         >> reasons that you
> explained,
> > I am
> >     > OK
> >     > > > with
> >     > > > > > it
> >     > > > > >     > since
> >     > > > > >     > > > we are
> >     > > > > >     > > >     > going
> >     > > > > >     > > >     >     > to meet the
> >     > > > > >     > > >     >     >         >> end goal down the road :)
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> On a side note can we
> > update the
> >     > doc
> >     > > > > here
> >     > > > > > on
> >     > > > > >     > magic
> >     > > > > >     > > > byte
> >     > > > > >     > > >     > to say
> >     > > > > >     > > >     >     > that "*it
> >     > > > > >     > > >     >     >         >> should be bumped whenever
> > the
> >     > > message
> >     > > > > > format is
> >     > > > > >     > > > changed
> >     > > > > >     > > >     > or the
> >     > > > > >     > > >     >     >         >> interpretation of message
> > format
> >     > > > (usage
> >     > > > > > of the
> >     > > > > >     > > > reserved
> >     > > > > >     > > >     > bits as
> >     > > > > >     > > >     >     > well) is
> >     > > > > >     > > >     >     >         >> changed*".
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Hi Michael,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Here is the update plan
> > that we
> >     > > > > discussed
> >     > > > > >     > offline
> >     > > > > >     > > >     > yesterday :
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Currently the magic-byte
> > which
> >     > > > > > corresponds to
> >     > > > > >     > the
> >     > > > > >     > > >     >     > "message.format.version"
> >     > > > > >     > > >     >     >         >> is set to 1.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 1) On broker it will be
> set
> > to 1
> >     > > > > > initially.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 2) When a producer client
> > sends a
> >     > > > > message
> >     > > > > > with
> >     > > > > >     > > > magic-byte
> >     > > > > >     > > >     > = 2,
> >     > > > > >     > > >     >     > since the
> >     > > > > >     > > >     >     >         >> broker is on magic-byte =
> > 1, we
> >     > will
> >     > > > > down
> >     > > > > >     > convert
> >     > > > > >     > > > it,
> >     > > > > >     > > >     > which
> >     > > > > >     > > >     >     > means if the
> >     > > > > >     > > >     >     >         >> tombstone bit is set, the
> > value
> >     > will
> >     > > > be
> >     > > > > > set to
> >     > > > > >     > > > null. A
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     >         >> understanding magic-byte =
> > 1, will
> >     > > > still
> >     > > > > > work
> >     > > > > >     > with
> >     > > > > >     > > > this. A
> >     > > > > >     > > >     >     > consumer
> >     > > > > >     > > >     >     >         >> working
> >     > > > > >     > > >     >     >         >> with magic-byte =2 will
> > also be
> >     > able
> >     > > > to
> >     > > > > >     > understand
> >     > > > > >     > > > this,
> >     > > > > >     > > >     > since
> >     > > > > >     > > >     >     > it
> >     > > > > >     > > >     >     >         >> understands the tombstone.
> >     > > > > >     > > >     >     >         >> Now there is still the
> > question of
> >     > > > > > supporting a
> >     > > > > >     > > >     > non-tombstone
> >     > > > > >     > > >     >     > and null
> >     > > > > >     > > >     >     >         >> value from producer client
> > with
> >     > > > > > magic-byte = 2.*
> >     > > > > >     > > (I
> >     > > > > >     > > > am
> >     > > > > >     > > >     > not sure
> >     > > > > >     > > >     >     > if we
> >     > > > > >     > > >     >     >         >> should support this.
> > Ismael/Becket
> >     > > can
> >     > > > > > comment
> >     > > > > >     > > > here)*
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 3) When almost all the
> > clients
> >     > have
> >     > > > > > upgraded,
> >     > > > > >     > the
> >     > > > > >     > > >     >     > message.format.version
> >     > > > > >     > > >     >     >         >> on
> >     > > > > >     > > >     >     >         >> the broker can be changed
> > to 2,
> >     > > where
> >     > > > in
> >     > > > > > the
> >     > > > > >     > down
> >     > > > > >     > > >     > conversion in
> >     > > > > >     > > >     >     > the above
> >     > > > > >     > > >     >     >         >> step will not happen. If
> at
> > this
> >     > > point
> >     > > > > we
> >     > > > > > get a
> >     > > > > >     > > > consumer
> >     > > > > >     > > >     >     > request from a
> >     > > > > >     > > >     >     >         >> older consumer, we might
> > have to
> >     > > down
> >     > > > > > convert
> >     > > > > >     > > where
> >     > > > > >     > > > in we
> >     > > > > >     > > >     > loose
> >     > > > > >     > > >     >     > zero copy,
> >     > > > > >     > > >     >     >         >> but these cases should be
> > rare.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Becket can you review this
> > plan
> >     > and
> >     > > > add
> >     > > > > > more
> >     > > > > >     > > > details if I
> >     > > > > >     > > >     > have
> >     > > > > >     > > >     >     >         >> missed/wronged something,
> > before
> >     > we
> >     > > > put
> >     > > > > > it on
> >     > > > > >     > KIP.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Thanks,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Mayuresh
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> > 11:07 PM,
> >     > > > > Michael
> >     > > > > >     > Pearce <
> >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> >     > > > > >     > > >     >     >         >> wrote:
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> > Thanks guys, for
> > discussing this
> >     > > > > > offline and
> >     > > > > >     > > > getting
> >     > > > > >     > > >     > some
> >     > > > > >     > > >     >     > consensus.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > So its clear for myself
> > and
> >     > others
> >     > > > > what
> >     > > > > > is
> >     > > > > >     > > > proposed now
> >     > > > > >     > > >     > (i
> >     > > > > >     > > >     >     > think i
> >     > > > > >     > > >     >     >         >> > understand, but want to
> > make
> >     > sure)
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Could i ask either
> > directly
> >     > update
> >     > > > the
> >     > > > > > kip to
> >     > > > > >     > > > detail the
> >     > > > > >     > > >     >     > migration
> >     > > > > >     > > >     >     >         >> > strategy, or (re-)state
> > your
> >     > > offline
> >     > > > > > discussed
> >     > > > > >     > > and
> >     > > > > >     > > >     > agreed
> >     > > > > >     > > >     >     > migration
> >     > > > > >     > > >     >     >         >> > strategy based on a
> magic
> > byte
> >     > is
> >     > > in
> >     > > > > > this
> >     > > > > >     > > thread.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > The main original driver
> > for the
> >     > > KIP
> >     > > > > > was to
> >     > > > > >     > > > support
> >     > > > > >     > > >     >     > compaction where
> >     > > > > >     > > >     >     >         >> value
> >     > > > > >     > > >     >     >         >> > isn't null, based off
> the
> >     > > > discussions
> >     > > > > on
> >     > > > > >     > KIP-82
> >     > > > > >     > > > thread.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > We should be able to
> > support
> >     > > > > > non-tombstone +
> >     > > > > >     > > null
> >     > > > > >     > > > value
> >     > > > > >     > > >     > by the
> >     > > > > >     > > >     >     >         >> completion
> >     > > > > >     > > >     >     >         >> > of the KIP, as we noted
> > when
> >     > > > > discussing
> >     > > > > > this
> >     > > > > >     > > kip,
> >     > > > > >     > > > having
> >     > > > > >     > > >     >     > logic based on
> >     > > > > >     > > >     >     >         >> a
> >     > > > > >     > > >     >     >         >> > null value isn't very
> > clean and
> >     > > also
> >     > > > > > separates
> >     > > > > >     > > the
> >     > > > > >     > > >     > concerns.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > As discussed already
> > though we
> >     > can
> >     > > > > > split this
> >     > > > > >     > > into
> >     > > > > >     > > >     > KIP-87a
> >     > > > > >     > > >     >     > and KIP-87b
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Where we look to deliver
> > KIP-87a
> >     > > on
> >     > > > a
> >     > > > > >     > compacted
> >     > > > > >     > > > topic
> >     > > > > >     > > >     > (to
> >     > > > > >     > > >     >     > address the
> >     > > > > >     > > >     >     >         >> > immediate issues)
> >     > > > > >     > > >     >     >         >> > * tombstone + null value
> >     > > > > >     > > >     >     >         >> > * tombstone + non-null
> > value
> >     > > > > >     > > >     >     >         >> > * non-tombstone +
> > non-null value
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Then we can discuss once
> > KIP-87a
> >     > > is
> >     > > > > > completed
> >     > > > > >     > > > options
> >     > > > > >     > > >     > later
> >     > > > > >     > > >     >     > and how we
> >     > > > > >     > > >     >     >         >> > support the second part
> > KIP-87b
> >     > to
> >     > > > > > deliver:
> >     > > > > >     > > >     >     >         >> > * non-tombstone + null
> > value
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Cheers
> >     > > > > >     > > >     >     >         >> > Mike
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> > ______________________________
> >     > > > > > __________
> >     > > > > >     > > >     >     >         >> > From: Becket Qin <
> >     > > > > becket.qin@gmail.com>
> >     > > > > >     > > >     >     >         >> > Sent: Thursday, November
> > 17,
> >     > 2016
> >     > > > 1:43
> >     > > > > > AM
> >     > > > > >     > > >     >     >         >> > To:
> dev@kafka.apache.org
> >     > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> > KIP-87 -
> >     > > Add
> >     > > > > > Compaction
> >     > > > > >     > > >     > Tombstone Flag
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had
> > an
> >     > > offline
> >     > > > > >     > discussion,
> >     > > > > >     > > > and
> >     > > > > >     > > >     > following
> >     > > > > >     > > >     >     > is a brief
> >     > > > > >     > > >     >     >         >> > summary.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > 1. We agreed that not
> > bumping up
> >     > > > magic
> >     > > > > > value
> >     > > > > >     > may
> >     > > > > >     > > > result
> >     > > > > >     > > >     > in
> >     > > > > >     > > >     >     > losing zero
> >     > > > > >     > > >     >     >         >> copy
> >     > > > > >     > > >     >     >         >> > during migration.
> >     > > > > >     > > >     >     >         >> > 2. Given that bumping up
> > magic
> >     > > value
> >     > > > > is
> >     > > > > > almost
> >     > > > > >     > > > free and
> >     > > > > >     > > >     > has
> >     > > > > >     > > >     >     > benefit of
> >     > > > > >     > > >     >     >         >> > avoiding potential
> > performance
> >     > > > issue.
> >     > > > > > It is
> >     > > > > >     > > > probably
> >     > > > > >     > > >     > worth
> >     > > > > >     > > >     >     > doing.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > One issue we still need
> > to think
> >     > > > about
> >     > > > > > is
> >     > > > > >     > > whether
> >     > > > > >     > > > we
> >     > > > > >     > > >     > want to
> >     > > > > >     > > >     >     > support a
> >     > > > > >     > > >     >     >         >> > non-tombstone message
> > with null
> >     > > > value.
> >     > > > > >     > > >     >     >         >> > Currently it is not
> > supported by
> >     > > > > Kafka.
> >     > > > > > If we
> >     > > > > >     > > > allow a
> >     > > > > >     > > >     >     > non-tombstone null
> >     > > > > >     > > >     >     >         >> > value message to exist
> > after
> >     > > KIP-87.
> >     > > > > The
> >     > > > > >     > problem
> >     > > > > >     > > > is
> >     > > > > >     > > >     > that such
> >     > > > > >     > > >     >     > message
> >     > > > > >     > > >     >     >         >> will
> >     > > > > >     > > >     >     >         >> > not be supported by the
> >     > consumers
> >     > > > > prior
> >     > > > > > to
> >     > > > > >     > > KIP-87.
> >     > > > > >     > > >     > Because a
> >     > > > > >     > > >     >     > null value
> >     > > > > >     > > >     >     >         >> > will always be
> > interpreted to a
> >     > > > > > tombstone.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > One option is that we
> > keep the
> >     > > > current
> >     > > > > > way,
> >     > > > > >     > i.e.
> >     > > > > >     > > > do not
> >     > > > > >     > > >     >     > support such
> >     > > > > >     > > >     >     >         >> > message. It would be
> good
> > to
> >     > know
> >     > > if
> >     > > > > > there is
> >     > > > > >     > a
> >     > > > > >     > > >     > concrete use
> >     > > > > >     > > >     >     > case for
> >     > > > > >     > > >     >     >         >> such
> >     > > > > >     > > >     >     >         >> > message. If there is
> not,
> > we can
> >     > > > > > probably just
> >     > > > > >     > > not
> >     > > > > >     > > >     > support it.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Thanks,
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at
> > 1:28 PM,
> >     > > > > > Mayuresh
> >     > > > > >     > > Gharat <
> >     > > > > >     > > >     >     >         >> >
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > wrote:
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > > Hi Ismael,
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > This is something I
> can
> > think
> >     > of
> >     > > > for
> >     > > > > >     > migration
> >     > > > > >     > > > plan:
> >     > > > > >     > > >     >     >         >> > > So the migration plan
> > can look
> >     > > > > > something
> >     > > > > >     > like
> >     > > > > >     > > > this,
> >     > > > > >     > > >     > with up
> >     > > > > >     > > >     >     >         >> conversion :
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > 1) Currently lets say
> > we have
> >     > > > Broker
> >     > > > > > at
> >     > > > > >     > > version
> >     > > > > >     > > > x.
> >     > > > > >     > > >     >     >         >> > > 2) Currently we have
> > clients
> >     > at
> >     > > > > > version x.
> >     > > > > >     > > >     >     >         >> > > 3) a) We move the
> > version to
> >     > > > > > Broker(x+1) :
> >     > > > > >     > > > supports
> >     > > > > >     > > >     > both
> >     > > > > >     > > >     >     > tombstone and
> >     > > > > >     > > >     >     >         >> > null
> >     > > > > >     > > >     >     >         >> > > for log compaction.
> >     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> > client
> >     > to
> >     > > > > > version
> >     > > > > >     > > > client(x+1) :
> >     > > > > >     > > >     > if in
> >     > > > > >     > > >     >     > the
> >     > > > > >     > > >     >     >         >> producer
> >     > > > > >     > > >     >     >         >> > > client(x+1) the value
> > is set
> >     > to
> >     > > > > null,
> >     > > > > > we
> >     > > > > >     > will
> >     > > > > >     > > >     > automatically
> >     > > > > >     > > >     >     > set the
> >     > > > > >     > > >     >     >         >> > > Tombstone bit
> > internally. If
> >     > the
> >     > > > > > producer
> >     > > > > >     > > > client(x+1)
> >     > > > > >     > > >     > sets
> >     > > > > >     > > >     >     > the
> >     > > > > >     > > >     >     >         >> tombstone
> >     > > > > >     > > >     >     >         >> > > itself, well and good.
> > For
> >     > > > producer
> >     > > > > >     > client(x),
> >     > > > > >     > > > the
> >     > > > > >     > > >     > broker
> >     > > > > >     > > >     >     > will up
> >     > > > > >     > > >     >     >         >> convert
> >     > > > > >     > > >     >     >         >> > > to have the tombstone
> > bit.
> >     > > > > > Broker(x+1) is
> >     > > > > >     > > > supporting
> >     > > > > >     > > >     > both.
> >     > > > > >     > > >     >     > Consumer
> >     > > > > >     > > >     >     >         >> > > client(x+1) will be
> > aware of
> >     > > this
> >     > > > > and
> >     > > > > > should
> >     > > > > >     > > be
> >     > > > > >     > > > able
> >     > > > > >     > > >     > to
> >     > > > > >     > > >     >     > handle this.
> >     > > > > >     > > >     >     >         >> For
> >     > > > > >     > > >     >     >         >> > > consumer client(x) we
> > will
> >     > down
> >     > > > > > convert the
> >     > > > > >     > > > message
> >     > > > > >     > > >     > on the
> >     > > > > >     > > >     >     > broker
> >     > > > > >     > > >     >     >         >> side.
> >     > > > > >     > > >     >     >         >> > >     c) At this point
> we
> > will
> >     > > have
> >     > > > to
> >     > > > > >     > specify a
> >     > > > > >     > > >     > warning or
> >     > > > > >     > > >     >     > clearly
> >     > > > > >     > > >     >     >         >> specify
> >     > > > > >     > > >     >     >         >> > > in docs that this
> > behavior is
> >     > > > about
> >     > > > > > to be
> >     > > > > >     > > > changed for
> >     > > > > >     > > >     > log
> >     > > > > >     > > >     >     > compaction.
> >     > > > > >     > > >     >     >         >> > > 4) a) In next release
> > of the
> >     > > > > > Broker(x+2), we
> >     > > > > >     > > > say that
> >     > > > > >     > > >     > only
> >     > > > > >     > > >     >     > Tombstone
> >     > > > > >     > > >     >     >         >> is
> >     > > > > >     > > >     >     >         >> > > used for log
> compaction
> > on the
> >     > > > > Broker
> >     > > > > > side.
> >     > > > > >     > > >     > Clients(x+1)
> >     > > > > >     > > >     >     > still is
> >     > > > > >     > > >     >     >         >> > > supported.
> >     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> > client
> >     > to
> >     > > > > > version
> >     > > > > >     > > > client(x+2) :
> >     > > > > >     > > >     > if
> >     > > > > >     > > >     >     > value is set
> >     > > > > >     > > >     >     >         >> to
> >     > > > > >     > > >     >     >         >> > > null, tombstone will
> > not be
> >     > set
> >     > > > > >     > automatically.
> >     > > > > >     > > > The
> >     > > > > >     > > >     > client
> >     > > > > >     > > >     >     > will have to
> >     > > > > >     > > >     >     >         >> > call
> >     > > > > >     > > >     >     >         >> > > setTombstone() to
> > actually set
> >     > > the
> >     > > > > >     > tombstone.
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > We should compare this
> >     > migration
> >     > > > > plan
> >     > > > > > with
> >     > > > > >     > the
> >     > > > > >     > > >     > migration
> >     > > > > >     > > >     >     > plan for
> >     > > > > >     > > >     >     >         >> magic
> >     > > > > >     > > >     >     >         >> > > byte bump and do
> > whatever
> >     > looks
> >     > > > > good.
> >     > > > > >     > > >     >     >         >> > > I am just worried that
> > if we
> >     > go
> >     > > > down
> >     > > > > > magic
> >     > > > > >     > > byte
> >     > > > > >     > > > route,
> >     > > > > >     > > >     >     > unless I am
> >     > > > > >     > > >     >     >         >> > missing
> >     > > > > >     > > >     >     >         >> > > something, it sounds
> > like
> >     > kafka
> >     > > > will
> >     > > > > > be
> >     > > > > >     > stuck
> >     > > > > >     > > > with
> >     > > > > >     > > >     >     > supporting both
> >     > > > > >     > > >     >     >         >> null
> >     > > > > >     > > >     >     >         >> > > value and tombstone
> bit
> > for
> >     > log
> >     > > > > > compaction
> >     > > > > >     > for
> >     > > > > >     > > > life
> >     > > > > >     > > >     > long,
> >     > > > > >     > > >     >     > which does
> >     > > > > >     > > >     >     >         >> not
> >     > > > > >     > > >     >     >         >> > > look like a good end
> > state.
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > Thanks,
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > Mayuresh
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016
> at
> > 9:32
> >     > AM,
> >     > > > > > Mayuresh
> >     > > > > >     > > > Gharat <
> >     > > > > >     > > >     >     >         >> > >
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > > wrote:
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > That's a very good
> > point
> >     > > which I
> >     > > > > > might
> >     > > > > >     > have
> >     > > > > >     > > > not
> >     > > > > >     > > >     >     > considered earlier.
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Here is a plan that
> I
> > can
> >     > > think
> >     > > > > of:
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Stage 1) The broker
> > from now
> >     > > on,
> >     > > > > up
> >     > > > > >     > converts
> >     > > > > >     > > > the
> >     > > > > >     > > >     > message
> >     > > > > >     > > >     >     > to have the
> >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> The
> > log
> >     > > > > compaction
> >     > > > > >     > thread
> >     > > > > >     > > > does log
> >     > > > > >     > > >     >     > compaction
> >     > > > > >     > > >     >     >         >> based
> >     > > > > >     > > >     >     >         >> > on
> >     > > > > >     > > >     >     >         >> > > > both null and
> > tombstone
> >     > > marker.
> >     > > > > > This is
> >     > > > > >     > our
> >     > > > > >     > > >     > transition
> >     > > > > >     > > >     >     > period.
> >     > > > > >     > > >     >     >         >> > > > Stage 2) The next
> > release we
> >     > > > only
> >     > > > > > say that
> >     > > > > >     > > log
> >     > > > > >     > > >     > compaction
> >     > > > > >     > > >     >     > is based
> >     > > > > >     > > >     >     >         >> on
> >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> > (Open
> >     > source
> >     > > > > > kafka makes
> >     > > > > >     > > > this as a
> >     > > > > >     > > >     >     > policy). By
> >     > > > > >     > > >     >     >         >> this
> >     > > > > >     > > >     >     >         >> > > time,
> >     > > > > >     > > >     >     >         >> > > > the organization
> > which is
> >     > > moving
> >     > > > > to
> >     > > > > > this
> >     > > > > >     > > > release
> >     > > > > >     > > >     > will be
> >     > > > > >     > > >     >     > sure that
> >     > > > > >     > > >     >     >         >> they
> >     > > > > >     > > >     >     >         >> > > > have gone through
> the
> > entire
> >     > > > > > transition
> >     > > > > >     > > > period.
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > My only goal of
> doing
> > this
> >     > is
> >     > > > that
> >     > > > > > Kafka
> >     > > > > >     > > > clearly
> >     > > > > >     > > >     >     > specifies the end
> >     > > > > >     > > >     >     >         >> > state
> >     > > > > >     > > >     >     >         >> > > > about what log
> > compaction
> >     > > means
> >     > > > > (is
> >     > > > > > it
> >     > > > > >     > null
> >     > > > > >     > > > value
> >     > > > > >     > > >     > or a
> >     > > > > >     > > >     >     > tombstone
> >     > > > > >     > > >     >     >         >> > marker,
> >     > > > > >     > > >     >     >         >> > > > but not both).
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > What do you think?
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Thanks,
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Mayuresh
> >     > > > > >     > > >     >     >         >> > > > .
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016
> > at 9:17
> >     > > AM,
> >     > > > > > Ismael
> >     > > > > >     > > Juma <
> >     > > > > >     > > >     >     > ismael@juma.me.uk>
> >     > > > > >     > > >     >     >         >> > wrote:
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >> One comment below.
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16,
> 2016
> > at
> >     > 5:08
> >     > > > PM,
> >     > > > > > Mayuresh
> >     > > > > >     > > > Gharat <
> >     > > > > >     > > >     >     >         >> > > >>
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > >> > wrote:
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> >    - If we don't
> > bump up
> >     > > the
> >     > > > > > magic
> >     > > > > >     > byte,
> >     > > > > >     > > > on the
> >     > > > > >     > > >     > broker
> >     > > > > >     > > >     >     > side, the
> >     > > > > >     > > >     >     >         >> > > broker
> >     > > > > >     > > >     >     >         >> > > >> >    will always
> > have to
> >     > look
> >     > > > at
> >     > > > > > both
> >     > > > > >     > > > tombstone
> >     > > > > >     > > >     > bit and
> >     > > > > >     > > >     >     > the value
> >     > > > > >     > > >     >     >         >> when
> >     > > > > >     > > >     >     >         >> > > do
> >     > > > > >     > > >     >     >         >> > > >> the
> >     > > > > >     > > >     >     >         >> > > >> >    compaction.
> > Assuming
> >     > we
> >     > > do
> >     > > > > > not bump
> >     > > > > >     > up
> >     > > > > >     > > > the
> >     > > > > >     > > >     > magic
> >     > > > > >     > > >     >     > byte,
> >     > > > > >     > > >     >     >         >> > > >> >    imagine the
> > broker
> >     > sees
> >     > > a
> >     > > > > > message
> >     > > > > >     > > which
> >     > > > > >     > > > does
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > have a
> >     > > > > >     > > >     >     >         >> tombstone
> >     > > > > >     > > >     >     >         >> > > bit
> >     > > > > >     > > >     >     >         >> > > >> >    set. The
> broker
> > does
> >     > not
> >     > > > > know
> >     > > > > > when
> >     > > > > >     > the
> >     > > > > >     > > >     > message was
> >     > > > > >     > > >     >     > produced
> >     > > > > >     > > >     >     >         >> (i.e.
> >     > > > > >     > > >     >     >         >> > > >> > whether
> >     > > > > >     > > >     >     >         >> > > >> >    the message
> has
> > been
> >     > up
> >     > > > > > converted or
> >     > > > > >     > > > not), it
> >     > > > > >     > > >     > has
> >     > > > > >     > > >     >     > to take a
> >     > > > > >     > > >     >     >         >> > further
> >     > > > > >     > > >     >     >         >> > > >> > look at
> >     > > > > >     > > >     >     >         >> > > >> >    the value to
> > see if it
> >     > > is
> >     > > > > > null or
> >     > > > > >     > not
> >     > > > > >     > > in
> >     > > > > >     > > >     > order to
> >     > > > > >     > > >     >     > determine
> >     > > > > >     > > >     >     >         >> if it
> >     > > > > >     > > >     >     >         >> > > is
> >     > > > > >     > > >     >     >         >> > > >> a
> >     > > > > >     > > >     >     >         >> > > >> >    tombstone. The
> > same
> >     > > logic
> >     > > > > has
> >     > > > > > to be
> >     > > > > >     > > put
> >     > > > > >     > > > on the
> >     > > > > >     > > >     >     > consumer as
> >     > > > > >     > > >     >     >         >> well
> >     > > > > >     > > >     >     >         >> > > >> because
> >     > > > > >     > > >     >     >         >> > > >> > the
> >     > > > > >     > > >     >     >         >> > > >> >    consumer does
> > not know
> >     > > if
> >     > > > > the
> >     > > > > >     > message
> >     > > > > >     > > > has
> >     > > > > >     > > >     > been up
> >     > > > > >     > > >     >     > converted or
> >     > > > > >     > > >     >     >         >> > not.
> >     > > > > >     > > >     >     >         >> > > >> >       - If we
> > upconvert
> >     > > while
> >     > > > > >     > appending,
> >     > > > > >     > > > this is
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > the case,
> >     > > > > >     > > >     >     >         >> > right?
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> If I understand you
> >     > > correctly,
> >     > > > > > this is
> >     > > > > >     > not
> >     > > > > >     > > >     > sufficient
> >     > > > > >     > > >     >     > because the
> >     > > > > >     > > >     >     >         >> log
> >     > > > > >     > > >     >     >         >> > > may
> >     > > > > >     > > >     >     >         >> > > >> have messages
> > appended
> >     > before
> >     > > > it
> >     > > > > > was
> >     > > > > >     > > > upgraded to
> >     > > > > >     > > >     > include
> >     > > > > >     > > >     >     > KIP-87.
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> Ismael
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > --
> >     > > > > >     > > >     >     >         >> > > > -Regards,
> >     > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > --
> >     > > > > >     > > >     >     >         >> > > -Regards,
> >     > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> > > (862) 250-7125
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > The information
> contained
> > in
> >     > this
> >     > > > > email
> >     > > > > > is
> >     > > > > >     > > > strictly
> >     > > > > >     > > >     >     > confidential and for
> >     > > > > >     > > >     >     >         >> > the use of the addressee
> > only,
> >     > > > unless
> >     > > > > >     > otherwise
> >     > > > > >     > > >     > indicated. If
> >     > > > > >     > > >     >     > you are
> >     > > > > >     > > >     >     >         >> not
> >     > > > > >     > > >     >     >         >> > the intended recipient,
> > please
> >     > do
> >     > > > not
> >     > > > > > read,
> >     > > > > >     > > copy,
> >     > > > > >     > > > use or
> >     > > > > >     > > >     >     > disclose to
> >     > > > > >     > > >     >     >         >> others
> >     > > > > >     > > >     >     >         >> > this message or any
> > attachment.
> >     > > > Please
> >     > > > > > also
> >     > > > > >     > > > notify the
> >     > > > > >     > > >     > sender
> >     > > > > >     > > >     >     > by
> >     > > > > >     > > >     >     >         >> replying
> >     > > > > >     > > >     >     >         >> > to this email or by
> > telephone
> >     > > > (+44(020
> >     > > > > > 7896
> >     > > > > >     > > 0011)
> >     > > > > >     > > > and
> >     > > > > >     > > >     > then
> >     > > > > >     > > >     >     > delete the
> >     > > > > >     > > >     >     >         >> email
> >     > > > > >     > > >     >     >         >> > and any copies of it.
> > Opinions,
> >     > > > > > conclusion
> >     > > > > >     > (etc)
> >     > > > > >     > > > that
> >     > > > > >     > > >     > do not
> >     > > > > >     > > >     >     > relate to
> >     > > > > >     > > >     >     >         >> the
> >     > > > > >     > > >     >     >         >> > official business of
> this
> >     > company
> >     > > > > shall
> >     > > > > > be
> >     > > > > >     > > > understood as
> >     > > > > >     > > >     >     > neither given
> >     > > > > >     > > >     >     >         >> nor
> >     > > > > >     > > >     >     >         >> > endorsed by it. IG is a
> > trading
> >     > > name
> >     > > > > of
> >     > > > > > IG
> >     > > > > >     > > Markets
> >     > > > > >     > > >     > Limited (a
> >     > > > > >     > > >     >     > company
> >     > > > > >     > > >     >     >         >> > registered in England
> and
> > Wales,
> >     > > > > company
> >     > > > > >     > number
> >     > > > > >     > > >     > 04008957) and
> >     > > > > >     > > >     >     > IG Index
> >     > > > > >     > > >     >     >         >> > Limited (a company
> > registered in
> >     > > > > > England and
> >     > > > > >     > > > Wales,
> >     > > > > >     > > >     > company
> >     > > > > >     > > >     >     > number
> >     > > > > >     > > >     >     >         >> > 01190902). Registered
> > address at
> >     > > > > Cannon
> >     > > > > > Bridge
> >     > > > > >     > > > House, 25
> >     > > > > >     > > >     >     > Dowgate Hill,
> >     > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
> > Markets
> >     > > > > Limited
> >     > > > > >     > > (register
> >     > > > > >     > > >     > number
> >     > > > > >     > > >     >     > 195355) and IG
> >     > > > > >     > > >     >     >         >> > Index Limited (register
> > number
> >     > > > 114059)
> >     > > > > > are
> >     > > > > >     > > > authorised
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > regulated by
> >     > > > > >     > > >     >     >         >> the
> >     > > > > >     > > >     >     >         >> > Financial Conduct
> > Authority.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> --
> >     > > > > >     > > >     >     >         >> -Regards,
> >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> (862) 250-7125
> >     > > > > >     > > >     >     >         >> The information contained
> > in this
> >     > > > email
> >     > > > > is
> >     > > > > >     > > strictly
> >     > > > > >     > > >     >     > confidential and for
> >     > > > > >     > > >     >     >         >> the use of the addressee
> > only,
> >     > > unless
> >     > > > > > otherwise
> >     > > > > >     > > >     > indicated. If
> >     > > > > >     > > >     >     > you are not
> >     > > > > >     > > >     >     >         >> the intended recipient,
> > please do
> >     > > not
> >     > > > > > read,
> >     > > > > >     > copy,
> >     > > > > >     > > > use or
> >     > > > > >     > > >     >     > disclose to others
> >     > > > > >     > > >     >     >         >> this message or any
> > attachment.
> >     > > Please
> >     > > > > > also
> >     > > > > >     > notify
> >     > > > > >     > > > the
> >     > > > > >     > > >     > sender
> >     > > > > >     > > >     >     > by replying
> >     > > > > >     > > >     >     >         >> to this email or by
> > telephone
> >     > > (+44(020
> >     > > > > > 7896
> >     > > > > >     > 0011)
> >     > > > > >     > > > and then
> >     > > > > >     > > >     >     > delete the email
> >     > > > > >     > > >     >     >         >> and any copies of it.
> > Opinions,
> >     > > > > > conclusion (etc)
> >     > > > > >     > > > that do
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > relate to the
> >     > > > > >     > > >     >     >         >> official business of this
> > company
> >     > > > shall
> >     > > > > be
> >     > > > > >     > > > understood as
> >     > > > > >     > > >     >     > neither given nor
> >     > > > > >     > > >     >     >         >> endorsed by it. IG is a
> > trading
> >     > name
> >     > > > of
> >     > > > > IG
> >     > > > > >     > Markets
> >     > > > > >     > > >     > Limited (a
> >     > > > > >     > > >     >     > company
> >     > > > > >     > > >     >     >         >> registered in England and
> > Wales,
> >     > > > company
> >     > > > > > number
> >     > > > > >     > > > 04008957)
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > IG Index
> >     > > > > >     > > >     >     >         >> Limited (a company
> > registered in
> >     > > > England
> >     > > > > > and
> >     > > > > >     > > Wales,
> >     > > > > >     > > >     > company
> >     > > > > >     > > >     >     > number
> >     > > > > >     > > >     >     >         >> 01190902). Registered
> > address at
> >     > > > Cannon
> >     > > > > > Bridge
> >     > > > > >     > > > House, 25
> >     > > > > >     > > >     >     > Dowgate Hill,
> >     > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> > Markets
> >     > > > Limited
> >     > > > > >     > (register
> >     > > > > >     > > > number
> >     > > > > >     > > >     >     > 195355) and IG
> >     > > > > >     > > >     >     >         >> Index Limited (register
> > number
> >     > > 114059)
> >     > > > > are
> >     > > > > >     > > > authorised and
> >     > > > > >     > > >     >     > regulated by the
> >     > > > > >     > > >     >     >         >> Financial Conduct
> Authority.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > --
> >     > > > > >     > > >     >     >         > -Regards,
> >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         > (862) 250-7125
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         --
> >     > > > > >     > > >     >     >         -Regards,
> >     > > > > >     > > >     >     >         Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         (862) 250-7125
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     The information contained in this
> > email
> >     > is
> >     > > > > > strictly
> >     > > > > >     > > > confidential
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > for the use of the addressee only,
> > unless
> >     > > > otherwise
> >     > > > > >     > > indicated.
> >     > > > > >     > > > If
> >     > > > > >     > > >     > you are
> >     > > > > >     > > >     >     > not the intended recipient, please do
> > not
> >     > read,
> >     > > > > > copy, use
> >     > > > > >     > or
> >     > > > > >     > > >     > disclose to
> >     > > > > >     > > >     >     > others this message or any
> attachment.
> > Please
> >     > > > also
> >     > > > > > notify
> >     > > > > >     > the
> >     > > > > >     > > > sender
> >     > > > > >     > > >     > by
> >     > > > > >     > > >     >     > replying to this email or by
> telephone
> >     > (+44(020
> >     > > > > 7896
> >     > > > > > 0011)
> >     > > > > >     > > and
> >     > > > > >     > > > then
> >     > > > > >     > > >     > delete
> >     > > > > >     > > >     >     > the email and any copies of it.
> > Opinions,
> >     > > > > conclusion
> >     > > > > > (etc)
> >     > > > > >     > > > that do
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > relate to the official business of
> this
> >     > company
> >     > > > > > shall be
> >     > > > > >     > > > understood
> >     > > > > >     > > >     > as
> >     > > > > >     > > >     >     > neither given nor endorsed by it. IG
> > is a
> >     > > trading
> >     > > > > > name of
> >     > > > > >     > IG
> >     > > > > >     > > > Markets
> >     > > > > >     > > >     >     > Limited (a company registered in
> > England and
> >     > > > Wales,
> >     > > > > > company
> >     > > > > >     > > > number
> >     > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> > company
> >     > > > > registered
> >     > > > > > in
> >     > > > > >     > > > England and
> >     > > > > >     > > >     > Wales,
> >     > > > > >     > > >     >     > company number 01190902). Registered
> > address
> >     > at
> >     > > > > > Cannon
> >     > > > > >     > Bridge
> >     > > > > >     > > > House,
> >     > > > > >     > > >     > 25
> >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both
> IG
> >     > Markets
> >     > > > > > Limited
> >     > > > > >     > > > (register
> >     > > > > >     > > >     > number
> >     > > > > >     > > >     >     > 195355) and IG Index Limited
> (register
> > number
> >     > > > > > 114059) are
> >     > > > > >     > > > authorised
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > regulated by the Financial Conduct
> > Authority.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > The information contained in this email is
> > strictly
> >     > > > > > confidential
> >     > > > > >     > > and
> >     > > > > >     > > > for
> >     > > > > >     > > >     > the use of the addressee only, unless
> > otherwise
> >     > > > > indicated.
> >     > > > > > If you
> >     > > > > >     > > > are not
> >     > > > > >     > > >     > the intended recipient, please do not read,
> > copy,
> >     > use
> >     > > > or
> >     > > > > > disclose
> >     > > > > >     > > to
> >     > > > > >     > > > others
> >     > > > > >     > > >     > this message or any attachment. Please also
> > notify
> >     > > the
> >     > > > > > sender by
> >     > > > > >     > > > replying
> >     > > > > >     > > >     > to this email or by telephone (+44(020 7896
> > 0011
> >     > <020%207896%200011>) and
> >     > > > > then
> >     > > > > > delete
> >     > > > > >     > > > the email
> >     > > > > >     > > >     > and any copies of it. Opinions, conclusion
> > (etc)
> >     > that
> >     > > > do
> >     > > > > > not
> >     > > > > >     > relate
> >     > > > > >     > > > to the
> >     > > > > >     > > >     > official business of this company shall be
> >     > understood
> >     > > > as
> >     > > > > > neither
> >     > > > > >     > > > given nor
> >     > > > > >     > > >     > endorsed by it. IG is a trading name of IG
> > Markets
> >     > > > > Limited
> >     > > > > > (a
> >     > > > > >     > > company
> >     > > > > >     > > >     > registered in England and Wales, company
> > number
> >     > > > 04008957)
> >     > > > > > and IG
> >     > > > > >     > > > Index
> >     > > > > >     > > >     > Limited (a company registered in England
> and
> > Wales,
> >     > > > > company
> >     > > > > >     > number
> >     > > > > >     > > >     > 01190902). Registered address at Cannon
> > Bridge
> >     > House,
> >     > > > 25
> >     > > > > > Dowgate
> >     > > > > >     > > > Hill,
> >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > > > number
> >     > > > > > 195355)
> >     > > > > >     > > > and IG
> >     > > > > >     > > >     > Index Limited (register number 114059) are
> >     > authorised
> >     > > > and
> >     > > > > >     > regulated
> >     > > > > >     > > > by the
> >     > > > > >     > > >     > Financial Conduct Authority.
> >     > > > > >     > > >     >
> >     > > > > >     > > >
> >     > > > > >     > > >
> >     > > > > >     > > > The information contained in this email is
> strictly
> >     > > > > confidential
> >     > > > > > and
> >     > > > > >     > for
> >     > > > > >     > > > the use of the addressee only, unless otherwise
> >     > indicated.
> >     > > If
> >     > > > > > you are
> >     > > > > >     > not
> >     > > > > >     > > > the intended recipient, please do not read, copy,
> > use or
> >     > > > > > disclose to
> >     > > > > >     > > others
> >     > > > > >     > > > this message or any attachment. Please also
> notify
> > the
> >     > > sender
> >     > > > > by
> >     > > > > >     > replying
> >     > > > > >     > > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > > > > delete the
> >     > > > > >     > > email
> >     > > > > >     > > > and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > > not
> >     > > > > > relate to
> >     > > > > >     > > the
> >     > > > > >     > > > official business of this company shall be
> > understood as
> >     > > > > neither
> >     > > > > > given
> >     > > > > >     > > nor
> >     > > > > >     > > > endorsed by it. IG is a trading name of IG
> Markets
> >     > Limited
> >     > > (a
> >     > > > > > company
> >     > > > > >     > > > registered in England and Wales, company number
> > 04008957)
> >     > > and
> >     > > > > IG
> >     > > > > > Index
> >     > > > > >     > > > Limited (a company registered in England and
> Wales,
> >     > company
> >     > > > > > number
> >     > > > > >     > > > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     > > > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> >     > > > > > 195355) and
> >     > > > > >     > IG
> >     > > > > >     > > > Index Limited (register number 114059) are
> > authorised and
> >     > > > > > regulated by
> >     > > > > >     > > the
> >     > > > > >     > > > Financial Conduct Authority.
> >     > > > > >     > > >
> >     > > > > >     > > The information contained in this email is strictly
> >     > > > confidential
> >     > > > > > and for
> >     > > > > >     > > the use of the addressee only, unless otherwise
> > indicated.
> >     > If
> >     > > > you
> >     > > > > > are not
> >     > > > > >     > > the intended recipient, please do not read, copy,
> > use or
> >     > > > disclose
> >     > > > > > to
> >     > > > > >     > others
> >     > > > > >     > > this message or any attachment. Please also notify
> > the
> >     > sender
> >     > > > by
> >     > > > > > replying
> >     > > > > >     > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > > delete
> >     > > > > > the
> >     > > > > >     > email
> >     > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     > > > > > relate to
> >     > > > > >     > the
> >     > > > > >     > > official business of this company shall be
> > understood as
> >     > > > neither
> >     > > > > > given
> >     > > > > >     > nor
> >     > > > > >     > > endorsed by it. IG is a trading name of IG Markets
> > Limited
> >     > (a
> >     > > > > > company
> >     > > > > >     > > registered in England and Wales, company number
> > 04008957)
> >     > and
> >     > > > IG
> >     > > > > > Index
> >     > > > > >     > > Limited (a company registered in England and Wales,
> > company
> >     > > > > number
> >     > > > > >     > > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     > > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     > > > 195355)
> >     > > > > > and IG
> >     > > > > >     > > Index Limited (register number 114059) are
> > authorised and
> >     > > > > > regulated by
> >     > > > > >     > the
> >     > > > > >     > > Financial Conduct Authority.
> >     > > > > >     > >
> >     > > > > >     > >
> >     > > > > >     > The information contained in this email is strictly
> >     > > confidential
> >     > > > > and
> >     > > > > > for
> >     > > > > >     > the use of the addressee only, unless otherwise
> > indicated. If
> >     > > you
> >     > > > > > are not
> >     > > > > >     > the intended recipient, please do not read, copy, use
> > or
> >     > > disclose
> >     > > > > to
> >     > > > > > others
> >     > > > > >     > this message or any attachment. Please also notify
> the
> > sender
> >     > > by
> >     > > > > > replying
> >     > > > > >     > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > delete
> >     > > > > > the email
> >     > > > > >     > and any copies of it. Opinions, conclusion (etc) that
> > do not
> >     > > > relate
> >     > > > > > to the
> >     > > > > >     > official business of this company shall be understood
> > as
> >     > > neither
> >     > > > > > given nor
> >     > > > > >     > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> >     > > > > company
> >     > > > > >     > registered in England and Wales, company number
> > 04008957) and
> >     > > IG
> >     > > > > > Index
> >     > > > > >     > Limited (a company registered in England and Wales,
> > company
> >     > > > number
> >     > > > > >     > 01190902). Registered address at Cannon Bridge House,
> > 25
> >     > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     > > 195355)
> >     > > > > > and IG
> >     > > > > >     > Index Limited (register number 114059) are authorised
> > and
> >     > > > regulated
> >     > > > > > by the
> >     > > > > >     > Financial Conduct Authority.
> >     > > > > >     >
> >     > > > > >     >
> >     > > > > >
> >     > > > > >
> >     > > > > > The information contained in this email is strictly
> > confidential
> >     > and
> >     > > > for
> >     > > > > > the use of the addressee only, unless otherwise indicated.
> > If you
> >     > are
> >     > > > not
> >     > > > > > the intended recipient, please do not read, copy, use or
> > disclose
> >     > to
> >     > > > > others
> >     > > > > > this message or any attachment. Please also notify the
> > sender by
> >     > > > replying
> >     > > > > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then delete the
> >     > > > > email
> >     > > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> >     > > to
> >     > > > > the
> >     > > > > > official business of this company shall be understood as
> > neither
> >     > > given
> >     > > > > nor
> >     > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> >     > company
> >     > > > > > registered in England and Wales, company number 04008957)
> > and IG
> >     > > Index
> >     > > > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > > Hill,
> >     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> >     > and
> >     > > > IG
> >     > > > > > Index Limited (register number 114059) are authorised and
> > regulated
> >     > > by
> >     > > > > the
> >     > > > > > Financial Conduct Authority.
> >     > > > > >
> >     > > > > The information contained in this email is strictly
> > confidential and
> >     > > for
> >     > > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > > not
> >     > > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > > others
> >     > > > > this message or any attachment. Please also notify the sender
> > by
> >     > > replying
> >     > > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the
> >     > > > email
> >     > > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> >     > to
> >     > > > the
> >     > > > > official business of this company shall be understood as
> > neither
> >     > given
> >     > > > nor
> >     > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > > registered in England and Wales, company number 04008957) and
> > IG
> >     > Index
> >     > > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > Hill,
> >     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > > IG
> >     > > > > Index Limited (register number 114059) are authorised and
> > regulated
> >     > by
> >     > > > the
> >     > > > > Financial Conduct Authority.
> >     > > > >
> >     > > > The information contained in this email is strictly
> confidential
> > and
> >     > for
> >     > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > not
> >     > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > others
> >     > > > this message or any attachment. Please also notify the sender
> by
> >     > replying
> >     > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the
> >     > > email
> >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > > the
> >     > > > official business of this company shall be understood as
> neither
> > given
> >     > > nor
> >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> >     > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > IG
> >     > > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > > the
> >     > > > Financial Conduct Authority.
> >     > > >
> >     > > The information contained in this email is strictly confidential
> > and for
> >     > > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > > the intended recipient, please do not read, copy, use or disclose
> > to
> >     > others
> >     > > this message or any attachment. Please also notify the sender by
> > replying
> >     > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the email
> >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > the
> >     > > official business of this company shall be understood as neither
> > given
> >     > nor
> >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > > Limited (a company registered in England and Wales, company
> number
> >     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > the
> >     > > Financial Conduct Authority.
> >     > >
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jay,

Agreed this stemmed as offshoot from KIP-82.

Which our main driver for was to be able to have some headers for a null value as such for our routing, audit, tracing and a few other bits which currently we are forced to do with a message wrapper, if we all agreed on KIP-82 that we need native headers and look to implement that the push for this would dissipate.

This KIP would allow for though one use case that comes to mind we could see which is to have business data with a delete. Though as said this isn't something we are pushing for think really we would have.

As such in summary yes, if you want to fully support KIP-82 and we can get that agreed in principle and a target release version, I think quite a few guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.

Cheers
Mike

Sent using OWA for iPhone
________________________________________
From: Jay Kreps <ja...@confluent.io>
Sent: Monday, December 19, 2016 8:51:23 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hey Michael,

Here is the compatibility concern I have:

   1. You have a consumer app that relies on value == null to indicate a
   delete (current semantics).
   2. You upgrade Kafka and your clients.
   3. Some producer starts using the tombstone field in combination with
   non-null.

I share Ismael's dislike of setting tombstones on records with null values.
This makes sense as a transitional state, but as an end state its a bit
weird. You'd expect to be able to mix null values and tombstones, and have
the null values remain and the tombstones get compacted. However what will
happen is both will be compacted and upon debugging this you'll learn that
we sometimes use null in the value to indicate tombstone. Ismael's solution
is a bigger compatibility break, though, so not sure if that is better.

My other question is the relationship to KIP-82. My read is that this KIP
solves some but not all of the problems KIP-82 is intended for. KIP-82, on
the other hand, seems to address most of the motivating uses for this KIP.
The exception is maybe item (5) on the list where you want to simultaneous
delete and convey some information to subscribers, but I couldn't construct
a concrete examples for that one. Do we need to rationalize these two KIPs?
That is, do you still advocate this proposal if we do KIP-82 and vice
versa? As you may have noticed I'm somewhat emotionally invested in the
simplicity of the core data model, so my default position is let's try to
avoid stuffing more stuff in, but if we have to add stuff I like each of
these individually more than doing both. :-)

-Jay




On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jay
>
> I disagree here that we are breaking any compatibility, we went through
> this on the discussion thread in fact with the help of that thread is how
> the kip came to the solution.
>
> Also on the supported combinations front you mention, we are not taking
> anything away afaik.
>
> Currently supported are only are:
> Null value = delete
> Non-null value = non delete
>
> With this kip we would support
> Null value + tombstone = delete
> Non null value + tombstone = delete
> Non null value + no tombstone = non delete
>
> As for the alternative idea, this is simply a new policy, how can there be
> confusion here? For this policy it would be explicit that tombstone marker
> would need to be set for a delete.
>
> I'm going to vent a little now as starting to get quite frustrated.
>
> We are going round in circles on kip-82 as per use cases there is now many
> use cases, how many more are needed? just because confluent don't see these
> doesn't mean they aren't real use cases other have, this is the point of
> the Apache foundation, it shouldn't be the view of just one organisation.
> It really is getting a feeling of the NIH syndrome. Rather than it being
> constructive on discussion of the implementation detail.
>
> kip-87 spawned from as on the kip call we all agreed this was needed. And
> would at least allow a custom wrapper be supported in a compacted topic,
> allowing meta data. Which again now I feel we are spinning wheels, and
> simply finding reasons not support it.
>
> Cheers
> Mike
>
>
>
> Sent using OWA for iPad
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Friday, December 16, 2016 7:09:23 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hey Michael,
>
> I do think it might have been better had we started with a separate concept
> of null vs delete. Given that we are where we are, I'm not sure that the
> possible cures we explored so far are better than the disease.
>
> I apologize for coming to this late, but I didn't really understand the
> implications of the KIP and that we'd be breaking compatibility with
> existing apps until the vote had begun, and in my defense I'm not sure the
> other folks voting did either.
>
> I think we all agree there are many existing apps that are built with the
> assumption of "null value non-tombstone" and it isn't possible to
> disambiguate these from tombstones on the producer. It isn't that anyone is
> saying we have to support all four possibilities at once, it's that we
> simply can't orphan one of the existing combinations or our users will eat
> us!
>
> If I've understood your alternate solution of adding another setting for
> compaction, I think this does fix the compatibility problem, but it adds an
> odd mode the user has to add on all their topics. While the current state
> is easily explainable, the resulting state where the meaning of tombstone
> and null are overlapping and ambiguous and dependent on a compaction
> setting that could change out of band or not be in sync with your code in
> some environment seems worse to me then where we currently are. I think the
> question is how would this combination be explained to users and does it
> make sense?
>
> -Jay
>
>
>
> On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Chaps,
> >
> > Can we either get one more +1 binding (we have 2 already) on the
> existing?
> >
> > Or have some response on the below possible alternative. We are keen to
> > get working on this, so we make next feature release.
> >
> > Cheers
> > Mike
> >
> >
> > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> >     Hi Ismael
> >
> >     Did you see our email this morning, what's your thoughts on this
> > approach to instead we simply have a brand new policy?
> >
> >     Cheers
> >     Mike
> >
> >
> >     Sent using OWA for iPhone
> >     ________________________________________
> >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
> Juma <
> > ismael@juma.me.uk>
> >     Sent: Tuesday, December 13, 2016 11:30:05 AM
> >     To: dev@kafka.apache.org
> >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> >     Yes, this is actually tricky to do in a way where we both have the
> > desired
> >     semantics and maintain compatibility. When someone creates a
> >     `ProducerRecord` with a `null` value today, the producer doesn't know
> > if
> >     it's meant to be a tombstone or not. For V3 messages, it's easy when
> > the
> >     constructor that takes a tombstone is used. However, if any other
> >     constructor is used, it's not clear. A couple of options I can think
> > of,
> >     none of them particularly nice:
> >
> >     1. Have a third state where tombstone = unknown and the broker would
> > set
> >     the tombstone bit if the value was null and the topic was compacted.
> > People
> >     that wanted to pass a non-null value for the tombstone would have to
> > use
> >     the constructor that takes a tombstone. The drawbacks: third state
> for
> >     tombstone in message format, message conversion at the broker for a
> > common
> >     case.
> >
> >     2. Extend MetadataResponse to optionally include topic configs, which
> > would
> >     make it possible for the producer to be smarter about setting the
> >     tombstone. It would only do it if a tombstone was not passed
> > explicitly,
> >     the value was null and the topic was compacted. The main drawback is
> > that
> >     the producer would be getting a bit more data for each topic even
> > though it
> >     probably won't use it most of the time. Extending MetadataResponse to
> >     return topic configs would be useful for other reasons as well, so
> that
> >     part seems OK.
> >
> >     In addition, for both proposals, we could consider adding warnings to
> > the
> >     documentation that the behaviour of the constructors that don't take
> a
> >     tombstone would change in the next major release so that tombstone =
> > false.
> >     Not sure if this would be worth it though.
> >
> >     Ismael
> >
> >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> > ewen@confluent.io>
> >     wrote:
> >
> >     > Michael,
> >     >
> >     > It kind of depends on how you want to interpret the tombstone flag.
> > If it's
> >     > purely a producer-facing Kafka-level thing that we treat as
> internal
> > to the
> >     > broker and log cleaner once the record is sent, then your approach
> > makes
> >     > sense. You're just moving copying the null-indicates-delete
> behavior
> > of the
> >     > old constructor into the tombstone flag.
> >     >
> >     > However, if you want this change to more generally decouple the
> idea
> > of
> >     > deletion and null values, then you are sometimes converting what
> > might be a
> >     > completely valid null value that doesn't indicate deletion into a
> >     > tombstone. Downstream applications could potentially handle these
> > cases
> >     > differently given the separation of deletion from value.
> >     >
> >     > I guess the question is if we want to try to support the latter
> even
> > for
> >     > topics where we have older produce requests. An example where this
> > could
> >     > come up is in something like a CDC Connector. If we try to support
> > the
> >     > semantic difference, a connector might write changes to Kafka using
> > the
> >     > tombstone flag to indicate when a row was truly deleted (vs an
> > update that
> >     > sets it to null but still present; this probably makes more sense
> > for CDC
> >     > from document stores or extracting single columns). There are
> various
> >     > reasons we might want to maintain the full log and not turn
> > compaction on
> >     > (or just use a time-based retention policy), but downstream
> > applications
> >     > might care to know the difference between a delete and a null
> value.
> > In
> >     > fact both versions of the same log (compacted and time-retention)
> > could be
> >     > useful and I don't think it'll be uncommon to maintain both or use
> > KIP-71
> >     > to maintain a hybrid compacted/retention topic.
> >     >
> >     > -Ewen
> >     >
> >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > wrote:
> >     >
> >     > > Hi Jay,
> >     > >
> >     > > Why wouldn't that work, the tombstone value is only looked at by
> > the
> >     > > broker, on a topic configured for compaction as such is benign on
> > non
> >     > > compacted topics. This is as much as sending a null value
> currently
> >     > >
> >     > >
> >     > > Regards
> >     > > Mike
> >     > >
> >     > >
> >     > >
> >     > > Sent using OWA for iPhone
> >     > > ________________________________________
> >     > > From: Jay Kreps <ja...@confluent.io>
> >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
> >     > > To: dev@kafka.apache.org
> >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > >
> >     > > Hey Michael,
> >     > >
> >     > > I'm not quite sure that works as that would translate ALL null
> > values to
> >     > > tombstones, even for non-compacted topics that use null as an
> > acceptable
> >     > > value sent by the producer and expected by the consumer.
> >     > >
> >     > > -Jay
> >     > >
> >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > > wrote:
> >     > >
> >     > > > Hi Ewen,
> >     > > >
> >     > > > I think the easiest way to show this is with code.
> >     > > >
> >     > > > As you can see we keep the existing behaviour for code/binaries
> > calling
> >     > > > the pre-existing constructors, whereby if the value is null the
> >     > tombstone
> >     > > > is set to true.
> >     > > >
> >     > > > Regards
> >     > > > Mike
> >     > > >
> >     > > >
> >     > > >
> >     > > >     /**
> >     > > >      * Creates a record with a specified timestamp to be sent
> to
> > a
> >     > > > specified topic and partition
> >     > > >      *
> >     > > >      * @param topic The topic the record will be appended to
> >     > > >      * @param partition The partition to which the record
> should
> > be
> >     > sent
> >     > > >      * @param timestamp The timestamp of the record
> >     > > >      * @param tombstone if the record should be treated as a
> > tombstone
> >     > if
> >     > > > the topic is compacted
> >     > > >      * @param key The key that will be included in the record
> >     > > >      * @param value The record contents
> >     > > >      */
> >     > > >     public ProducerRecord(String topic, Integer partition,
> > Boolean
> >     > > > tombstone, Long timestamp, K key, V value) {
> >     > > >         if (topic == null)
> >     > > >             throw new IllegalArgumentException("Topic cannot
> be
> >     > null.");
> >     > > >         if (timestamp != null && timestamp < 0)
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Invalid timestamp: %d.
> > Timestamp
> >     > > should
> >     > > > always be non-negative or null.", timestamp));
> >     > > >         if (partition != null && partition < 0)
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Invalid partition: %d.
> > Partition
> >     > > number
> >     > > > should always be non-negative or null.", partition));
> >     > > >         if (!tombstone && value == null){
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Tombstone must be true if
> null
> >     > > value"));
> >     > > >         }
> >     > > >         this.topic = topic;
> >     > > >         this.partition = partition;
> >     > > >         this.tombstone = tombstone;
> >     > > >         this.key = key;
> >     > > >         this.value = value;
> >     > > >         this.timestamp = timestamp;
> >     > > >     }
> >     > > >
> >     > > >     public ProducerRecord(String topic, Integer partition, Long
> >     > > timestamp,
> >     > > > K key, V value) {
> >     > > >         this(topic, partition, value == null, timestamp, key,
> > value);
> >     > > >     }
> >     > > > ________________________________________
> >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> >     > > > Sent: Sunday, December 11, 2016 5:45 AM
> >     > > > To: dev@kafka.apache.org
> >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > >
> >     > > > It seemed like this was addressed in the migration section,
> > wasn't it?
> >     > > The
> >     > > > V2 vs V3 requests and the fact that the broker will downgrade
> the
> >     > message
> >     > > > format (losing zero copy) if you issues a V2 request to a
> broker
> > using
> >     > V3
> >     > > > format handles compatibility, does it not? The existing
> > consumers won't
> >     > > see
> >     > > > the extra metadata in the value, but they will get a null
> > instead and
> >     > > treat
> >     > > > it as a tombstone. Certainly there is a performance impact, but
> > it
> >     > seemed
> >     > > > compatible.
> >     > > >
> >     > > > I'm worried about this though:
> >     > > >
> >     > > >
> >     > > >    - *NOTE *: With the new version of producer client using
> >     > > ProduceRequest
> >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set)
> > and
> >     > null
> >     > > > value
> >     > > >    should not be allowed as the older version of consumer using
> >     > > > FetchRequest
> >     > > >    V2 will think of this as a tombstone when its actually not.
> >     > > >
> >     > > >
> >     > > > Unless I'm misunderstanding, this ends up breaking binary
> > compatibility
> >     > > for
> >     > > > the Java API. It sounds like this suggests that passing a null
> > value to
> >     > > the
> >     > > > existing ProducerRecord constructors would generate an
> exception
> > since
> >     > > you
> >     > > > didn't explicitly enable the tombstone (via whatever new
> > constructor is
> >     > > > provided). But that means you can't swap in a newer client jar
> > without
> >     > > > recompiling and get the same behavior if your app deletes keys
> > using
> >     > the
> >     > > > current approach because your app will start throwing
> > exceptions. Maybe
> >     > > > this is a tradeoff we're ok with, but we've tried pretty hard
> to
> > avoid
> >     > > > breaking compatibility like this so far -- it makes picking up
> > bug
> >     > fixes
> >     > > in
> >     > > > the clients harder for users of frameworks that have to pin to
> > earlier
> >     > > > default versions for compatibility.
> >     > > >
> >     > > > But then later the KIP says:
> >     > > >
> >     > > >
> >     > > >    - The old constructors for ProducerRecord without this
> > parameter
> >     > will
> >     > > be
> >     > > >    kept but updated so that their default behaviour if setting
> > null
> >     > value
> >     > > > will
> >     > > >    be the tombstone will be set to true to keep existing
> > behaviour.
> >     > > >
> >     > > >
> >     > > > So maybe I am misinterpreting something.
> >     > > >
> >     > > > And just a nit re: motivation section, item 6 would be clearer
> > for a
> >     > > union
> >     > > > schema with null (or optional schemas in other formats), e.g.
> > [null,
> >     > > > string], in which case losing the schema truly is losing
> > information
> >     > > > (whereas null is already the only valid value for a pure null
> > schema).
> >     > > >
> >     > > > -Ewen
> >     > > >
> >     > > >
> >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> >     > >
> >     > > > wrote:
> >     > > >
> >     > > > > Hi Jay,
> >     > > > >
> >     > > > > Good point this detail is missing in the KIP write up. Ive
> > added this
> >     > > > now.
> >     > > > >
> >     > > > > Essentially simply just upgrading the clients we do not
> expect
> > any
> >     > > client
> >     > > > > app code change needed.
> >     > > > >
> >     > > > > Cheers
> >     > > > > Mike
> >     > > > > ________________________________________
> >     > > > > From: Jay Kreps <ja...@confluent.io>
> >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
> >     > > > > To: dev@kafka.apache.org
> >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > > >
> >     > > > > Michael,
> >     > > > >
> >     > > > > The compatibility section goes through the migration path,
> but
> > isn't
> >     > > the
> >     > > > > bigger compatibility issue with existing apps? There are many
> >     > (probably
> >     > > > > thousands) of apps in production that use this feature and
> > send null
> >     > to
> >     > > > > mean delete. It seems like this would break compatibility
> with
> > them,
> >     > > and
> >     > > > > they would have to be rewritten to right?
> >     > > > >
> >     > > > > -Jay
> >     > > > >
> >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> >     > Michael.Pearce@ig.com
> >     > > >
> >     > > > > wrote:
> >     > > > >
> >     > > > > > Hi Jun,
> >     > > > > >
> >     > > > > > 4) On v3 we honour the tombstone. As such we expect it to
> be
> > set
> >     > > > > correctly
> >     > > > > > as per the KIP.
> >     > > > > >
> >     > > > > > 4.1) why would we want to produce an error when its v3?
> This
> > is the
> >     > > > exact
> >     > > > > > purpose to support non-null tombstone's
> >     > > > > > 4.2) again here im unclear on the question on the v3,
> produce
> >     > request
> >     > > > we
> >     > > > > > expect the tombstone flag to be set correctly.
> >     > > > > >
> >     > > > > > 4.4) compaction only occurs on compacted topics, the bit
> > makes no
> >     > > > > > difference and not looked at on un-compacted (time/size
> based
> >     > > > eviction).
> >     > > > > >
> >     > > > > >
> >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> >     > > > > >
> >     > > > > >     Hi, Michael,
> >     > > > > >
> >     > > > > >     4. Then, I think I misunderstood this point. Could you
> > document
> >     > > the
> >     > > > > >     following points in the wiki?
> >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
> > non-null
> >     > value,
> >     > > > > does
> >     > > > > > the
> >     > > > > >     send() get an error or does the producer automatically
> > set the
> >     > > > value
> >     > > > > to
> >     > > > > >     null?
> >     > > > > >     4.2 If producer V3 doesn't set tombstone, but provides
> a
> > null
> >     > > > value,
> >     > > > > > does
> >     > > > > >     the send() get an error or does the producer
> > automatically sets
> >     > > the
> >     > > > > >     tombstone?
> >     > > > > >     4.3 Does the broker only expect messages that (a) have
> no
> >     > > tombstone
> >     > > > > and
> >     > > > > >     non-null value; (b) have tombstone and null value and
> > reject
> >     > the
> >     > > > > > messages
> >     > > > > >     with an error code otherwise?
> >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
> > compacted
> >     > on
> >     > > > > not?
> >     > > > > >
> >     > > > > >     Thanks,
> >     > > > > >
> >     > > > > >     Jun
> >     > > > > >
> >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> >     > > > > Michael.Pearce@ig.com
> >     > > > > > >
> >     > > > > >     wrote:
> >     > > > > >
> >     > > > > >     > Not at all.  This only acts on compacted topics just
> > as what
> >     > > > occurs
> >     > > > > > today
> >     > > > > >     >
> >     > > > > >     > Sent using OWA for iPhone
> >     > > > > >     > ________________________________________
> >     > > > > >     > From: Jun Rao <ju...@confluent.io>
> >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> >     > > > > >     > To: dev@kafka.apache.org
> >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> > Flag
> >     > > > > >     >
> >     > > > > >     > Hi, Michael,
> >     > > > > >     >
> >     > > > > >     > 4. Hmm, does that mean the new client library can
> > never send
> >     > a
> >     > > > null
> >     > > > > > message
> >     > > > > >     > even to a regular topic? This seems like a change of
> > the
> >     > > existing
> >     > > > > > behavior.
> >     > > > > >     >
> >     > > > > >     > Thanks,
> >     > > > > >     >
> >     > > > > >     > Jun
> >     > > > > >     >
> >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> >     > > > > > Michael.Pearce@ig.com>
> >     > > > > >     > wrote:
> >     > > > > >     >
> >     > > > > >     > > Hi Jun,
> >     > > > > >     > >
> >     > > > > >     > > Re 4) That's because we expect the tombstone value
> > to be
> >     > set
> >     > > > > > correctly if
> >     > > > > >     > > message bit is 2, as such if an older client sends
> > in on
> >     > old
> >     > > > > > message the
> >     > > > > >     > > message is upcast and the bit is set correctly. And
> > such no
> >     > > > > longer
> >     > > > > > need
> >     > > > > >     > to
> >     > > > > >     > > check the value. Mayuresh can you confirm my
> > thinking and
> >     > > > > > understanding
> >     > > > > >     > of
> >     > > > > >     > > what we've discussed?
> >     > > > > >     > >
> >     > > > > >     > > The second point I understand what you're getting
> at
> > now my
> >     > > > > > apologies.
> >     > > > > >     > Yes
> >     > > > > >     > > this makes sense to save on touching the message,
> if
> > we're
> >     > > the
> >     > > > > > only kip
> >     > > > > >     > > going in, in this release.
> >     > > > > >     > >
> >     > > > > >     > > Cheers
> >     > > > > >     > > Mike
> >     > > > > >     > >
> >     > > > > >     > > Sent using OWA for iPhone
> >     > > > > >     > > ________________________________________
> >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
> >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> >     > > > > >     > > To: dev@kafka.apache.org
> >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone Flag
> >     > > > > >     > >
> >     > > > > >     > > Hi, Michael,
> >     > > > > >     > >
> >     > > > > >     > > 4. Is this updated in the wiki? The text "If the
> > magic byte
> >     > > on
> >     > > > > > message is
> >     > > > > >     > > 2, the broker should use the tombstone bit for log
> >     > > compaction."
> >     > > > > > doesn't
> >     > > > > >     > > seem to have changed.
> >     > > > > >     > >
> >     > > > > >     > > 2. My point is that if we change the message format
> > just
> >     > for
> >     > > > this
> >     > > > > > KIP, we
> >     > > > > >     > > should consider whether it's worth optimizing the
> > down
> >     > > > conversion
> >     > > > > > path
> >     > > > > >     > > (i.e., decide whether a conversion is needed by
> just
> >     > looking
> >     > > at
> >     > > > > the
> >     > > > > >     > > tombstone bit in the wrapper message) since
> > tombstone will
> >     > be
> >     > > > > used
> >     > > > > >     > rarely.
> >     > > > > >     > > However, if the message format change here is
> > combined with
> >     > > > other
> >     > > > > > KIPs,
> >     > > > > >     > > then this optimization likely won't be needed. The
> > latter
> >     > > > > probably
> >     > > > > > makes
> >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> > think?
> >     > > > > >     > >
> >     > > > > >     > > Other than those, +1 from me,
> >     > > > > >     > >
> >     > > > > >     > > Thanks,
> >     > > > > >     > >
> >     > > > > >     > > Jun
> >     > > > > >     > >
> >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> >     > > > > > Michael.Pearce@ig.com>
> >     > > > > >     > > wrote:
> >     > > > > >     > >
> >     > > > > >     > > > Hi Jun
> >     > > > > >     > > >
> >     > > > > >     > > > do we have your vote on this now?
> >     > > > > >     > > >
> >     > > > > >     > > > Any other concerns?
> >     > > > > >     > > >
> >     > > > > >     > > > Cheers
> >     > > > > >     > > > Mike
> >     > > > > >     > > >
> >     > > > > >     > > > Sent using OWA for iPhone
> >     > > > > >     > > > ________________________________________
> >     > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> >     > > > > >     > > > To: dev@kafka.apache.org
> >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone
> >     > Flag
> >     > > > > >     > > >
> >     > > > > >     > > > Hi Jun,
> >     > > > > >     > > >
> >     > > > > >     > > > Have updated. Thanks again for the feedback.
> >     > > > > >     > > >
> >     > > > > >     > > > Agree yes we should align up when it gets to
> that,
> > I
> >     > assume
> >     > > > > > you've
> >     > > > > >     > > flagged
> >     > > > > >     > > > the same in the other KIP?
> >     > > > > >     > > >
> >     > > > > >     > > > I think for now let's get this KIP approved, then
> > we can
> >     > > > start
> >     > > > > > the work
> >     > > > > >     > > to
> >     > > > > >     > > > get an initial PR, then we can discuss how to
> > align the
> >     > two
> >     > > > if
> >     > > > > > needed.
> >     > > > > >     > > >
> >     > > > > >     > > > Cheers,
> >     > > > > >     > > > Mike
> >     > > > > >     > > >
> >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
> jun@confluent.io>
> >     > wrote:
> >     > > > > >     > > >
> >     > > > > >     > > >     Hi, Michael,
> >     > > > > >     > > >
> >     > > > > >     > > >     For 2), this is fine. Could you update the
> KIP
> > wiki
> >     > to
> >     > > > make
> >     > > > > > this
> >     > > > > >     > and
> >     > > > > >     > > > other
> >     > > > > >     > > >     points clearer? Other than that, the KIP
> looks
> > good
> >     > to
> >     > > > me.
> >     > > > > >     > > >
> >     > > > > >     > > >     An orthogonal thing is that there are other
> > KIPs such
> >     > > as
> >     > > > > > KIP-98
> >     > > > > >     > that
> >     > > > > >     > > > also
> >     > > > > >     > > >     intend to change the message format. If they
> > all get
> >     > > > > > approved, we
> >     > > > > >     > > > should
> >     > > > > >     > > >     think about whether it's better to just bump
> > up the
> >     > > magic
> >     > > > > > byte once
> >     > > > > >     > > to
> >     > > > > >     > > >     incorporate multiple format changes like we
> > did in
> >     > > > > > KIP-31/KIP-32.
> >     > > > > >     > > >
> >     > > > > >     > > >     Thanks,
> >     > > > > >     > > >
> >     > > > > >     > > >     Jun
> >     > > > > >     > > >
> >     > > > > >     > > >
> >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
> Pearce
> > <
> >     > > > > >     > > Michael.Pearce@ig.com>
> >     > > > > >     > > >     wrote:
> >     > > > > >     > > >
> >     > > > > >     > > >     > Hi Jao
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > Thanks for the response. Sorry for slow
> > reply, both
> >     > > > with
> >     > > > > > personal
> >     > > > > >     > > > sickness
> >     > > > > >     > > >     > and also battling some critical issues
> > encountered
> >     > > > since
> >     > > > > >     > upgrading
> >     > > > > >     > > to
> >     > > > > >     > > >     > 0.10.1.0
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > 1) Thans for spotting, Document error where
> > we
> >     > > branched
> >     > > > > > this KIP
> >     > > > > >     > > from
> >     > > > > >     > > >     > KIP-82, will get that removed.
> >     > > > > >     > > >     > 2) Intent is to do this just at the record
> > message
> >     > > > level.
> >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> >     > > corrected.
> >     > > > > >     > > >     > 4) As per discussion thread we will support
> >     > > tombstone +
> >     > > > > > null
> >     > > > > >     > value,
> >     > > > > >     > > >     > tombstone + non null value, no tombstone +
> > null
> >     > > value.
> >     > > > > >     > > >     > 5) I believe this was in the discussion
> > thread,
> >     > > > @Mayuresh
> >     > > > > > is this
> >     > > > > >     > > >     > something we've overlooked? I thought we
> > would down
> >     > > > > > convert and
> >     > > > > >     > > > remove the
> >     > > > > >     > > >     > value so the old consumer had existing
> > behavior, or
> >     > > is
> >     > > > > > there
> >     > > > > >     > > > something we
> >     > > > > >     > > >     > haven't thought about?
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > Cheers
> >     > > > > >     > > >     > Mike
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> > jun@confluent.io>
> >     > > > > wrote:
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Hi, Michael,
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Thanks for the KIP. A few comments
> below.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     1. The message format change contains
> >     > > > "HeadersLength
> >     > > > > >     > Headers".
> >     > > > > >     > > > Is that
> >     > > > > >     > > >     >     intended?
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     2. For compressed messageset, is the
> > tombstone
> >     > > bit
> >     > > > > > only set
> >     > > > > >     > at
> >     > > > > >     > > > the
> >     > > > > >     > > >     > shallow
> >     > > > > >     > > >     >     level? Do we always leave that bit in
> the
> >     > wrapper
> >     > > > > > message
> >     > > > > >     > > unset?
> >     > > > > >     > > > An
> >     > > > > >     > > >     >     alternative is to set the tombstone bit
> > in the
> >     > > > > wrapper
> >     > > > > > if at
> >     > > > > >     > > > least one
> >     > > > > >     > > >     >     inner message has the tombstone bit
> set.
> > This
> >     > > makes
> >     > > > > > things a
> >     > > > > >     > > bit
> >     > > > > >     > > > more
> >     > > > > >     > > >     >     complicated, but we could potentially
> > exploit
> >     > > that
> >     > > > > for
> >     > > > > >     > > > optimizing down
> >     > > > > >     > > >     >     conversion. For example, we only need
> to
> >     > convert
> >     > > > > > messages
> >     > > > > >     > with
> >     > > > > >     > > > magic 2
> >     > > > > >     > > >     > to
> >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit
> > is set
> >     > > > > > (conversion is
> >     > > > > >     > > > always
> >     > > > > >     > > >     > needed
> >     > > > > >     > > >     >     from magic 2 to magic 0). Not sure if
> the
> >     > > > > optimization
> >     > > > > > is
> >     > > > > >     > worth
> >     > > > > >     > > > the
> >     > > > > >     > > >     >     complexity though.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     3. The referencing of the new version
> of
> >     > > > > >     > > > ProducerRequest/FetchRequest
> >     > > > > >     > > >     > is
> >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> > convention
> >     > > > starts
> >     > > > > at
> >     > > > > >     > version
> >     > > > > >     > > > at 0, I
> >     > > > > >     > > >     >     think the new version would be 3.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     4. "If the magic byte on message is 2,
> > the
> >     > broker
> >     > > > > > should use
> >     > > > > >     > > the
> >     > > > > >     > > >     > tombstone
> >     > > > > >     > > >     >     bit for log compaction." What about
> null
> > value?
> >     > > My
> >     > > > > >     > > understanding
> >     > > > > >     > > > is
> >     > > > > >     > > >     > that
> >     > > > > >     > > >     >     null value will be treated the same as
> > setting
> >     > > the
> >     > > > > > tombstone
> >     > > > > >     > > bit.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     5. For the migration path, it would be
> > useful
> >     > to
> >     > > > > > describe the
> >     > > > > >     > > > down
> >     > > > > >     > > >     >     conversion path to consumers (i.e.,
> > brokers on
> >     > > > > message
> >     > > > > > format
> >     > > > > >     > > > 0.10.2
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     consumers on older version).
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Thanks,
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Jun
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
> Michael
> >     > Pearce <
> >     > > > > >     > > > Michael.Pearce@ig.com
> >     > > > > >     > > >     > >
> >     > > > > >     > > >     >     wrote:
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     > Hi All,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > We have been discussing in the below
> > thread
> >     > and
> >     > > > > final
> >     > > > > >     > changes
> >     > > > > >     > > > have
> >     > > > > >     > > >     > been
> >     > > > > >     > > >     >     > made to the KIP wiki based on these
> >     > > discussions.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > We would now like to put to the vote
> > the
> >     > > > following
> >     > > > > > KIP:
> >     > > > > >     > > >     >     > https://cwiki.apache.org/
> >     > > > > > confluence/display/KAFKA/KIP-
> >     > > > > >     > 87+-+
> >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > This kip is for having a distinct
> > compaction
> >     > > > > > attribute
> >     > > > > >     > > > "tombstone"
> >     > > > > >     > > >     > flag
> >     > > > > >     > > >     >     > instead of relying on null value,
> > allowing
> >     > > > non-null
> >     > > > > > value
> >     > > > > >     > > > delete
> >     > > > > >     > > >     > messages.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > Many thanks,
> >     > > > > >     > > >     >     > Michael
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
> Pearce"
> > <
> >     > > > > >     > > Michael.Pearce@ig.com>
> >     > > > > >     > > >     > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Hi Mayuresh,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     LGTM. Ive just made one small
> > adjustment
> >     > > > > > updating the
> >     > > > > >     > > wire
> >     > > > > >     > > >     > protocol to
> >     > > > > >     > > >     >     > show the magic byte bump.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Do we think we're good to put to
> a
> > vote?
> >     > Is
> >     > > > > > there any
> >     > > > > >     > > > other bits
> >     > > > > >     > > >     >     > needing discussion?
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Cheers
> >     > > > > >     > > >     >     >     Mike
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
> > Gharat" <
> >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
> >     > > > > >     > > >     >     > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Hi Michael,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         I have updated the migration
> > section
> >     > of
> >     > > > the
> >     > > > > > KIP.
> >     > > > > >     > Can
> >     > > > > >     > > > you
> >     > > > > >     > > >     > please
> >     > > > > >     > > >     >     > take a look?
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Thanks,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Mayuresh
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07
> > AM,
> >     > > Mayuresh
> >     > > > > > Gharat <
> >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         > Hi Michael,
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > That whilst sending
> > tombstone and
> >     > non
> >     > > > > null
> >     > > > > > value,
> >     > > > > >     > > the
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     > can expect
> >     > > > > >     > > >     >     >         > only to receive the
> non-null
> >     > message
> >     > > > only
> >     > > > > > in step
> >     > > > > >     > > > (3) is
> >     > > > > >     > > >     > this
> >     > > > > >     > > >     >     > correct?
> >     > > > > >     > > >     >     >         > ---> I do agree with you
> > here.
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Becket, Ismael : can you
> guys
> >     > review
> >     > > > the
> >     > > > > >     > migration
> >     > > > > >     > > > plan
> >     > > > > >     > > >     > listed
> >     > > > > >     > > >     >     > above using
> >     > > > > >     > > >     >     >         > magic byte?
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Thanks,
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Mayuresh
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at
> 8:58
> > AM,
> >     > > > Michael
> >     > > > > > Pearce <
> >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> >     > > > > >     > > >     >     >         > wrote:
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >> Many thanks for this
> > Mayuresh. I
> >     > > don't
> >     > > > > > have any
> >     > > > > >     > > >     > objections.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> I assume we should state:
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> That whilst sending
> > tombstone and
> >     > > non
> >     > > > > null
> >     > > > > >     > value,
> >     > > > > >     > > > the
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     > can expect
> >     > > > > >     > > >     >     >         >> only to receive the
> non-null
> >     > message
> >     > > > > only
> >     > > > > > in
> >     > > > > >     > step
> >     > > > > >     > > > (3) is
> >     > > > > >     > > >     > this
> >     > > > > >     > > >     >     > correct?
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Cheers
> >     > > > > >     > > >     >     >         >> Mike
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Sent using OWA for iPhone
> >     > > > > >     > > >     >     >         >>
> > ______________________________
> >     > > > > __________
> >     > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
> >     > > > > >     > gharatmayuresh15@gmail.com
> >     > > > > >     > > >
> >     > > > > >     > > >     >     >         >> Sent: Thursday, November
> > 17, 2016
> >     > > > > 5:18:41
> >     > > > > > PM
> >     > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
> >     > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> > KIP-87 -
> >     > Add
> >     > > > > > Compaction
> >     > > > > >     > > > Tombstone
> >     > > > > >     > > >     > Flag
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Hi Ismael,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Thanks for the
> explanation.
> >     > > > > >     > > >     >     >         >> Specially I like this part
> > where
> >     > in
> >     > > > you
> >     > > > > >     > mentioned
> >     > > > > >     > > > we can
> >     > > > > >     > > >     > get
> >     > > > > >     > > >     >     > rid of the
> >     > > > > >     > > >     >     >         >> older null value support
> > for log
> >     > > > > > compaction
> >     > > > > >     > later
> >     > > > > >     > > > on,
> >     > > > > >     > > >     > here :
> >     > > > > >     > > >     >     >         >> We can't change semantics
> > of the
> >     > > > message
> >     > > > > > format
> >     > > > > >     > > > without
> >     > > > > >     > > >     > having
> >     > > > > >     > > >     >     > a long
> >     > > > > >     > > >     >     >         >> transition period. And we
> > can't
> >     > rely
> >     > > > > >     > > >     >     >         >> on people reading
> > documentation or
> >     > > > > acting
> >     > > > > > on a
> >     > > > > >     > > > warning for
> >     > > > > >     > > >     >     > something so
> >     > > > > >     > > >     >     >         >> fundamental. As such, my
> > take is
> >     > > that
> >     > > > we
> >     > > > > > need to
> >     > > > > >     > > > bump the
> >     > > > > >     > > >     > magic
> >     > > > > >     > > >     >     > byte. The
> >     > > > > >     > > >     >     >         >> good news is
> >     > > > > >     > > >     >     >         >> that we don't have to
> > support all
> >     > > > > versions
> >     > > > > >     > > forever.
> >     > > > > >     > > > We
> >     > > > > >     > > >     > have
> >     > > > > >     > > >     >     > said that we
> >     > > > > >     > > >     >     >         >> will support direct
> > upgrades for 2
> >     > > > > years.
> >     > > > > > That
> >     > > > > >     > > > means that
> >     > > > > >     > > >     >     > message format
> >     > > > > >     > > >     >     >         >> version n could, in
> theory,
> > be
> >     > > > removed 2
> >     > > > > > years
> >     > > > > >     > > > after the
> >     > > > > >     > > >     > it's
> >     > > > > >     > > >     >     > introduced.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Just a heads up, I would
> > like to
> >     > > > mention
> >     > > > > > that
> >     > > > > >     > even
> >     > > > > >     > > > without
> >     > > > > >     > > >     >     > bumping magic
> >     > > > > >     > > >     >     >         >> byte, we will *NOT* loose
> > zero
> >     > copy
> >     > > as
> >     > > > > in
> >     > > > > > the
> >     > > > > >     > > > client(x+1)
> >     > > > > >     > > >     > in my
> >     > > > > >     > > >     >     >         >> explanation
> >     > > > > >     > > >     >     >         >> above will convert
> > internally a
> >     > null
> >     > > > > > value to
> >     > > > > >     > > have a
> >     > > > > >     > > >     > tombstone
> >     > > > > >     > > >     >     > bit set and
> >     > > > > >     > > >     >     >         >> a tombstone bit set to
> have
> > a null
> >     > > > value
> >     > > > > >     > > > automatically
> >     > > > > >     > > >     >     > internally and by
> >     > > > > >     > > >     >     >         >> the time we move to
> version
> > (x+2),
> >     > > the
> >     > > > > > clients
> >     > > > > >     > > > would have
> >     > > > > >     > > >     >     > upgraded.
> >     > > > > >     > > >     >     >         >> Obviously if we support a
> > request
> >     > > from
> >     > > > > >     > > consumer(x),
> >     > > > > >     > > > we
> >     > > > > >     > > >     > will
> >     > > > > >     > > >     >     > loose zero
> >     > > > > >     > > >     >     >         >> copy
> >     > > > > >     > > >     >     >         >> but that is the same case
> > with
> >     > magic
> >     > > > > byte.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> But if magic byte bump
> > makes life
> >     > > > easier
> >     > > > > > for
> >     > > > > >     > > > transition
> >     > > > > >     > > >     > for the
> >     > > > > >     > > >     >     > above
> >     > > > > >     > > >     >     >         >> reasons that you
> explained,
> > I am
> >     > OK
> >     > > > with
> >     > > > > > it
> >     > > > > >     > since
> >     > > > > >     > > > we are
> >     > > > > >     > > >     > going
> >     > > > > >     > > >     >     > to meet the
> >     > > > > >     > > >     >     >         >> end goal down the road :)
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> On a side note can we
> > update the
> >     > doc
> >     > > > > here
> >     > > > > > on
> >     > > > > >     > magic
> >     > > > > >     > > > byte
> >     > > > > >     > > >     > to say
> >     > > > > >     > > >     >     > that "*it
> >     > > > > >     > > >     >     >         >> should be bumped whenever
> > the
> >     > > message
> >     > > > > > format is
> >     > > > > >     > > > changed
> >     > > > > >     > > >     > or the
> >     > > > > >     > > >     >     >         >> interpretation of message
> > format
> >     > > > (usage
> >     > > > > > of the
> >     > > > > >     > > > reserved
> >     > > > > >     > > >     > bits as
> >     > > > > >     > > >     >     > well) is
> >     > > > > >     > > >     >     >         >> changed*".
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Hi Michael,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Here is the update plan
> > that we
> >     > > > > discussed
> >     > > > > >     > offline
> >     > > > > >     > > >     > yesterday :
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Currently the magic-byte
> > which
> >     > > > > > corresponds to
> >     > > > > >     > the
> >     > > > > >     > > >     >     > "message.format.version"
> >     > > > > >     > > >     >     >         >> is set to 1.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 1) On broker it will be
> set
> > to 1
> >     > > > > > initially.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 2) When a producer client
> > sends a
> >     > > > > message
> >     > > > > > with
> >     > > > > >     > > > magic-byte
> >     > > > > >     > > >     > = 2,
> >     > > > > >     > > >     >     > since the
> >     > > > > >     > > >     >     >         >> broker is on magic-byte =
> > 1, we
> >     > will
> >     > > > > down
> >     > > > > >     > convert
> >     > > > > >     > > > it,
> >     > > > > >     > > >     > which
> >     > > > > >     > > >     >     > means if the
> >     > > > > >     > > >     >     >         >> tombstone bit is set, the
> > value
> >     > will
> >     > > > be
> >     > > > > > set to
> >     > > > > >     > > > null. A
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     >         >> understanding magic-byte =
> > 1, will
> >     > > > still
> >     > > > > > work
> >     > > > > >     > with
> >     > > > > >     > > > this. A
> >     > > > > >     > > >     >     > consumer
> >     > > > > >     > > >     >     >         >> working
> >     > > > > >     > > >     >     >         >> with magic-byte =2 will
> > also be
> >     > able
> >     > > > to
> >     > > > > >     > understand
> >     > > > > >     > > > this,
> >     > > > > >     > > >     > since
> >     > > > > >     > > >     >     > it
> >     > > > > >     > > >     >     >         >> understands the tombstone.
> >     > > > > >     > > >     >     >         >> Now there is still the
> > question of
> >     > > > > > supporting a
> >     > > > > >     > > >     > non-tombstone
> >     > > > > >     > > >     >     > and null
> >     > > > > >     > > >     >     >         >> value from producer client
> > with
> >     > > > > > magic-byte = 2.*
> >     > > > > >     > > (I
> >     > > > > >     > > > am
> >     > > > > >     > > >     > not sure
> >     > > > > >     > > >     >     > if we
> >     > > > > >     > > >     >     >         >> should support this.
> > Ismael/Becket
> >     > > can
> >     > > > > > comment
> >     > > > > >     > > > here)*
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 3) When almost all the
> > clients
> >     > have
> >     > > > > > upgraded,
> >     > > > > >     > the
> >     > > > > >     > > >     >     > message.format.version
> >     > > > > >     > > >     >     >         >> on
> >     > > > > >     > > >     >     >         >> the broker can be changed
> > to 2,
> >     > > where
> >     > > > in
> >     > > > > > the
> >     > > > > >     > down
> >     > > > > >     > > >     > conversion in
> >     > > > > >     > > >     >     > the above
> >     > > > > >     > > >     >     >         >> step will not happen. If
> at
> > this
> >     > > point
> >     > > > > we
> >     > > > > > get a
> >     > > > > >     > > > consumer
> >     > > > > >     > > >     >     > request from a
> >     > > > > >     > > >     >     >         >> older consumer, we might
> > have to
> >     > > down
> >     > > > > > convert
> >     > > > > >     > > where
> >     > > > > >     > > > in we
> >     > > > > >     > > >     > loose
> >     > > > > >     > > >     >     > zero copy,
> >     > > > > >     > > >     >     >         >> but these cases should be
> > rare.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Becket can you review this
> > plan
> >     > and
> >     > > > add
> >     > > > > > more
> >     > > > > >     > > > details if I
> >     > > > > >     > > >     > have
> >     > > > > >     > > >     >     >         >> missed/wronged something,
> > before
> >     > we
> >     > > > put
> >     > > > > > it on
> >     > > > > >     > KIP.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Thanks,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Mayuresh
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> > 11:07 PM,
> >     > > > > Michael
> >     > > > > >     > Pearce <
> >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> >     > > > > >     > > >     >     >         >> wrote:
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> > Thanks guys, for
> > discussing this
> >     > > > > > offline and
> >     > > > > >     > > > getting
> >     > > > > >     > > >     > some
> >     > > > > >     > > >     >     > consensus.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > So its clear for myself
> > and
> >     > others
> >     > > > > what
> >     > > > > > is
> >     > > > > >     > > > proposed now
> >     > > > > >     > > >     > (i
> >     > > > > >     > > >     >     > think i
> >     > > > > >     > > >     >     >         >> > understand, but want to
> > make
> >     > sure)
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Could i ask either
> > directly
> >     > update
> >     > > > the
> >     > > > > > kip to
> >     > > > > >     > > > detail the
> >     > > > > >     > > >     >     > migration
> >     > > > > >     > > >     >     >         >> > strategy, or (re-)state
> > your
> >     > > offline
> >     > > > > > discussed
> >     > > > > >     > > and
> >     > > > > >     > > >     > agreed
> >     > > > > >     > > >     >     > migration
> >     > > > > >     > > >     >     >         >> > strategy based on a
> magic
> > byte
> >     > is
> >     > > in
> >     > > > > > this
> >     > > > > >     > > thread.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > The main original driver
> > for the
> >     > > KIP
> >     > > > > > was to
> >     > > > > >     > > > support
> >     > > > > >     > > >     >     > compaction where
> >     > > > > >     > > >     >     >         >> value
> >     > > > > >     > > >     >     >         >> > isn't null, based off
> the
> >     > > > discussions
> >     > > > > on
> >     > > > > >     > KIP-82
> >     > > > > >     > > > thread.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > We should be able to
> > support
> >     > > > > > non-tombstone +
> >     > > > > >     > > null
> >     > > > > >     > > > value
> >     > > > > >     > > >     > by the
> >     > > > > >     > > >     >     >         >> completion
> >     > > > > >     > > >     >     >         >> > of the KIP, as we noted
> > when
> >     > > > > discussing
> >     > > > > > this
> >     > > > > >     > > kip,
> >     > > > > >     > > > having
> >     > > > > >     > > >     >     > logic based on
> >     > > > > >     > > >     >     >         >> a
> >     > > > > >     > > >     >     >         >> > null value isn't very
> > clean and
> >     > > also
> >     > > > > > separates
> >     > > > > >     > > the
> >     > > > > >     > > >     > concerns.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > As discussed already
> > though we
> >     > can
> >     > > > > > split this
> >     > > > > >     > > into
> >     > > > > >     > > >     > KIP-87a
> >     > > > > >     > > >     >     > and KIP-87b
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Where we look to deliver
> > KIP-87a
> >     > > on
> >     > > > a
> >     > > > > >     > compacted
> >     > > > > >     > > > topic
> >     > > > > >     > > >     > (to
> >     > > > > >     > > >     >     > address the
> >     > > > > >     > > >     >     >         >> > immediate issues)
> >     > > > > >     > > >     >     >         >> > * tombstone + null value
> >     > > > > >     > > >     >     >         >> > * tombstone + non-null
> > value
> >     > > > > >     > > >     >     >         >> > * non-tombstone +
> > non-null value
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Then we can discuss once
> > KIP-87a
> >     > > is
> >     > > > > > completed
> >     > > > > >     > > > options
> >     > > > > >     > > >     > later
> >     > > > > >     > > >     >     > and how we
> >     > > > > >     > > >     >     >         >> > support the second part
> > KIP-87b
> >     > to
> >     > > > > > deliver:
> >     > > > > >     > > >     >     >         >> > * non-tombstone + null
> > value
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Cheers
> >     > > > > >     > > >     >     >         >> > Mike
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> > ______________________________
> >     > > > > > __________
> >     > > > > >     > > >     >     >         >> > From: Becket Qin <
> >     > > > > becket.qin@gmail.com>
> >     > > > > >     > > >     >     >         >> > Sent: Thursday, November
> > 17,
> >     > 2016
> >     > > > 1:43
> >     > > > > > AM
> >     > > > > >     > > >     >     >         >> > To:
> dev@kafka.apache.org
> >     > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> > KIP-87 -
> >     > > Add
> >     > > > > > Compaction
> >     > > > > >     > > >     > Tombstone Flag
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had
> > an
> >     > > offline
> >     > > > > >     > discussion,
> >     > > > > >     > > > and
> >     > > > > >     > > >     > following
> >     > > > > >     > > >     >     > is a brief
> >     > > > > >     > > >     >     >         >> > summary.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > 1. We agreed that not
> > bumping up
> >     > > > magic
> >     > > > > > value
> >     > > > > >     > may
> >     > > > > >     > > > result
> >     > > > > >     > > >     > in
> >     > > > > >     > > >     >     > losing zero
> >     > > > > >     > > >     >     >         >> copy
> >     > > > > >     > > >     >     >         >> > during migration.
> >     > > > > >     > > >     >     >         >> > 2. Given that bumping up
> > magic
> >     > > value
> >     > > > > is
> >     > > > > > almost
> >     > > > > >     > > > free and
> >     > > > > >     > > >     > has
> >     > > > > >     > > >     >     > benefit of
> >     > > > > >     > > >     >     >         >> > avoiding potential
> > performance
> >     > > > issue.
> >     > > > > > It is
> >     > > > > >     > > > probably
> >     > > > > >     > > >     > worth
> >     > > > > >     > > >     >     > doing.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > One issue we still need
> > to think
> >     > > > about
> >     > > > > > is
> >     > > > > >     > > whether
> >     > > > > >     > > > we
> >     > > > > >     > > >     > want to
> >     > > > > >     > > >     >     > support a
> >     > > > > >     > > >     >     >         >> > non-tombstone message
> > with null
> >     > > > value.
> >     > > > > >     > > >     >     >         >> > Currently it is not
> > supported by
> >     > > > > Kafka.
> >     > > > > > If we
> >     > > > > >     > > > allow a
> >     > > > > >     > > >     >     > non-tombstone null
> >     > > > > >     > > >     >     >         >> > value message to exist
> > after
> >     > > KIP-87.
> >     > > > > The
> >     > > > > >     > problem
> >     > > > > >     > > > is
> >     > > > > >     > > >     > that such
> >     > > > > >     > > >     >     > message
> >     > > > > >     > > >     >     >         >> will
> >     > > > > >     > > >     >     >         >> > not be supported by the
> >     > consumers
> >     > > > > prior
> >     > > > > > to
> >     > > > > >     > > KIP-87.
> >     > > > > >     > > >     > Because a
> >     > > > > >     > > >     >     > null value
> >     > > > > >     > > >     >     >         >> > will always be
> > interpreted to a
> >     > > > > > tombstone.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > One option is that we
> > keep the
> >     > > > current
> >     > > > > > way,
> >     > > > > >     > i.e.
> >     > > > > >     > > > do not
> >     > > > > >     > > >     >     > support such
> >     > > > > >     > > >     >     >         >> > message. It would be
> good
> > to
> >     > know
> >     > > if
> >     > > > > > there is
> >     > > > > >     > a
> >     > > > > >     > > >     > concrete use
> >     > > > > >     > > >     >     > case for
> >     > > > > >     > > >     >     >         >> such
> >     > > > > >     > > >     >     >         >> > message. If there is
> not,
> > we can
> >     > > > > > probably just
> >     > > > > >     > > not
> >     > > > > >     > > >     > support it.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Thanks,
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at
> > 1:28 PM,
> >     > > > > > Mayuresh
> >     > > > > >     > > Gharat <
> >     > > > > >     > > >     >     >         >> >
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > wrote:
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > > Hi Ismael,
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > This is something I
> can
> > think
> >     > of
> >     > > > for
> >     > > > > >     > migration
> >     > > > > >     > > > plan:
> >     > > > > >     > > >     >     >         >> > > So the migration plan
> > can look
> >     > > > > > something
> >     > > > > >     > like
> >     > > > > >     > > > this,
> >     > > > > >     > > >     > with up
> >     > > > > >     > > >     >     >         >> conversion :
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > 1) Currently lets say
> > we have
> >     > > > Broker
> >     > > > > > at
> >     > > > > >     > > version
> >     > > > > >     > > > x.
> >     > > > > >     > > >     >     >         >> > > 2) Currently we have
> > clients
> >     > at
> >     > > > > > version x.
> >     > > > > >     > > >     >     >         >> > > 3) a) We move the
> > version to
> >     > > > > > Broker(x+1) :
> >     > > > > >     > > > supports
> >     > > > > >     > > >     > both
> >     > > > > >     > > >     >     > tombstone and
> >     > > > > >     > > >     >     >         >> > null
> >     > > > > >     > > >     >     >         >> > > for log compaction.
> >     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> > client
> >     > to
> >     > > > > > version
> >     > > > > >     > > > client(x+1) :
> >     > > > > >     > > >     > if in
> >     > > > > >     > > >     >     > the
> >     > > > > >     > > >     >     >         >> producer
> >     > > > > >     > > >     >     >         >> > > client(x+1) the value
> > is set
> >     > to
> >     > > > > null,
> >     > > > > > we
> >     > > > > >     > will
> >     > > > > >     > > >     > automatically
> >     > > > > >     > > >     >     > set the
> >     > > > > >     > > >     >     >         >> > > Tombstone bit
> > internally. If
> >     > the
> >     > > > > > producer
> >     > > > > >     > > > client(x+1)
> >     > > > > >     > > >     > sets
> >     > > > > >     > > >     >     > the
> >     > > > > >     > > >     >     >         >> tombstone
> >     > > > > >     > > >     >     >         >> > > itself, well and good.
> > For
> >     > > > producer
> >     > > > > >     > client(x),
> >     > > > > >     > > > the
> >     > > > > >     > > >     > broker
> >     > > > > >     > > >     >     > will up
> >     > > > > >     > > >     >     >         >> convert
> >     > > > > >     > > >     >     >         >> > > to have the tombstone
> > bit.
> >     > > > > > Broker(x+1) is
> >     > > > > >     > > > supporting
> >     > > > > >     > > >     > both.
> >     > > > > >     > > >     >     > Consumer
> >     > > > > >     > > >     >     >         >> > > client(x+1) will be
> > aware of
> >     > > this
> >     > > > > and
> >     > > > > > should
> >     > > > > >     > > be
> >     > > > > >     > > > able
> >     > > > > >     > > >     > to
> >     > > > > >     > > >     >     > handle this.
> >     > > > > >     > > >     >     >         >> For
> >     > > > > >     > > >     >     >         >> > > consumer client(x) we
> > will
> >     > down
> >     > > > > > convert the
> >     > > > > >     > > > message
> >     > > > > >     > > >     > on the
> >     > > > > >     > > >     >     > broker
> >     > > > > >     > > >     >     >         >> side.
> >     > > > > >     > > >     >     >         >> > >     c) At this point
> we
> > will
> >     > > have
> >     > > > to
> >     > > > > >     > specify a
> >     > > > > >     > > >     > warning or
> >     > > > > >     > > >     >     > clearly
> >     > > > > >     > > >     >     >         >> specify
> >     > > > > >     > > >     >     >         >> > > in docs that this
> > behavior is
> >     > > > about
> >     > > > > > to be
> >     > > > > >     > > > changed for
> >     > > > > >     > > >     > log
> >     > > > > >     > > >     >     > compaction.
> >     > > > > >     > > >     >     >         >> > > 4) a) In next release
> > of the
> >     > > > > > Broker(x+2), we
> >     > > > > >     > > > say that
> >     > > > > >     > > >     > only
> >     > > > > >     > > >     >     > Tombstone
> >     > > > > >     > > >     >     >         >> is
> >     > > > > >     > > >     >     >         >> > > used for log
> compaction
> > on the
> >     > > > > Broker
> >     > > > > > side.
> >     > > > > >     > > >     > Clients(x+1)
> >     > > > > >     > > >     >     > still is
> >     > > > > >     > > >     >     >         >> > > supported.
> >     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> > client
> >     > to
> >     > > > > > version
> >     > > > > >     > > > client(x+2) :
> >     > > > > >     > > >     > if
> >     > > > > >     > > >     >     > value is set
> >     > > > > >     > > >     >     >         >> to
> >     > > > > >     > > >     >     >         >> > > null, tombstone will
> > not be
> >     > set
> >     > > > > >     > automatically.
> >     > > > > >     > > > The
> >     > > > > >     > > >     > client
> >     > > > > >     > > >     >     > will have to
> >     > > > > >     > > >     >     >         >> > call
> >     > > > > >     > > >     >     >         >> > > setTombstone() to
> > actually set
> >     > > the
> >     > > > > >     > tombstone.
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > We should compare this
> >     > migration
> >     > > > > plan
> >     > > > > > with
> >     > > > > >     > the
> >     > > > > >     > > >     > migration
> >     > > > > >     > > >     >     > plan for
> >     > > > > >     > > >     >     >         >> magic
> >     > > > > >     > > >     >     >         >> > > byte bump and do
> > whatever
> >     > looks
> >     > > > > good.
> >     > > > > >     > > >     >     >         >> > > I am just worried that
> > if we
> >     > go
> >     > > > down
> >     > > > > > magic
> >     > > > > >     > > byte
> >     > > > > >     > > > route,
> >     > > > > >     > > >     >     > unless I am
> >     > > > > >     > > >     >     >         >> > missing
> >     > > > > >     > > >     >     >         >> > > something, it sounds
> > like
> >     > kafka
> >     > > > will
> >     > > > > > be
> >     > > > > >     > stuck
> >     > > > > >     > > > with
> >     > > > > >     > > >     >     > supporting both
> >     > > > > >     > > >     >     >         >> null
> >     > > > > >     > > >     >     >         >> > > value and tombstone
> bit
> > for
> >     > log
> >     > > > > > compaction
> >     > > > > >     > for
> >     > > > > >     > > > life
> >     > > > > >     > > >     > long,
> >     > > > > >     > > >     >     > which does
> >     > > > > >     > > >     >     >         >> not
> >     > > > > >     > > >     >     >         >> > > look like a good end
> > state.
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > Thanks,
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > Mayuresh
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016
> at
> > 9:32
> >     > AM,
> >     > > > > > Mayuresh
> >     > > > > >     > > > Gharat <
> >     > > > > >     > > >     >     >         >> > >
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > > wrote:
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > That's a very good
> > point
> >     > > which I
> >     > > > > > might
> >     > > > > >     > have
> >     > > > > >     > > > not
> >     > > > > >     > > >     >     > considered earlier.
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Here is a plan that
> I
> > can
> >     > > think
> >     > > > > of:
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Stage 1) The broker
> > from now
> >     > > on,
> >     > > > > up
> >     > > > > >     > converts
> >     > > > > >     > > > the
> >     > > > > >     > > >     > message
> >     > > > > >     > > >     >     > to have the
> >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> The
> > log
> >     > > > > compaction
> >     > > > > >     > thread
> >     > > > > >     > > > does log
> >     > > > > >     > > >     >     > compaction
> >     > > > > >     > > >     >     >         >> based
> >     > > > > >     > > >     >     >         >> > on
> >     > > > > >     > > >     >     >         >> > > > both null and
> > tombstone
> >     > > marker.
> >     > > > > > This is
> >     > > > > >     > our
> >     > > > > >     > > >     > transition
> >     > > > > >     > > >     >     > period.
> >     > > > > >     > > >     >     >         >> > > > Stage 2) The next
> > release we
> >     > > > only
> >     > > > > > say that
> >     > > > > >     > > log
> >     > > > > >     > > >     > compaction
> >     > > > > >     > > >     >     > is based
> >     > > > > >     > > >     >     >         >> on
> >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> > (Open
> >     > source
> >     > > > > > kafka makes
> >     > > > > >     > > > this as a
> >     > > > > >     > > >     >     > policy). By
> >     > > > > >     > > >     >     >         >> this
> >     > > > > >     > > >     >     >         >> > > time,
> >     > > > > >     > > >     >     >         >> > > > the organization
> > which is
> >     > > moving
> >     > > > > to
> >     > > > > > this
> >     > > > > >     > > > release
> >     > > > > >     > > >     > will be
> >     > > > > >     > > >     >     > sure that
> >     > > > > >     > > >     >     >         >> they
> >     > > > > >     > > >     >     >         >> > > > have gone through
> the
> > entire
> >     > > > > > transition
> >     > > > > >     > > > period.
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > My only goal of
> doing
> > this
> >     > is
> >     > > > that
> >     > > > > > Kafka
> >     > > > > >     > > > clearly
> >     > > > > >     > > >     >     > specifies the end
> >     > > > > >     > > >     >     >         >> > state
> >     > > > > >     > > >     >     >         >> > > > about what log
> > compaction
> >     > > means
> >     > > > > (is
> >     > > > > > it
> >     > > > > >     > null
> >     > > > > >     > > > value
> >     > > > > >     > > >     > or a
> >     > > > > >     > > >     >     > tombstone
> >     > > > > >     > > >     >     >         >> > marker,
> >     > > > > >     > > >     >     >         >> > > > but not both).
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > What do you think?
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Thanks,
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Mayuresh
> >     > > > > >     > > >     >     >         >> > > > .
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016
> > at 9:17
> >     > > AM,
> >     > > > > > Ismael
> >     > > > > >     > > Juma <
> >     > > > > >     > > >     >     > ismael@juma.me.uk>
> >     > > > > >     > > >     >     >         >> > wrote:
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >> One comment below.
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16,
> 2016
> > at
> >     > 5:08
> >     > > > PM,
> >     > > > > > Mayuresh
> >     > > > > >     > > > Gharat <
> >     > > > > >     > > >     >     >         >> > > >>
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > >> > wrote:
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> >    - If we don't
> > bump up
> >     > > the
> >     > > > > > magic
> >     > > > > >     > byte,
> >     > > > > >     > > > on the
> >     > > > > >     > > >     > broker
> >     > > > > >     > > >     >     > side, the
> >     > > > > >     > > >     >     >         >> > > broker
> >     > > > > >     > > >     >     >         >> > > >> >    will always
> > have to
> >     > look
> >     > > > at
> >     > > > > > both
> >     > > > > >     > > > tombstone
> >     > > > > >     > > >     > bit and
> >     > > > > >     > > >     >     > the value
> >     > > > > >     > > >     >     >         >> when
> >     > > > > >     > > >     >     >         >> > > do
> >     > > > > >     > > >     >     >         >> > > >> the
> >     > > > > >     > > >     >     >         >> > > >> >    compaction.
> > Assuming
> >     > we
> >     > > do
> >     > > > > > not bump
> >     > > > > >     > up
> >     > > > > >     > > > the
> >     > > > > >     > > >     > magic
> >     > > > > >     > > >     >     > byte,
> >     > > > > >     > > >     >     >         >> > > >> >    imagine the
> > broker
> >     > sees
> >     > > a
> >     > > > > > message
> >     > > > > >     > > which
> >     > > > > >     > > > does
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > have a
> >     > > > > >     > > >     >     >         >> tombstone
> >     > > > > >     > > >     >     >         >> > > bit
> >     > > > > >     > > >     >     >         >> > > >> >    set. The
> broker
> > does
> >     > not
> >     > > > > know
> >     > > > > > when
> >     > > > > >     > the
> >     > > > > >     > > >     > message was
> >     > > > > >     > > >     >     > produced
> >     > > > > >     > > >     >     >         >> (i.e.
> >     > > > > >     > > >     >     >         >> > > >> > whether
> >     > > > > >     > > >     >     >         >> > > >> >    the message
> has
> > been
> >     > up
> >     > > > > > converted or
> >     > > > > >     > > > not), it
> >     > > > > >     > > >     > has
> >     > > > > >     > > >     >     > to take a
> >     > > > > >     > > >     >     >         >> > further
> >     > > > > >     > > >     >     >         >> > > >> > look at
> >     > > > > >     > > >     >     >         >> > > >> >    the value to
> > see if it
> >     > > is
> >     > > > > > null or
> >     > > > > >     > not
> >     > > > > >     > > in
> >     > > > > >     > > >     > order to
> >     > > > > >     > > >     >     > determine
> >     > > > > >     > > >     >     >         >> if it
> >     > > > > >     > > >     >     >         >> > > is
> >     > > > > >     > > >     >     >         >> > > >> a
> >     > > > > >     > > >     >     >         >> > > >> >    tombstone. The
> > same
> >     > > logic
> >     > > > > has
> >     > > > > > to be
> >     > > > > >     > > put
> >     > > > > >     > > > on the
> >     > > > > >     > > >     >     > consumer as
> >     > > > > >     > > >     >     >         >> well
> >     > > > > >     > > >     >     >         >> > > >> because
> >     > > > > >     > > >     >     >         >> > > >> > the
> >     > > > > >     > > >     >     >         >> > > >> >    consumer does
> > not know
> >     > > if
> >     > > > > the
> >     > > > > >     > message
> >     > > > > >     > > > has
> >     > > > > >     > > >     > been up
> >     > > > > >     > > >     >     > converted or
> >     > > > > >     > > >     >     >         >> > not.
> >     > > > > >     > > >     >     >         >> > > >> >       - If we
> > upconvert
> >     > > while
> >     > > > > >     > appending,
> >     > > > > >     > > > this is
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > the case,
> >     > > > > >     > > >     >     >         >> > right?
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> If I understand you
> >     > > correctly,
> >     > > > > > this is
> >     > > > > >     > not
> >     > > > > >     > > >     > sufficient
> >     > > > > >     > > >     >     > because the
> >     > > > > >     > > >     >     >         >> log
> >     > > > > >     > > >     >     >         >> > > may
> >     > > > > >     > > >     >     >         >> > > >> have messages
> > appended
> >     > before
> >     > > > it
> >     > > > > > was
> >     > > > > >     > > > upgraded to
> >     > > > > >     > > >     > include
> >     > > > > >     > > >     >     > KIP-87.
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> Ismael
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > --
> >     > > > > >     > > >     >     >         >> > > > -Regards,
> >     > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > --
> >     > > > > >     > > >     >     >         >> > > -Regards,
> >     > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> > > (862) 250-7125
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > The information
> contained
> > in
> >     > this
> >     > > > > email
> >     > > > > > is
> >     > > > > >     > > > strictly
> >     > > > > >     > > >     >     > confidential and for
> >     > > > > >     > > >     >     >         >> > the use of the addressee
> > only,
> >     > > > unless
> >     > > > > >     > otherwise
> >     > > > > >     > > >     > indicated. If
> >     > > > > >     > > >     >     > you are
> >     > > > > >     > > >     >     >         >> not
> >     > > > > >     > > >     >     >         >> > the intended recipient,
> > please
> >     > do
> >     > > > not
> >     > > > > > read,
> >     > > > > >     > > copy,
> >     > > > > >     > > > use or
> >     > > > > >     > > >     >     > disclose to
> >     > > > > >     > > >     >     >         >> others
> >     > > > > >     > > >     >     >         >> > this message or any
> > attachment.
> >     > > > Please
> >     > > > > > also
> >     > > > > >     > > > notify the
> >     > > > > >     > > >     > sender
> >     > > > > >     > > >     >     > by
> >     > > > > >     > > >     >     >         >> replying
> >     > > > > >     > > >     >     >         >> > to this email or by
> > telephone
> >     > > > (+44(020
> >     > > > > > 7896
> >     > > > > >     > > 0011)
> >     > > > > >     > > > and
> >     > > > > >     > > >     > then
> >     > > > > >     > > >     >     > delete the
> >     > > > > >     > > >     >     >         >> email
> >     > > > > >     > > >     >     >         >> > and any copies of it.
> > Opinions,
> >     > > > > > conclusion
> >     > > > > >     > (etc)
> >     > > > > >     > > > that
> >     > > > > >     > > >     > do not
> >     > > > > >     > > >     >     > relate to
> >     > > > > >     > > >     >     >         >> the
> >     > > > > >     > > >     >     >         >> > official business of
> this
> >     > company
> >     > > > > shall
> >     > > > > > be
> >     > > > > >     > > > understood as
> >     > > > > >     > > >     >     > neither given
> >     > > > > >     > > >     >     >         >> nor
> >     > > > > >     > > >     >     >         >> > endorsed by it. IG is a
> > trading
> >     > > name
> >     > > > > of
> >     > > > > > IG
> >     > > > > >     > > Markets
> >     > > > > >     > > >     > Limited (a
> >     > > > > >     > > >     >     > company
> >     > > > > >     > > >     >     >         >> > registered in England
> and
> > Wales,
> >     > > > > company
> >     > > > > >     > number
> >     > > > > >     > > >     > 04008957) and
> >     > > > > >     > > >     >     > IG Index
> >     > > > > >     > > >     >     >         >> > Limited (a company
> > registered in
> >     > > > > > England and
> >     > > > > >     > > > Wales,
> >     > > > > >     > > >     > company
> >     > > > > >     > > >     >     > number
> >     > > > > >     > > >     >     >         >> > 01190902). Registered
> > address at
> >     > > > > Cannon
> >     > > > > > Bridge
> >     > > > > >     > > > House, 25
> >     > > > > >     > > >     >     > Dowgate Hill,
> >     > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
> > Markets
> >     > > > > Limited
> >     > > > > >     > > (register
> >     > > > > >     > > >     > number
> >     > > > > >     > > >     >     > 195355) and IG
> >     > > > > >     > > >     >     >         >> > Index Limited (register
> > number
> >     > > > 114059)
> >     > > > > > are
> >     > > > > >     > > > authorised
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > regulated by
> >     > > > > >     > > >     >     >         >> the
> >     > > > > >     > > >     >     >         >> > Financial Conduct
> > Authority.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> --
> >     > > > > >     > > >     >     >         >> -Regards,
> >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> (862) 250-7125
> >     > > > > >     > > >     >     >         >> The information contained
> > in this
> >     > > > email
> >     > > > > is
> >     > > > > >     > > strictly
> >     > > > > >     > > >     >     > confidential and for
> >     > > > > >     > > >     >     >         >> the use of the addressee
> > only,
> >     > > unless
> >     > > > > > otherwise
> >     > > > > >     > > >     > indicated. If
> >     > > > > >     > > >     >     > you are not
> >     > > > > >     > > >     >     >         >> the intended recipient,
> > please do
> >     > > not
> >     > > > > > read,
> >     > > > > >     > copy,
> >     > > > > >     > > > use or
> >     > > > > >     > > >     >     > disclose to others
> >     > > > > >     > > >     >     >         >> this message or any
> > attachment.
> >     > > Please
> >     > > > > > also
> >     > > > > >     > notify
> >     > > > > >     > > > the
> >     > > > > >     > > >     > sender
> >     > > > > >     > > >     >     > by replying
> >     > > > > >     > > >     >     >         >> to this email or by
> > telephone
> >     > > (+44(020
> >     > > > > > 7896
> >     > > > > >     > 0011)
> >     > > > > >     > > > and then
> >     > > > > >     > > >     >     > delete the email
> >     > > > > >     > > >     >     >         >> and any copies of it.
> > Opinions,
> >     > > > > > conclusion (etc)
> >     > > > > >     > > > that do
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > relate to the
> >     > > > > >     > > >     >     >         >> official business of this
> > company
> >     > > > shall
> >     > > > > be
> >     > > > > >     > > > understood as
> >     > > > > >     > > >     >     > neither given nor
> >     > > > > >     > > >     >     >         >> endorsed by it. IG is a
> > trading
> >     > name
> >     > > > of
> >     > > > > IG
> >     > > > > >     > Markets
> >     > > > > >     > > >     > Limited (a
> >     > > > > >     > > >     >     > company
> >     > > > > >     > > >     >     >         >> registered in England and
> > Wales,
> >     > > > company
> >     > > > > > number
> >     > > > > >     > > > 04008957)
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > IG Index
> >     > > > > >     > > >     >     >         >> Limited (a company
> > registered in
> >     > > > England
> >     > > > > > and
> >     > > > > >     > > Wales,
> >     > > > > >     > > >     > company
> >     > > > > >     > > >     >     > number
> >     > > > > >     > > >     >     >         >> 01190902). Registered
> > address at
> >     > > > Cannon
> >     > > > > > Bridge
> >     > > > > >     > > > House, 25
> >     > > > > >     > > >     >     > Dowgate Hill,
> >     > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> > Markets
> >     > > > Limited
> >     > > > > >     > (register
> >     > > > > >     > > > number
> >     > > > > >     > > >     >     > 195355) and IG
> >     > > > > >     > > >     >     >         >> Index Limited (register
> > number
> >     > > 114059)
> >     > > > > are
> >     > > > > >     > > > authorised and
> >     > > > > >     > > >     >     > regulated by the
> >     > > > > >     > > >     >     >         >> Financial Conduct
> Authority.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > --
> >     > > > > >     > > >     >     >         > -Regards,
> >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         > (862) 250-7125
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         --
> >     > > > > >     > > >     >     >         -Regards,
> >     > > > > >     > > >     >     >         Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         (862) 250-7125
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     The information contained in this
> > email
> >     > is
> >     > > > > > strictly
> >     > > > > >     > > > confidential
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > for the use of the addressee only,
> > unless
> >     > > > otherwise
> >     > > > > >     > > indicated.
> >     > > > > >     > > > If
> >     > > > > >     > > >     > you are
> >     > > > > >     > > >     >     > not the intended recipient, please do
> > not
> >     > read,
> >     > > > > > copy, use
> >     > > > > >     > or
> >     > > > > >     > > >     > disclose to
> >     > > > > >     > > >     >     > others this message or any
> attachment.
> > Please
> >     > > > also
> >     > > > > > notify
> >     > > > > >     > the
> >     > > > > >     > > > sender
> >     > > > > >     > > >     > by
> >     > > > > >     > > >     >     > replying to this email or by
> telephone
> >     > (+44(020
> >     > > > > 7896
> >     > > > > > 0011)
> >     > > > > >     > > and
> >     > > > > >     > > > then
> >     > > > > >     > > >     > delete
> >     > > > > >     > > >     >     > the email and any copies of it.
> > Opinions,
> >     > > > > conclusion
> >     > > > > > (etc)
> >     > > > > >     > > > that do
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > relate to the official business of
> this
> >     > company
> >     > > > > > shall be
> >     > > > > >     > > > understood
> >     > > > > >     > > >     > as
> >     > > > > >     > > >     >     > neither given nor endorsed by it. IG
> > is a
> >     > > trading
> >     > > > > > name of
> >     > > > > >     > IG
> >     > > > > >     > > > Markets
> >     > > > > >     > > >     >     > Limited (a company registered in
> > England and
> >     > > > Wales,
> >     > > > > > company
> >     > > > > >     > > > number
> >     > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> > company
> >     > > > > registered
> >     > > > > > in
> >     > > > > >     > > > England and
> >     > > > > >     > > >     > Wales,
> >     > > > > >     > > >     >     > company number 01190902). Registered
> > address
> >     > at
> >     > > > > > Cannon
> >     > > > > >     > Bridge
> >     > > > > >     > > > House,
> >     > > > > >     > > >     > 25
> >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both
> IG
> >     > Markets
> >     > > > > > Limited
> >     > > > > >     > > > (register
> >     > > > > >     > > >     > number
> >     > > > > >     > > >     >     > 195355) and IG Index Limited
> (register
> > number
> >     > > > > > 114059) are
> >     > > > > >     > > > authorised
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > regulated by the Financial Conduct
> > Authority.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > The information contained in this email is
> > strictly
> >     > > > > > confidential
> >     > > > > >     > > and
> >     > > > > >     > > > for
> >     > > > > >     > > >     > the use of the addressee only, unless
> > otherwise
> >     > > > > indicated.
> >     > > > > > If you
> >     > > > > >     > > > are not
> >     > > > > >     > > >     > the intended recipient, please do not read,
> > copy,
> >     > use
> >     > > > or
> >     > > > > > disclose
> >     > > > > >     > > to
> >     > > > > >     > > > others
> >     > > > > >     > > >     > this message or any attachment. Please also
> > notify
> >     > > the
> >     > > > > > sender by
> >     > > > > >     > > > replying
> >     > > > > >     > > >     > to this email or by telephone (+44(020 7896
> > 0011
> >     > <020%207896%200011>) and
> >     > > > > then
> >     > > > > > delete
> >     > > > > >     > > > the email
> >     > > > > >     > > >     > and any copies of it. Opinions, conclusion
> > (etc)
> >     > that
> >     > > > do
> >     > > > > > not
> >     > > > > >     > relate
> >     > > > > >     > > > to the
> >     > > > > >     > > >     > official business of this company shall be
> >     > understood
> >     > > > as
> >     > > > > > neither
> >     > > > > >     > > > given nor
> >     > > > > >     > > >     > endorsed by it. IG is a trading name of IG
> > Markets
> >     > > > > Limited
> >     > > > > > (a
> >     > > > > >     > > company
> >     > > > > >     > > >     > registered in England and Wales, company
> > number
> >     > > > 04008957)
> >     > > > > > and IG
> >     > > > > >     > > > Index
> >     > > > > >     > > >     > Limited (a company registered in England
> and
> > Wales,
> >     > > > > company
> >     > > > > >     > number
> >     > > > > >     > > >     > 01190902). Registered address at Cannon
> > Bridge
> >     > House,
> >     > > > 25
> >     > > > > > Dowgate
> >     > > > > >     > > > Hill,
> >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > > > number
> >     > > > > > 195355)
> >     > > > > >     > > > and IG
> >     > > > > >     > > >     > Index Limited (register number 114059) are
> >     > authorised
> >     > > > and
> >     > > > > >     > regulated
> >     > > > > >     > > > by the
> >     > > > > >     > > >     > Financial Conduct Authority.
> >     > > > > >     > > >     >
> >     > > > > >     > > >
> >     > > > > >     > > >
> >     > > > > >     > > > The information contained in this email is
> strictly
> >     > > > > confidential
> >     > > > > > and
> >     > > > > >     > for
> >     > > > > >     > > > the use of the addressee only, unless otherwise
> >     > indicated.
> >     > > If
> >     > > > > > you are
> >     > > > > >     > not
> >     > > > > >     > > > the intended recipient, please do not read, copy,
> > use or
> >     > > > > > disclose to
> >     > > > > >     > > others
> >     > > > > >     > > > this message or any attachment. Please also
> notify
> > the
> >     > > sender
> >     > > > > by
> >     > > > > >     > replying
> >     > > > > >     > > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > > > > delete the
> >     > > > > >     > > email
> >     > > > > >     > > > and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > > not
> >     > > > > > relate to
> >     > > > > >     > > the
> >     > > > > >     > > > official business of this company shall be
> > understood as
> >     > > > > neither
> >     > > > > > given
> >     > > > > >     > > nor
> >     > > > > >     > > > endorsed by it. IG is a trading name of IG
> Markets
> >     > Limited
> >     > > (a
> >     > > > > > company
> >     > > > > >     > > > registered in England and Wales, company number
> > 04008957)
> >     > > and
> >     > > > > IG
> >     > > > > > Index
> >     > > > > >     > > > Limited (a company registered in England and
> Wales,
> >     > company
> >     > > > > > number
> >     > > > > >     > > > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     > > > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> >     > > > > > 195355) and
> >     > > > > >     > IG
> >     > > > > >     > > > Index Limited (register number 114059) are
> > authorised and
> >     > > > > > regulated by
> >     > > > > >     > > the
> >     > > > > >     > > > Financial Conduct Authority.
> >     > > > > >     > > >
> >     > > > > >     > > The information contained in this email is strictly
> >     > > > confidential
> >     > > > > > and for
> >     > > > > >     > > the use of the addressee only, unless otherwise
> > indicated.
> >     > If
> >     > > > you
> >     > > > > > are not
> >     > > > > >     > > the intended recipient, please do not read, copy,
> > use or
> >     > > > disclose
> >     > > > > > to
> >     > > > > >     > others
> >     > > > > >     > > this message or any attachment. Please also notify
> > the
> >     > sender
> >     > > > by
> >     > > > > > replying
> >     > > > > >     > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > > delete
> >     > > > > > the
> >     > > > > >     > email
> >     > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     > > > > > relate to
> >     > > > > >     > the
> >     > > > > >     > > official business of this company shall be
> > understood as
> >     > > > neither
> >     > > > > > given
> >     > > > > >     > nor
> >     > > > > >     > > endorsed by it. IG is a trading name of IG Markets
> > Limited
> >     > (a
> >     > > > > > company
> >     > > > > >     > > registered in England and Wales, company number
> > 04008957)
> >     > and
> >     > > > IG
> >     > > > > > Index
> >     > > > > >     > > Limited (a company registered in England and Wales,
> > company
> >     > > > > number
> >     > > > > >     > > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     > > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     > > > 195355)
> >     > > > > > and IG
> >     > > > > >     > > Index Limited (register number 114059) are
> > authorised and
> >     > > > > > regulated by
> >     > > > > >     > the
> >     > > > > >     > > Financial Conduct Authority.
> >     > > > > >     > >
> >     > > > > >     > >
> >     > > > > >     > The information contained in this email is strictly
> >     > > confidential
> >     > > > > and
> >     > > > > > for
> >     > > > > >     > the use of the addressee only, unless otherwise
> > indicated. If
> >     > > you
> >     > > > > > are not
> >     > > > > >     > the intended recipient, please do not read, copy, use
> > or
> >     > > disclose
> >     > > > > to
> >     > > > > > others
> >     > > > > >     > this message or any attachment. Please also notify
> the
> > sender
> >     > > by
> >     > > > > > replying
> >     > > > > >     > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > delete
> >     > > > > > the email
> >     > > > > >     > and any copies of it. Opinions, conclusion (etc) that
> > do not
> >     > > > relate
> >     > > > > > to the
> >     > > > > >     > official business of this company shall be understood
> > as
> >     > > neither
> >     > > > > > given nor
> >     > > > > >     > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> >     > > > > company
> >     > > > > >     > registered in England and Wales, company number
> > 04008957) and
> >     > > IG
> >     > > > > > Index
> >     > > > > >     > Limited (a company registered in England and Wales,
> > company
> >     > > > number
> >     > > > > >     > 01190902). Registered address at Cannon Bridge House,
> > 25
> >     > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     > > 195355)
> >     > > > > > and IG
> >     > > > > >     > Index Limited (register number 114059) are authorised
> > and
> >     > > > regulated
> >     > > > > > by the
> >     > > > > >     > Financial Conduct Authority.
> >     > > > > >     >
> >     > > > > >     >
> >     > > > > >
> >     > > > > >
> >     > > > > > The information contained in this email is strictly
> > confidential
> >     > and
> >     > > > for
> >     > > > > > the use of the addressee only, unless otherwise indicated.
> > If you
> >     > are
> >     > > > not
> >     > > > > > the intended recipient, please do not read, copy, use or
> > disclose
> >     > to
> >     > > > > others
> >     > > > > > this message or any attachment. Please also notify the
> > sender by
> >     > > > replying
> >     > > > > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then delete the
> >     > > > > email
> >     > > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> >     > > to
> >     > > > > the
> >     > > > > > official business of this company shall be understood as
> > neither
> >     > > given
> >     > > > > nor
> >     > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> >     > company
> >     > > > > > registered in England and Wales, company number 04008957)
> > and IG
> >     > > Index
> >     > > > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > > Hill,
> >     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> >     > and
> >     > > > IG
> >     > > > > > Index Limited (register number 114059) are authorised and
> > regulated
> >     > > by
> >     > > > > the
> >     > > > > > Financial Conduct Authority.
> >     > > > > >
> >     > > > > The information contained in this email is strictly
> > confidential and
> >     > > for
> >     > > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > > not
> >     > > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > > others
> >     > > > > this message or any attachment. Please also notify the sender
> > by
> >     > > replying
> >     > > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the
> >     > > > email
> >     > > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> >     > to
> >     > > > the
> >     > > > > official business of this company shall be understood as
> > neither
> >     > given
> >     > > > nor
> >     > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > > registered in England and Wales, company number 04008957) and
> > IG
> >     > Index
> >     > > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > Hill,
> >     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > > IG
> >     > > > > Index Limited (register number 114059) are authorised and
> > regulated
> >     > by
> >     > > > the
> >     > > > > Financial Conduct Authority.
> >     > > > >
> >     > > > The information contained in this email is strictly
> confidential
> > and
> >     > for
> >     > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > not
> >     > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > others
> >     > > > this message or any attachment. Please also notify the sender
> by
> >     > replying
> >     > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the
> >     > > email
> >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > > the
> >     > > > official business of this company shall be understood as
> neither
> > given
> >     > > nor
> >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> >     > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > IG
> >     > > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > > the
> >     > > > Financial Conduct Authority.
> >     > > >
> >     > > The information contained in this email is strictly confidential
> > and for
> >     > > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > > the intended recipient, please do not read, copy, use or disclose
> > to
> >     > others
> >     > > this message or any attachment. Please also notify the sender by
> > replying
> >     > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the email
> >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > the
> >     > > official business of this company shall be understood as neither
> > given
> >     > nor
> >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > > Limited (a company registered in England and Wales, company
> number
> >     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > the
> >     > > Financial Conduct Authority.
> >     > >
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jay Kreps <ja...@confluent.io>]

Hey Michael,

Here is the compatibility concern I have:

   1. You have a consumer app that relies on value == null to indicate a
   delete (current semantics).
   2. You upgrade Kafka and your clients.
   3. Some producer starts using the tombstone field in combination with
   non-null.

I share Ismael's dislike of setting tombstones on records with null values.
This makes sense as a transitional state, but as an end state its a bit
weird. You'd expect to be able to mix null values and tombstones, and have
the null values remain and the tombstones get compacted. However what will
happen is both will be compacted and upon debugging this you'll learn that
we sometimes use null in the value to indicate tombstone. Ismael's solution
is a bigger compatibility break, though, so not sure if that is better.

My other question is the relationship to KIP-82. My read is that this KIP
solves some but not all of the problems KIP-82 is intended for. KIP-82, on
the other hand, seems to address most of the motivating uses for this KIP.
The exception is maybe item (5) on the list where you want to simultaneous
delete and convey some information to subscribers, but I couldn't construct
a concrete examples for that one. Do we need to rationalize these two KIPs?
That is, do you still advocate this proposal if we do KIP-82 and vice
versa? As you may have noticed I'm somewhat emotionally invested in the
simplicity of the core data model, so my default position is let's try to
avoid stuffing more stuff in, but if we have to add stuff I like each of
these individually more than doing both. :-)

-Jay




On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jay
>
> I disagree here that we are breaking any compatibility, we went through
> this on the discussion thread in fact with the help of that thread is how
> the kip came to the solution.
>
> Also on the supported combinations front you mention, we are not taking
> anything away afaik.
>
> Currently supported are only are:
> Null value = delete
> Non-null value = non delete
>
> With this kip we would support
> Null value + tombstone = delete
> Non null value + tombstone = delete
> Non null value + no tombstone = non delete
>
> As for the alternative idea, this is simply a new policy, how can there be
> confusion here? For this policy it would be explicit that tombstone marker
> would need to be set for a delete.
>
> I'm going to vent a little now as starting to get quite frustrated.
>
> We are going round in circles on kip-82 as per use cases there is now many
> use cases, how many more are needed? just because confluent don't see these
> doesn't mean they aren't real use cases other have, this is the point of
> the Apache foundation, it shouldn't be the view of just one organisation.
> It really is getting a feeling of the NIH syndrome. Rather than it being
> constructive on discussion of the implementation detail.
>
> kip-87 spawned from as on the kip call we all agreed this was needed. And
> would at least allow a custom wrapper be supported in a compacted topic,
> allowing meta data. Which again now I feel we are spinning wheels, and
> simply finding reasons not support it.
>
> Cheers
> Mike
>
>
>
> Sent using OWA for iPad
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Friday, December 16, 2016 7:09:23 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hey Michael,
>
> I do think it might have been better had we started with a separate concept
> of null vs delete. Given that we are where we are, I'm not sure that the
> possible cures we explored so far are better than the disease.
>
> I apologize for coming to this late, but I didn't really understand the
> implications of the KIP and that we'd be breaking compatibility with
> existing apps until the vote had begun, and in my defense I'm not sure the
> other folks voting did either.
>
> I think we all agree there are many existing apps that are built with the
> assumption of "null value non-tombstone" and it isn't possible to
> disambiguate these from tombstones on the producer. It isn't that anyone is
> saying we have to support all four possibilities at once, it's that we
> simply can't orphan one of the existing combinations or our users will eat
> us!
>
> If I've understood your alternate solution of adding another setting for
> compaction, I think this does fix the compatibility problem, but it adds an
> odd mode the user has to add on all their topics. While the current state
> is easily explainable, the resulting state where the meaning of tombstone
> and null are overlapping and ambiguous and dependent on a compaction
> setting that could change out of band or not be in sync with your code in
> some environment seems worse to me then where we currently are. I think the
> question is how would this combination be explained to users and does it
> make sense?
>
> -Jay
>
>
>
> On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Chaps,
> >
> > Can we either get one more +1 binding (we have 2 already) on the
> existing?
> >
> > Or have some response on the below possible alternative. We are keen to
> > get working on this, so we make next feature release.
> >
> > Cheers
> > Mike
> >
> >
> > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> >     Hi Ismael
> >
> >     Did you see our email this morning, what's your thoughts on this
> > approach to instead we simply have a brand new policy?
> >
> >     Cheers
> >     Mike
> >
> >
> >     Sent using OWA for iPhone
> >     ________________________________________
> >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
> Juma <
> > ismael@juma.me.uk>
> >     Sent: Tuesday, December 13, 2016 11:30:05 AM
> >     To: dev@kafka.apache.org
> >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> >     Yes, this is actually tricky to do in a way where we both have the
> > desired
> >     semantics and maintain compatibility. When someone creates a
> >     `ProducerRecord` with a `null` value today, the producer doesn't know
> > if
> >     it's meant to be a tombstone or not. For V3 messages, it's easy when
> > the
> >     constructor that takes a tombstone is used. However, if any other
> >     constructor is used, it's not clear. A couple of options I can think
> > of,
> >     none of them particularly nice:
> >
> >     1. Have a third state where tombstone = unknown and the broker would
> > set
> >     the tombstone bit if the value was null and the topic was compacted.
> > People
> >     that wanted to pass a non-null value for the tombstone would have to
> > use
> >     the constructor that takes a tombstone. The drawbacks: third state
> for
> >     tombstone in message format, message conversion at the broker for a
> > common
> >     case.
> >
> >     2. Extend MetadataResponse to optionally include topic configs, which
> > would
> >     make it possible for the producer to be smarter about setting the
> >     tombstone. It would only do it if a tombstone was not passed
> > explicitly,
> >     the value was null and the topic was compacted. The main drawback is
> > that
> >     the producer would be getting a bit more data for each topic even
> > though it
> >     probably won't use it most of the time. Extending MetadataResponse to
> >     return topic configs would be useful for other reasons as well, so
> that
> >     part seems OK.
> >
> >     In addition, for both proposals, we could consider adding warnings to
> > the
> >     documentation that the behaviour of the constructors that don't take
> a
> >     tombstone would change in the next major release so that tombstone =
> > false.
> >     Not sure if this would be worth it though.
> >
> >     Ismael
> >
> >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> > ewen@confluent.io>
> >     wrote:
> >
> >     > Michael,
> >     >
> >     > It kind of depends on how you want to interpret the tombstone flag.
> > If it's
> >     > purely a producer-facing Kafka-level thing that we treat as
> internal
> > to the
> >     > broker and log cleaner once the record is sent, then your approach
> > makes
> >     > sense. You're just moving copying the null-indicates-delete
> behavior
> > of the
> >     > old constructor into the tombstone flag.
> >     >
> >     > However, if you want this change to more generally decouple the
> idea
> > of
> >     > deletion and null values, then you are sometimes converting what
> > might be a
> >     > completely valid null value that doesn't indicate deletion into a
> >     > tombstone. Downstream applications could potentially handle these
> > cases
> >     > differently given the separation of deletion from value.
> >     >
> >     > I guess the question is if we want to try to support the latter
> even
> > for
> >     > topics where we have older produce requests. An example where this
> > could
> >     > come up is in something like a CDC Connector. If we try to support
> > the
> >     > semantic difference, a connector might write changes to Kafka using
> > the
> >     > tombstone flag to indicate when a row was truly deleted (vs an
> > update that
> >     > sets it to null but still present; this probably makes more sense
> > for CDC
> >     > from document stores or extracting single columns). There are
> various
> >     > reasons we might want to maintain the full log and not turn
> > compaction on
> >     > (or just use a time-based retention policy), but downstream
> > applications
> >     > might care to know the difference between a delete and a null
> value.
> > In
> >     > fact both versions of the same log (compacted and time-retention)
> > could be
> >     > useful and I don't think it'll be uncommon to maintain both or use
> > KIP-71
> >     > to maintain a hybrid compacted/retention topic.
> >     >
> >     > -Ewen
> >     >
> >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > wrote:
> >     >
> >     > > Hi Jay,
> >     > >
> >     > > Why wouldn't that work, the tombstone value is only looked at by
> > the
> >     > > broker, on a topic configured for compaction as such is benign on
> > non
> >     > > compacted topics. This is as much as sending a null value
> currently
> >     > >
> >     > >
> >     > > Regards
> >     > > Mike
> >     > >
> >     > >
> >     > >
> >     > > Sent using OWA for iPhone
> >     > > ________________________________________
> >     > > From: Jay Kreps <ja...@confluent.io>
> >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
> >     > > To: dev@kafka.apache.org
> >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > >
> >     > > Hey Michael,
> >     > >
> >     > > I'm not quite sure that works as that would translate ALL null
> > values to
> >     > > tombstones, even for non-compacted topics that use null as an
> > acceptable
> >     > > value sent by the producer and expected by the consumer.
> >     > >
> >     > > -Jay
> >     > >
> >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > > wrote:
> >     > >
> >     > > > Hi Ewen,
> >     > > >
> >     > > > I think the easiest way to show this is with code.
> >     > > >
> >     > > > As you can see we keep the existing behaviour for code/binaries
> > calling
> >     > > > the pre-existing constructors, whereby if the value is null the
> >     > tombstone
> >     > > > is set to true.
> >     > > >
> >     > > > Regards
> >     > > > Mike
> >     > > >
> >     > > >
> >     > > >
> >     > > >     /**
> >     > > >      * Creates a record with a specified timestamp to be sent
> to
> > a
> >     > > > specified topic and partition
> >     > > >      *
> >     > > >      * @param topic The topic the record will be appended to
> >     > > >      * @param partition The partition to which the record
> should
> > be
> >     > sent
> >     > > >      * @param timestamp The timestamp of the record
> >     > > >      * @param tombstone if the record should be treated as a
> > tombstone
> >     > if
> >     > > > the topic is compacted
> >     > > >      * @param key The key that will be included in the record
> >     > > >      * @param value The record contents
> >     > > >      */
> >     > > >     public ProducerRecord(String topic, Integer partition,
> > Boolean
> >     > > > tombstone, Long timestamp, K key, V value) {
> >     > > >         if (topic == null)
> >     > > >             throw new IllegalArgumentException("Topic cannot
> be
> >     > null.");
> >     > > >         if (timestamp != null && timestamp < 0)
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Invalid timestamp: %d.
> > Timestamp
> >     > > should
> >     > > > always be non-negative or null.", timestamp));
> >     > > >         if (partition != null && partition < 0)
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Invalid partition: %d.
> > Partition
> >     > > number
> >     > > > should always be non-negative or null.", partition));
> >     > > >         if (!tombstone && value == null){
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Tombstone must be true if
> null
> >     > > value"));
> >     > > >         }
> >     > > >         this.topic = topic;
> >     > > >         this.partition = partition;
> >     > > >         this.tombstone = tombstone;
> >     > > >         this.key = key;
> >     > > >         this.value = value;
> >     > > >         this.timestamp = timestamp;
> >     > > >     }
> >     > > >
> >     > > >     public ProducerRecord(String topic, Integer partition, Long
> >     > > timestamp,
> >     > > > K key, V value) {
> >     > > >         this(topic, partition, value == null, timestamp, key,
> > value);
> >     > > >     }
> >     > > > ________________________________________
> >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> >     > > > Sent: Sunday, December 11, 2016 5:45 AM
> >     > > > To: dev@kafka.apache.org
> >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > >
> >     > > > It seemed like this was addressed in the migration section,
> > wasn't it?
> >     > > The
> >     > > > V2 vs V3 requests and the fact that the broker will downgrade
> the
> >     > message
> >     > > > format (losing zero copy) if you issues a V2 request to a
> broker
> > using
> >     > V3
> >     > > > format handles compatibility, does it not? The existing
> > consumers won't
> >     > > see
> >     > > > the extra metadata in the value, but they will get a null
> > instead and
> >     > > treat
> >     > > > it as a tombstone. Certainly there is a performance impact, but
> > it
> >     > seemed
> >     > > > compatible.
> >     > > >
> >     > > > I'm worried about this though:
> >     > > >
> >     > > >
> >     > > >    - *NOTE *: With the new version of producer client using
> >     > > ProduceRequest
> >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set)
> > and
> >     > null
> >     > > > value
> >     > > >    should not be allowed as the older version of consumer using
> >     > > > FetchRequest
> >     > > >    V2 will think of this as a tombstone when its actually not.
> >     > > >
> >     > > >
> >     > > > Unless I'm misunderstanding, this ends up breaking binary
> > compatibility
> >     > > for
> >     > > > the Java API. It sounds like this suggests that passing a null
> > value to
> >     > > the
> >     > > > existing ProducerRecord constructors would generate an
> exception
> > since
> >     > > you
> >     > > > didn't explicitly enable the tombstone (via whatever new
> > constructor is
> >     > > > provided). But that means you can't swap in a newer client jar
> > without
> >     > > > recompiling and get the same behavior if your app deletes keys
> > using
> >     > the
> >     > > > current approach because your app will start throwing
> > exceptions. Maybe
> >     > > > this is a tradeoff we're ok with, but we've tried pretty hard
> to
> > avoid
> >     > > > breaking compatibility like this so far -- it makes picking up
> > bug
> >     > fixes
> >     > > in
> >     > > > the clients harder for users of frameworks that have to pin to
> > earlier
> >     > > > default versions for compatibility.
> >     > > >
> >     > > > But then later the KIP says:
> >     > > >
> >     > > >
> >     > > >    - The old constructors for ProducerRecord without this
> > parameter
> >     > will
> >     > > be
> >     > > >    kept but updated so that their default behaviour if setting
> > null
> >     > value
> >     > > > will
> >     > > >    be the tombstone will be set to true to keep existing
> > behaviour.
> >     > > >
> >     > > >
> >     > > > So maybe I am misinterpreting something.
> >     > > >
> >     > > > And just a nit re: motivation section, item 6 would be clearer
> > for a
> >     > > union
> >     > > > schema with null (or optional schemas in other formats), e.g.
> > [null,
> >     > > > string], in which case losing the schema truly is losing
> > information
> >     > > > (whereas null is already the only valid value for a pure null
> > schema).
> >     > > >
> >     > > > -Ewen
> >     > > >
> >     > > >
> >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> >     > >
> >     > > > wrote:
> >     > > >
> >     > > > > Hi Jay,
> >     > > > >
> >     > > > > Good point this detail is missing in the KIP write up. Ive
> > added this
> >     > > > now.
> >     > > > >
> >     > > > > Essentially simply just upgrading the clients we do not
> expect
> > any
> >     > > client
> >     > > > > app code change needed.
> >     > > > >
> >     > > > > Cheers
> >     > > > > Mike
> >     > > > > ________________________________________
> >     > > > > From: Jay Kreps <ja...@confluent.io>
> >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
> >     > > > > To: dev@kafka.apache.org
> >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > > >
> >     > > > > Michael,
> >     > > > >
> >     > > > > The compatibility section goes through the migration path,
> but
> > isn't
> >     > > the
> >     > > > > bigger compatibility issue with existing apps? There are many
> >     > (probably
> >     > > > > thousands) of apps in production that use this feature and
> > send null
> >     > to
> >     > > > > mean delete. It seems like this would break compatibility
> with
> > them,
> >     > > and
> >     > > > > they would have to be rewritten to right?
> >     > > > >
> >     > > > > -Jay
> >     > > > >
> >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> >     > Michael.Pearce@ig.com
> >     > > >
> >     > > > > wrote:
> >     > > > >
> >     > > > > > Hi Jun,
> >     > > > > >
> >     > > > > > 4) On v3 we honour the tombstone. As such we expect it to
> be
> > set
> >     > > > > correctly
> >     > > > > > as per the KIP.
> >     > > > > >
> >     > > > > > 4.1) why would we want to produce an error when its v3?
> This
> > is the
> >     > > > exact
> >     > > > > > purpose to support non-null tombstone’s
> >     > > > > > 4.2) again here im unclear on the question on the v3,
> produce
> >     > request
> >     > > > we
> >     > > > > > expect the tombstone flag to be set correctly.
> >     > > > > >
> >     > > > > > 4.4) compaction only occurs on compacted topics, the bit
> > makes no
> >     > > > > > difference and not looked at on un-compacted (time/size
> based
> >     > > > eviction).
> >     > > > > >
> >     > > > > >
> >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> >     > > > > >
> >     > > > > >     Hi, Michael,
> >     > > > > >
> >     > > > > >     4. Then, I think I misunderstood this point. Could you
> > document
> >     > > the
> >     > > > > >     following points in the wiki?
> >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
> > non-null
> >     > value,
> >     > > > > does
> >     > > > > > the
> >     > > > > >     send() get an error or does the producer automatically
> > set the
> >     > > > value
> >     > > > > to
> >     > > > > >     null?
> >     > > > > >     4.2 If producer V3 doesn't set tombstone, but provides
> a
> > null
> >     > > > value,
> >     > > > > > does
> >     > > > > >     the send() get an error or does the producer
> > automatically sets
> >     > > the
> >     > > > > >     tombstone?
> >     > > > > >     4.3 Does the broker only expect messages that (a) have
> no
> >     > > tombstone
> >     > > > > and
> >     > > > > >     non-null value; (b) have tombstone and null value and
> > reject
> >     > the
> >     > > > > > messages
> >     > > > > >     with an error code otherwise?
> >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
> > compacted
> >     > on
> >     > > > > not?
> >     > > > > >
> >     > > > > >     Thanks,
> >     > > > > >
> >     > > > > >     Jun
> >     > > > > >
> >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> >     > > > > Michael.Pearce@ig.com
> >     > > > > > >
> >     > > > > >     wrote:
> >     > > > > >
> >     > > > > >     > Not at all.  This only acts on compacted topics just
> > as what
> >     > > > occurs
> >     > > > > > today
> >     > > > > >     >
> >     > > > > >     > Sent using OWA for iPhone
> >     > > > > >     > ________________________________________
> >     > > > > >     > From: Jun Rao <ju...@confluent.io>
> >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> >     > > > > >     > To: dev@kafka.apache.org
> >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> > Flag
> >     > > > > >     >
> >     > > > > >     > Hi, Michael,
> >     > > > > >     >
> >     > > > > >     > 4. Hmm, does that mean the new client library can
> > never send
> >     > a
> >     > > > null
> >     > > > > > message
> >     > > > > >     > even to a regular topic? This seems like a change of
> > the
> >     > > existing
> >     > > > > > behavior.
> >     > > > > >     >
> >     > > > > >     > Thanks,
> >     > > > > >     >
> >     > > > > >     > Jun
> >     > > > > >     >
> >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> >     > > > > > Michael.Pearce@ig.com>
> >     > > > > >     > wrote:
> >     > > > > >     >
> >     > > > > >     > > Hi Jun,
> >     > > > > >     > >
> >     > > > > >     > > Re 4) That's because we expect the tombstone value
> > to be
> >     > set
> >     > > > > > correctly if
> >     > > > > >     > > message bit is 2, as such if an older client sends
> > in on
> >     > old
> >     > > > > > message the
> >     > > > > >     > > message is upcast and the bit is set correctly. And
> > such no
> >     > > > > longer
> >     > > > > > need
> >     > > > > >     > to
> >     > > > > >     > > check the value. Mayuresh can you confirm my
> > thinking and
> >     > > > > > understanding
> >     > > > > >     > of
> >     > > > > >     > > what we've discussed?
> >     > > > > >     > >
> >     > > > > >     > > The second point I understand what you're getting
> at
> > now my
> >     > > > > > apologies.
> >     > > > > >     > Yes
> >     > > > > >     > > this makes sense to save on touching the message,
> if
> > we're
> >     > > the
> >     > > > > > only kip
> >     > > > > >     > > going in, in this release.
> >     > > > > >     > >
> >     > > > > >     > > Cheers
> >     > > > > >     > > Mike
> >     > > > > >     > >
> >     > > > > >     > > Sent using OWA for iPhone
> >     > > > > >     > > ________________________________________
> >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
> >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> >     > > > > >     > > To: dev@kafka.apache.org
> >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone Flag
> >     > > > > >     > >
> >     > > > > >     > > Hi, Michael,
> >     > > > > >     > >
> >     > > > > >     > > 4. Is this updated in the wiki? The text "If the
> > magic byte
> >     > > on
> >     > > > > > message is
> >     > > > > >     > > 2, the broker should use the tombstone bit for log
> >     > > compaction."
> >     > > > > > doesn't
> >     > > > > >     > > seem to have changed.
> >     > > > > >     > >
> >     > > > > >     > > 2. My point is that if we change the message format
> > just
> >     > for
> >     > > > this
> >     > > > > > KIP, we
> >     > > > > >     > > should consider whether it's worth optimizing the
> > down
> >     > > > conversion
> >     > > > > > path
> >     > > > > >     > > (i.e., decide whether a conversion is needed by
> just
> >     > looking
> >     > > at
> >     > > > > the
> >     > > > > >     > > tombstone bit in the wrapper message) since
> > tombstone will
> >     > be
> >     > > > > used
> >     > > > > >     > rarely.
> >     > > > > >     > > However, if the message format change here is
> > combined with
> >     > > > other
> >     > > > > > KIPs,
> >     > > > > >     > > then this optimization likely won't be needed. The
> > latter
> >     > > > > probably
> >     > > > > > makes
> >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> > think?
> >     > > > > >     > >
> >     > > > > >     > > Other than those, +1 from me,
> >     > > > > >     > >
> >     > > > > >     > > Thanks,
> >     > > > > >     > >
> >     > > > > >     > > Jun
> >     > > > > >     > >
> >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> >     > > > > > Michael.Pearce@ig.com>
> >     > > > > >     > > wrote:
> >     > > > > >     > >
> >     > > > > >     > > > Hi Jun
> >     > > > > >     > > >
> >     > > > > >     > > > do we have your vote on this now?
> >     > > > > >     > > >
> >     > > > > >     > > > Any other concerns?
> >     > > > > >     > > >
> >     > > > > >     > > > Cheers
> >     > > > > >     > > > Mike
> >     > > > > >     > > >
> >     > > > > >     > > > Sent using OWA for iPhone
> >     > > > > >     > > > ________________________________________
> >     > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> >     > > > > >     > > > To: dev@kafka.apache.org
> >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone
> >     > Flag
> >     > > > > >     > > >
> >     > > > > >     > > > Hi Jun,
> >     > > > > >     > > >
> >     > > > > >     > > > Have updated. Thanks again for the feedback.
> >     > > > > >     > > >
> >     > > > > >     > > > Agree yes we should align up when it gets to
> that,
> > I
> >     > assume
> >     > > > > > you’ve
> >     > > > > >     > > flagged
> >     > > > > >     > > > the same in the other KIP?
> >     > > > > >     > > >
> >     > > > > >     > > > I think for now let’s get this KIP approved, then
> > we can
> >     > > > start
> >     > > > > > the work
> >     > > > > >     > > to
> >     > > > > >     > > > get an initial PR, then we can discuss how to
> > align the
> >     > two
> >     > > > if
> >     > > > > > needed.
> >     > > > > >     > > >
> >     > > > > >     > > > Cheers,
> >     > > > > >     > > > Mike
> >     > > > > >     > > >
> >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
> jun@confluent.io>
> >     > wrote:
> >     > > > > >     > > >
> >     > > > > >     > > >     Hi, Michael,
> >     > > > > >     > > >
> >     > > > > >     > > >     For 2), this is fine. Could you update the
> KIP
> > wiki
> >     > to
> >     > > > make
> >     > > > > > this
> >     > > > > >     > and
> >     > > > > >     > > > other
> >     > > > > >     > > >     points clearer? Other than that, the KIP
> looks
> > good
> >     > to
> >     > > > me.
> >     > > > > >     > > >
> >     > > > > >     > > >     An orthogonal thing is that there are other
> > KIPs such
> >     > > as
> >     > > > > > KIP-98
> >     > > > > >     > that
> >     > > > > >     > > > also
> >     > > > > >     > > >     intend to change the message format. If they
> > all get
> >     > > > > > approved, we
> >     > > > > >     > > > should
> >     > > > > >     > > >     think about whether it's better to just bump
> > up the
> >     > > magic
> >     > > > > > byte once
> >     > > > > >     > > to
> >     > > > > >     > > >     incorporate multiple format changes like we
> > did in
> >     > > > > > KIP-31/KIP-32.
> >     > > > > >     > > >
> >     > > > > >     > > >     Thanks,
> >     > > > > >     > > >
> >     > > > > >     > > >     Jun
> >     > > > > >     > > >
> >     > > > > >     > > >
> >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
> Pearce
> > <
> >     > > > > >     > > Michael.Pearce@ig.com>
> >     > > > > >     > > >     wrote:
> >     > > > > >     > > >
> >     > > > > >     > > >     > Hi Jao
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > Thanks for the response. Sorry for slow
> > reply, both
> >     > > > with
> >     > > > > > personal
> >     > > > > >     > > > sickness
> >     > > > > >     > > >     > and also battling some critical issues
> > encountered
> >     > > > since
> >     > > > > >     > upgrading
> >     > > > > >     > > to
> >     > > > > >     > > >     > 0.10.1.0
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > 1) Thans for spotting, Document error where
> > we
> >     > > branched
> >     > > > > > this KIP
> >     > > > > >     > > from
> >     > > > > >     > > >     > KIP-82, will get that removed.
> >     > > > > >     > > >     > 2) Intent is to do this just at the record
> > message
> >     > > > level.
> >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> >     > > corrected.
> >     > > > > >     > > >     > 4) As per discussion thread we will support
> >     > > tombstone +
> >     > > > > > null
> >     > > > > >     > value,
> >     > > > > >     > > >     > tombstone + non null value, no tombstone +
> > null
> >     > > value.
> >     > > > > >     > > >     > 5) I believe this was in the discussion
> > thread,
> >     > > > @Mayuresh
> >     > > > > > is this
> >     > > > > >     > > >     > something we’ve overlooked? I thought we
> > would down
> >     > > > > > convert and
> >     > > > > >     > > > remove the
> >     > > > > >     > > >     > value so the old consumer had existing
> > behavior, or
> >     > > is
> >     > > > > > there
> >     > > > > >     > > > something we
> >     > > > > >     > > >     > haven’t thought about?
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > Cheers
> >     > > > > >     > > >     > Mike
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> > jun@confluent.io>
> >     > > > > wrote:
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Hi, Michael,
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Thanks for the KIP. A few comments
> below.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     1. The message format change contains
> >     > > > "HeadersLength
> >     > > > > >     > Headers".
> >     > > > > >     > > > Is that
> >     > > > > >     > > >     >     intended?
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     2. For compressed messageset, is the
> > tombstone
> >     > > bit
> >     > > > > > only set
> >     > > > > >     > at
> >     > > > > >     > > > the
> >     > > > > >     > > >     > shallow
> >     > > > > >     > > >     >     level? Do we always leave that bit in
> the
> >     > wrapper
> >     > > > > > message
> >     > > > > >     > > unset?
> >     > > > > >     > > > An
> >     > > > > >     > > >     >     alternative is to set the tombstone bit
> > in the
> >     > > > > wrapper
> >     > > > > > if at
> >     > > > > >     > > > least one
> >     > > > > >     > > >     >     inner message has the tombstone bit
> set.
> > This
> >     > > makes
> >     > > > > > things a
> >     > > > > >     > > bit
> >     > > > > >     > > > more
> >     > > > > >     > > >     >     complicated, but we could potentially
> > exploit
> >     > > that
> >     > > > > for
> >     > > > > >     > > > optimizing down
> >     > > > > >     > > >     >     conversion. For example, we only need
> to
> >     > convert
> >     > > > > > messages
> >     > > > > >     > with
> >     > > > > >     > > > magic 2
> >     > > > > >     > > >     > to
> >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit
> > is set
> >     > > > > > (conversion is
> >     > > > > >     > > > always
> >     > > > > >     > > >     > needed
> >     > > > > >     > > >     >     from magic 2 to magic 0). Not sure if
> the
> >     > > > > optimization
> >     > > > > > is
> >     > > > > >     > worth
> >     > > > > >     > > > the
> >     > > > > >     > > >     >     complexity though.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     3. The referencing of the new version
> of
> >     > > > > >     > > > ProducerRequest/FetchRequest
> >     > > > > >     > > >     > is
> >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> > convention
> >     > > > starts
> >     > > > > at
> >     > > > > >     > version
> >     > > > > >     > > > at 0, I
> >     > > > > >     > > >     >     think the new version would be 3.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     4. "If the magic byte on message is 2,
> > the
> >     > broker
> >     > > > > > should use
> >     > > > > >     > > the
> >     > > > > >     > > >     > tombstone
> >     > > > > >     > > >     >     bit for log compaction." What about
> null
> > value?
> >     > > My
> >     > > > > >     > > understanding
> >     > > > > >     > > > is
> >     > > > > >     > > >     > that
> >     > > > > >     > > >     >     null value will be treated the same as
> > setting
> >     > > the
> >     > > > > > tombstone
> >     > > > > >     > > bit.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     5. For the migration path, it would be
> > useful
> >     > to
> >     > > > > > describe the
> >     > > > > >     > > > down
> >     > > > > >     > > >     >     conversion path to consumers (i.e.,
> > brokers on
> >     > > > > message
> >     > > > > > format
> >     > > > > >     > > > 0.10.2
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     consumers on older version).
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Thanks,
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Jun
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
> Michael
> >     > Pearce <
> >     > > > > >     > > > Michael.Pearce@ig.com
> >     > > > > >     > > >     > >
> >     > > > > >     > > >     >     wrote:
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     > Hi All,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > We have been discussing in the below
> > thread
> >     > and
> >     > > > > final
> >     > > > > >     > changes
> >     > > > > >     > > > have
> >     > > > > >     > > >     > been
> >     > > > > >     > > >     >     > made to the KIP wiki based on these
> >     > > discussions.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > We would now like to put to the vote
> > the
> >     > > > following
> >     > > > > > KIP:
> >     > > > > >     > > >     >     > https://cwiki.apache.org/
> >     > > > > > confluence/display/KAFKA/KIP-
> >     > > > > >     > 87+-+
> >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > This kip is for having a distinct
> > compaction
> >     > > > > > attribute
> >     > > > > >     > > > “tombstone”
> >     > > > > >     > > >     > flag
> >     > > > > >     > > >     >     > instead of relying on null value,
> > allowing
> >     > > > non-null
> >     > > > > > value
> >     > > > > >     > > > delete
> >     > > > > >     > > >     > messages.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > Many thanks,
> >     > > > > >     > > >     >     > Michael
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
> Pearce"
> > <
> >     > > > > >     > > Michael.Pearce@ig.com>
> >     > > > > >     > > >     > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Hi Mayuresh,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     LGTM. Ive just made one small
> > adjustment
> >     > > > > > updating the
> >     > > > > >     > > wire
> >     > > > > >     > > >     > protocol to
> >     > > > > >     > > >     >     > show the magic byte bump.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Do we think we’re good to put to
> a
> > vote?
> >     > Is
> >     > > > > > there any
> >     > > > > >     > > > other bits
> >     > > > > >     > > >     >     > needing discussion?
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Cheers
> >     > > > > >     > > >     >     >     Mike
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
> > Gharat" <
> >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
> >     > > > > >     > > >     >     > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Hi Michael,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         I have updated the migration
> > section
> >     > of
> >     > > > the
> >     > > > > > KIP.
> >     > > > > >     > Can
> >     > > > > >     > > > you
> >     > > > > >     > > >     > please
> >     > > > > >     > > >     >     > take a look?
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Thanks,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Mayuresh
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07
> > AM,
> >     > > Mayuresh
> >     > > > > > Gharat <
> >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         > Hi Michael,
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > That whilst sending
> > tombstone and
> >     > non
> >     > > > > null
> >     > > > > > value,
> >     > > > > >     > > the
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     > can expect
> >     > > > > >     > > >     >     >         > only to receive the
> non-null
> >     > message
> >     > > > only
> >     > > > > > in step
> >     > > > > >     > > > (3) is
> >     > > > > >     > > >     > this
> >     > > > > >     > > >     >     > correct?
> >     > > > > >     > > >     >     >         > ---> I do agree with you
> > here.
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Becket, Ismael : can you
> guys
> >     > review
> >     > > > the
> >     > > > > >     > migration
> >     > > > > >     > > > plan
> >     > > > > >     > > >     > listed
> >     > > > > >     > > >     >     > above using
> >     > > > > >     > > >     >     >         > magic byte?
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Thanks,
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Mayuresh
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at
> 8:58
> > AM,
> >     > > > Michael
> >     > > > > > Pearce <
> >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> >     > > > > >     > > >     >     >         > wrote:
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >> Many thanks for this
> > Mayuresh. I
> >     > > don't
> >     > > > > > have any
> >     > > > > >     > > >     > objections.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> I assume we should state:
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> That whilst sending
> > tombstone and
> >     > > non
> >     > > > > null
> >     > > > > >     > value,
> >     > > > > >     > > > the
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     > can expect
> >     > > > > >     > > >     >     >         >> only to receive the
> non-null
> >     > message
> >     > > > > only
> >     > > > > > in
> >     > > > > >     > step
> >     > > > > >     > > > (3) is
> >     > > > > >     > > >     > this
> >     > > > > >     > > >     >     > correct?
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Cheers
> >     > > > > >     > > >     >     >         >> Mike
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Sent using OWA for iPhone
> >     > > > > >     > > >     >     >         >>
> > ______________________________
> >     > > > > __________
> >     > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
> >     > > > > >     > gharatmayuresh15@gmail.com
> >     > > > > >     > > >
> >     > > > > >     > > >     >     >         >> Sent: Thursday, November
> > 17, 2016
> >     > > > > 5:18:41
> >     > > > > > PM
> >     > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
> >     > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> > KIP-87 -
> >     > Add
> >     > > > > > Compaction
> >     > > > > >     > > > Tombstone
> >     > > > > >     > > >     > Flag
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Hi Ismael,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Thanks for the
> explanation.
> >     > > > > >     > > >     >     >         >> Specially I like this part
> > where
> >     > in
> >     > > > you
> >     > > > > >     > mentioned
> >     > > > > >     > > > we can
> >     > > > > >     > > >     > get
> >     > > > > >     > > >     >     > rid of the
> >     > > > > >     > > >     >     >         >> older null value support
> > for log
> >     > > > > > compaction
> >     > > > > >     > later
> >     > > > > >     > > > on,
> >     > > > > >     > > >     > here :
> >     > > > > >     > > >     >     >         >> We can't change semantics
> > of the
> >     > > > message
> >     > > > > > format
> >     > > > > >     > > > without
> >     > > > > >     > > >     > having
> >     > > > > >     > > >     >     > a long
> >     > > > > >     > > >     >     >         >> transition period. And we
> > can't
> >     > rely
> >     > > > > >     > > >     >     >         >> on people reading
> > documentation or
> >     > > > > acting
> >     > > > > > on a
> >     > > > > >     > > > warning for
> >     > > > > >     > > >     >     > something so
> >     > > > > >     > > >     >     >         >> fundamental. As such, my
> > take is
> >     > > that
> >     > > > we
> >     > > > > > need to
> >     > > > > >     > > > bump the
> >     > > > > >     > > >     > magic
> >     > > > > >     > > >     >     > byte. The
> >     > > > > >     > > >     >     >         >> good news is
> >     > > > > >     > > >     >     >         >> that we don't have to
> > support all
> >     > > > > versions
> >     > > > > >     > > forever.
> >     > > > > >     > > > We
> >     > > > > >     > > >     > have
> >     > > > > >     > > >     >     > said that we
> >     > > > > >     > > >     >     >         >> will support direct
> > upgrades for 2
> >     > > > > years.
> >     > > > > > That
> >     > > > > >     > > > means that
> >     > > > > >     > > >     >     > message format
> >     > > > > >     > > >     >     >         >> version n could, in
> theory,
> > be
> >     > > > removed 2
> >     > > > > > years
> >     > > > > >     > > > after the
> >     > > > > >     > > >     > it's
> >     > > > > >     > > >     >     > introduced.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Just a heads up, I would
> > like to
> >     > > > mention
> >     > > > > > that
> >     > > > > >     > even
> >     > > > > >     > > > without
> >     > > > > >     > > >     >     > bumping magic
> >     > > > > >     > > >     >     >         >> byte, we will *NOT* loose
> > zero
> >     > copy
> >     > > as
> >     > > > > in
> >     > > > > > the
> >     > > > > >     > > > client(x+1)
> >     > > > > >     > > >     > in my
> >     > > > > >     > > >     >     >         >> explanation
> >     > > > > >     > > >     >     >         >> above will convert
> > internally a
> >     > null
> >     > > > > > value to
> >     > > > > >     > > have a
> >     > > > > >     > > >     > tombstone
> >     > > > > >     > > >     >     > bit set and
> >     > > > > >     > > >     >     >         >> a tombstone bit set to
> have
> > a null
> >     > > > value
> >     > > > > >     > > > automatically
> >     > > > > >     > > >     >     > internally and by
> >     > > > > >     > > >     >     >         >> the time we move to
> version
> > (x+2),
> >     > > the
> >     > > > > > clients
> >     > > > > >     > > > would have
> >     > > > > >     > > >     >     > upgraded.
> >     > > > > >     > > >     >     >         >> Obviously if we support a
> > request
> >     > > from
> >     > > > > >     > > consumer(x),
> >     > > > > >     > > > we
> >     > > > > >     > > >     > will
> >     > > > > >     > > >     >     > loose zero
> >     > > > > >     > > >     >     >         >> copy
> >     > > > > >     > > >     >     >         >> but that is the same case
> > with
> >     > magic
> >     > > > > byte.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> But if magic byte bump
> > makes life
> >     > > > easier
> >     > > > > > for
> >     > > > > >     > > > transition
> >     > > > > >     > > >     > for the
> >     > > > > >     > > >     >     > above
> >     > > > > >     > > >     >     >         >> reasons that you
> explained,
> > I am
> >     > OK
> >     > > > with
> >     > > > > > it
> >     > > > > >     > since
> >     > > > > >     > > > we are
> >     > > > > >     > > >     > going
> >     > > > > >     > > >     >     > to meet the
> >     > > > > >     > > >     >     >         >> end goal down the road :)
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> On a side note can we
> > update the
> >     > doc
> >     > > > > here
> >     > > > > > on
> >     > > > > >     > magic
> >     > > > > >     > > > byte
> >     > > > > >     > > >     > to say
> >     > > > > >     > > >     >     > that "*it
> >     > > > > >     > > >     >     >         >> should be bumped whenever
> > the
> >     > > message
> >     > > > > > format is
> >     > > > > >     > > > changed
> >     > > > > >     > > >     > or the
> >     > > > > >     > > >     >     >         >> interpretation of message
> > format
> >     > > > (usage
> >     > > > > > of the
> >     > > > > >     > > > reserved
> >     > > > > >     > > >     > bits as
> >     > > > > >     > > >     >     > well) is
> >     > > > > >     > > >     >     >         >> changed*".
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Hi Michael,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Here is the update plan
> > that we
> >     > > > > discussed
> >     > > > > >     > offline
> >     > > > > >     > > >     > yesterday :
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Currently the magic-byte
> > which
> >     > > > > > corresponds to
> >     > > > > >     > the
> >     > > > > >     > > >     >     > "message.format.version"
> >     > > > > >     > > >     >     >         >> is set to 1.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 1) On broker it will be
> set
> > to 1
> >     > > > > > initially.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 2) When a producer client
> > sends a
> >     > > > > message
> >     > > > > > with
> >     > > > > >     > > > magic-byte
> >     > > > > >     > > >     > = 2,
> >     > > > > >     > > >     >     > since the
> >     > > > > >     > > >     >     >         >> broker is on magic-byte =
> > 1, we
> >     > will
> >     > > > > down
> >     > > > > >     > convert
> >     > > > > >     > > > it,
> >     > > > > >     > > >     > which
> >     > > > > >     > > >     >     > means if the
> >     > > > > >     > > >     >     >         >> tombstone bit is set, the
> > value
> >     > will
> >     > > > be
> >     > > > > > set to
> >     > > > > >     > > > null. A
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     >         >> understanding magic-byte =
> > 1, will
> >     > > > still
> >     > > > > > work
> >     > > > > >     > with
> >     > > > > >     > > > this. A
> >     > > > > >     > > >     >     > consumer
> >     > > > > >     > > >     >     >         >> working
> >     > > > > >     > > >     >     >         >> with magic-byte =2 will
> > also be
> >     > able
> >     > > > to
> >     > > > > >     > understand
> >     > > > > >     > > > this,
> >     > > > > >     > > >     > since
> >     > > > > >     > > >     >     > it
> >     > > > > >     > > >     >     >         >> understands the tombstone.
> >     > > > > >     > > >     >     >         >> Now there is still the
> > question of
> >     > > > > > supporting a
> >     > > > > >     > > >     > non-tombstone
> >     > > > > >     > > >     >     > and null
> >     > > > > >     > > >     >     >         >> value from producer client
> > with
> >     > > > > > magic-byte = 2.*
> >     > > > > >     > > (I
> >     > > > > >     > > > am
> >     > > > > >     > > >     > not sure
> >     > > > > >     > > >     >     > if we
> >     > > > > >     > > >     >     >         >> should support this.
> > Ismael/Becket
> >     > > can
> >     > > > > > comment
> >     > > > > >     > > > here)*
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 3) When almost all the
> > clients
> >     > have
> >     > > > > > upgraded,
> >     > > > > >     > the
> >     > > > > >     > > >     >     > message.format.version
> >     > > > > >     > > >     >     >         >> on
> >     > > > > >     > > >     >     >         >> the broker can be changed
> > to 2,
> >     > > where
> >     > > > in
> >     > > > > > the
> >     > > > > >     > down
> >     > > > > >     > > >     > conversion in
> >     > > > > >     > > >     >     > the above
> >     > > > > >     > > >     >     >         >> step will not happen. If
> at
> > this
> >     > > point
> >     > > > > we
> >     > > > > > get a
> >     > > > > >     > > > consumer
> >     > > > > >     > > >     >     > request from a
> >     > > > > >     > > >     >     >         >> older consumer, we might
> > have to
> >     > > down
> >     > > > > > convert
> >     > > > > >     > > where
> >     > > > > >     > > > in we
> >     > > > > >     > > >     > loose
> >     > > > > >     > > >     >     > zero copy,
> >     > > > > >     > > >     >     >         >> but these cases should be
> > rare.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Becket can you review this
> > plan
> >     > and
> >     > > > add
> >     > > > > > more
> >     > > > > >     > > > details if I
> >     > > > > >     > > >     > have
> >     > > > > >     > > >     >     >         >> missed/wronged something,
> > before
> >     > we
> >     > > > put
> >     > > > > > it on
> >     > > > > >     > KIP.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Thanks,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Mayuresh
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> > 11:07 PM,
> >     > > > > Michael
> >     > > > > >     > Pearce <
> >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> >     > > > > >     > > >     >     >         >> wrote:
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> > Thanks guys, for
> > discussing this
> >     > > > > > offline and
> >     > > > > >     > > > getting
> >     > > > > >     > > >     > some
> >     > > > > >     > > >     >     > consensus.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > So its clear for myself
> > and
> >     > others
> >     > > > > what
> >     > > > > > is
> >     > > > > >     > > > proposed now
> >     > > > > >     > > >     > (i
> >     > > > > >     > > >     >     > think i
> >     > > > > >     > > >     >     >         >> > understand, but want to
> > make
> >     > sure)
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Could i ask either
> > directly
> >     > update
> >     > > > the
> >     > > > > > kip to
> >     > > > > >     > > > detail the
> >     > > > > >     > > >     >     > migration
> >     > > > > >     > > >     >     >         >> > strategy, or (re-)state
> > your
> >     > > offline
> >     > > > > > discussed
> >     > > > > >     > > and
> >     > > > > >     > > >     > agreed
> >     > > > > >     > > >     >     > migration
> >     > > > > >     > > >     >     >         >> > strategy based on a
> magic
> > byte
> >     > is
> >     > > in
> >     > > > > > this
> >     > > > > >     > > thread.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > The main original driver
> > for the
> >     > > KIP
> >     > > > > > was to
> >     > > > > >     > > > support
> >     > > > > >     > > >     >     > compaction where
> >     > > > > >     > > >     >     >         >> value
> >     > > > > >     > > >     >     >         >> > isn't null, based off
> the
> >     > > > discussions
> >     > > > > on
> >     > > > > >     > KIP-82
> >     > > > > >     > > > thread.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > We should be able to
> > support
> >     > > > > > non-tombstone +
> >     > > > > >     > > null
> >     > > > > >     > > > value
> >     > > > > >     > > >     > by the
> >     > > > > >     > > >     >     >         >> completion
> >     > > > > >     > > >     >     >         >> > of the KIP, as we noted
> > when
> >     > > > > discussing
> >     > > > > > this
> >     > > > > >     > > kip,
> >     > > > > >     > > > having
> >     > > > > >     > > >     >     > logic based on
> >     > > > > >     > > >     >     >         >> a
> >     > > > > >     > > >     >     >         >> > null value isn't very
> > clean and
> >     > > also
> >     > > > > > separates
> >     > > > > >     > > the
> >     > > > > >     > > >     > concerns.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > As discussed already
> > though we
> >     > can
> >     > > > > > split this
> >     > > > > >     > > into
> >     > > > > >     > > >     > KIP-87a
> >     > > > > >     > > >     >     > and KIP-87b
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Where we look to deliver
> > KIP-87a
> >     > > on
> >     > > > a
> >     > > > > >     > compacted
> >     > > > > >     > > > topic
> >     > > > > >     > > >     > (to
> >     > > > > >     > > >     >     > address the
> >     > > > > >     > > >     >     >         >> > immediate issues)
> >     > > > > >     > > >     >     >         >> > * tombstone + null value
> >     > > > > >     > > >     >     >         >> > * tombstone + non-null
> > value
> >     > > > > >     > > >     >     >         >> > * non-tombstone +
> > non-null value
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Then we can discuss once
> > KIP-87a
> >     > > is
> >     > > > > > completed
> >     > > > > >     > > > options
> >     > > > > >     > > >     > later
> >     > > > > >     > > >     >     > and how we
> >     > > > > >     > > >     >     >         >> > support the second part
> > KIP-87b
> >     > to
> >     > > > > > deliver:
> >     > > > > >     > > >     >     >         >> > * non-tombstone + null
> > value
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Cheers
> >     > > > > >     > > >     >     >         >> > Mike
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> > ______________________________
> >     > > > > > __________
> >     > > > > >     > > >     >     >         >> > From: Becket Qin <
> >     > > > > becket.qin@gmail.com>
> >     > > > > >     > > >     >     >         >> > Sent: Thursday, November
> > 17,
> >     > 2016
> >     > > > 1:43
> >     > > > > > AM
> >     > > > > >     > > >     >     >         >> > To:
> dev@kafka.apache.org
> >     > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> > KIP-87 -
> >     > > Add
> >     > > > > > Compaction
> >     > > > > >     > > >     > Tombstone Flag
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had
> > an
> >     > > offline
> >     > > > > >     > discussion,
> >     > > > > >     > > > and
> >     > > > > >     > > >     > following
> >     > > > > >     > > >     >     > is a brief
> >     > > > > >     > > >     >     >         >> > summary.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > 1. We agreed that not
> > bumping up
> >     > > > magic
> >     > > > > > value
> >     > > > > >     > may
> >     > > > > >     > > > result
> >     > > > > >     > > >     > in
> >     > > > > >     > > >     >     > losing zero
> >     > > > > >     > > >     >     >         >> copy
> >     > > > > >     > > >     >     >         >> > during migration.
> >     > > > > >     > > >     >     >         >> > 2. Given that bumping up
> > magic
> >     > > value
> >     > > > > is
> >     > > > > > almost
> >     > > > > >     > > > free and
> >     > > > > >     > > >     > has
> >     > > > > >     > > >     >     > benefit of
> >     > > > > >     > > >     >     >         >> > avoiding potential
> > performance
> >     > > > issue.
> >     > > > > > It is
> >     > > > > >     > > > probably
> >     > > > > >     > > >     > worth
> >     > > > > >     > > >     >     > doing.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > One issue we still need
> > to think
> >     > > > about
> >     > > > > > is
> >     > > > > >     > > whether
> >     > > > > >     > > > we
> >     > > > > >     > > >     > want to
> >     > > > > >     > > >     >     > support a
> >     > > > > >     > > >     >     >         >> > non-tombstone message
> > with null
> >     > > > value.
> >     > > > > >     > > >     >     >         >> > Currently it is not
> > supported by
> >     > > > > Kafka.
> >     > > > > > If we
> >     > > > > >     > > > allow a
> >     > > > > >     > > >     >     > non-tombstone null
> >     > > > > >     > > >     >     >         >> > value message to exist
> > after
> >     > > KIP-87.
> >     > > > > The
> >     > > > > >     > problem
> >     > > > > >     > > > is
> >     > > > > >     > > >     > that such
> >     > > > > >     > > >     >     > message
> >     > > > > >     > > >     >     >         >> will
> >     > > > > >     > > >     >     >         >> > not be supported by the
> >     > consumers
> >     > > > > prior
> >     > > > > > to
> >     > > > > >     > > KIP-87.
> >     > > > > >     > > >     > Because a
> >     > > > > >     > > >     >     > null value
> >     > > > > >     > > >     >     >         >> > will always be
> > interpreted to a
> >     > > > > > tombstone.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > One option is that we
> > keep the
> >     > > > current
> >     > > > > > way,
> >     > > > > >     > i.e.
> >     > > > > >     > > > do not
> >     > > > > >     > > >     >     > support such
> >     > > > > >     > > >     >     >         >> > message. It would be
> good
> > to
> >     > know
> >     > > if
> >     > > > > > there is
> >     > > > > >     > a
> >     > > > > >     > > >     > concrete use
> >     > > > > >     > > >     >     > case for
> >     > > > > >     > > >     >     >         >> such
> >     > > > > >     > > >     >     >         >> > message. If there is
> not,
> > we can
> >     > > > > > probably just
> >     > > > > >     > > not
> >     > > > > >     > > >     > support it.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Thanks,
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at
> > 1:28 PM,
> >     > > > > > Mayuresh
> >     > > > > >     > > Gharat <
> >     > > > > >     > > >     >     >         >> >
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > wrote:
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > > Hi Ismael,
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > This is something I
> can
> > think
> >     > of
> >     > > > for
> >     > > > > >     > migration
> >     > > > > >     > > > plan:
> >     > > > > >     > > >     >     >         >> > > So the migration plan
> > can look
> >     > > > > > something
> >     > > > > >     > like
> >     > > > > >     > > > this,
> >     > > > > >     > > >     > with up
> >     > > > > >     > > >     >     >         >> conversion :
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > 1) Currently lets say
> > we have
> >     > > > Broker
> >     > > > > > at
> >     > > > > >     > > version
> >     > > > > >     > > > x.
> >     > > > > >     > > >     >     >         >> > > 2) Currently we have
> > clients
> >     > at
> >     > > > > > version x.
> >     > > > > >     > > >     >     >         >> > > 3) a) We move the
> > version to
> >     > > > > > Broker(x+1) :
> >     > > > > >     > > > supports
> >     > > > > >     > > >     > both
> >     > > > > >     > > >     >     > tombstone and
> >     > > > > >     > > >     >     >         >> > null
> >     > > > > >     > > >     >     >         >> > > for log compaction.
> >     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> > client
> >     > to
> >     > > > > > version
> >     > > > > >     > > > client(x+1) :
> >     > > > > >     > > >     > if in
> >     > > > > >     > > >     >     > the
> >     > > > > >     > > >     >     >         >> producer
> >     > > > > >     > > >     >     >         >> > > client(x+1) the value
> > is set
> >     > to
> >     > > > > null,
> >     > > > > > we
> >     > > > > >     > will
> >     > > > > >     > > >     > automatically
> >     > > > > >     > > >     >     > set the
> >     > > > > >     > > >     >     >         >> > > Tombstone bit
> > internally. If
> >     > the
> >     > > > > > producer
> >     > > > > >     > > > client(x+1)
> >     > > > > >     > > >     > sets
> >     > > > > >     > > >     >     > the
> >     > > > > >     > > >     >     >         >> tombstone
> >     > > > > >     > > >     >     >         >> > > itself, well and good.
> > For
> >     > > > producer
> >     > > > > >     > client(x),
> >     > > > > >     > > > the
> >     > > > > >     > > >     > broker
> >     > > > > >     > > >     >     > will up
> >     > > > > >     > > >     >     >         >> convert
> >     > > > > >     > > >     >     >         >> > > to have the tombstone
> > bit.
> >     > > > > > Broker(x+1) is
> >     > > > > >     > > > supporting
> >     > > > > >     > > >     > both.
> >     > > > > >     > > >     >     > Consumer
> >     > > > > >     > > >     >     >         >> > > client(x+1) will be
> > aware of
> >     > > this
> >     > > > > and
> >     > > > > > should
> >     > > > > >     > > be
> >     > > > > >     > > > able
> >     > > > > >     > > >     > to
> >     > > > > >     > > >     >     > handle this.
> >     > > > > >     > > >     >     >         >> For
> >     > > > > >     > > >     >     >         >> > > consumer client(x) we
> > will
> >     > down
> >     > > > > > convert the
> >     > > > > >     > > > message
> >     > > > > >     > > >     > on the
> >     > > > > >     > > >     >     > broker
> >     > > > > >     > > >     >     >         >> side.
> >     > > > > >     > > >     >     >         >> > >     c) At this point
> we
> > will
> >     > > have
> >     > > > to
> >     > > > > >     > specify a
> >     > > > > >     > > >     > warning or
> >     > > > > >     > > >     >     > clearly
> >     > > > > >     > > >     >     >         >> specify
> >     > > > > >     > > >     >     >         >> > > in docs that this
> > behavior is
> >     > > > about
> >     > > > > > to be
> >     > > > > >     > > > changed for
> >     > > > > >     > > >     > log
> >     > > > > >     > > >     >     > compaction.
> >     > > > > >     > > >     >     >         >> > > 4) a) In next release
> > of the
> >     > > > > > Broker(x+2), we
> >     > > > > >     > > > say that
> >     > > > > >     > > >     > only
> >     > > > > >     > > >     >     > Tombstone
> >     > > > > >     > > >     >     >         >> is
> >     > > > > >     > > >     >     >         >> > > used for log
> compaction
> > on the
> >     > > > > Broker
> >     > > > > > side.
> >     > > > > >     > > >     > Clients(x+1)
> >     > > > > >     > > >     >     > still is
> >     > > > > >     > > >     >     >         >> > > supported.
> >     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> > client
> >     > to
> >     > > > > > version
> >     > > > > >     > > > client(x+2) :
> >     > > > > >     > > >     > if
> >     > > > > >     > > >     >     > value is set
> >     > > > > >     > > >     >     >         >> to
> >     > > > > >     > > >     >     >         >> > > null, tombstone will
> > not be
> >     > set
> >     > > > > >     > automatically.
> >     > > > > >     > > > The
> >     > > > > >     > > >     > client
> >     > > > > >     > > >     >     > will have to
> >     > > > > >     > > >     >     >         >> > call
> >     > > > > >     > > >     >     >         >> > > setTombstone() to
> > actually set
> >     > > the
> >     > > > > >     > tombstone.
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > We should compare this
> >     > migration
> >     > > > > plan
> >     > > > > > with
> >     > > > > >     > the
> >     > > > > >     > > >     > migration
> >     > > > > >     > > >     >     > plan for
> >     > > > > >     > > >     >     >         >> magic
> >     > > > > >     > > >     >     >         >> > > byte bump and do
> > whatever
> >     > looks
> >     > > > > good.
> >     > > > > >     > > >     >     >         >> > > I am just worried that
> > if we
> >     > go
> >     > > > down
> >     > > > > > magic
> >     > > > > >     > > byte
> >     > > > > >     > > > route,
> >     > > > > >     > > >     >     > unless I am
> >     > > > > >     > > >     >     >         >> > missing
> >     > > > > >     > > >     >     >         >> > > something, it sounds
> > like
> >     > kafka
> >     > > > will
> >     > > > > > be
> >     > > > > >     > stuck
> >     > > > > >     > > > with
> >     > > > > >     > > >     >     > supporting both
> >     > > > > >     > > >     >     >         >> null
> >     > > > > >     > > >     >     >         >> > > value and tombstone
> bit
> > for
> >     > log
> >     > > > > > compaction
> >     > > > > >     > for
> >     > > > > >     > > > life
> >     > > > > >     > > >     > long,
> >     > > > > >     > > >     >     > which does
> >     > > > > >     > > >     >     >         >> not
> >     > > > > >     > > >     >     >         >> > > look like a good end
> > state.
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > Thanks,
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > Mayuresh
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016
> at
> > 9:32
> >     > AM,
> >     > > > > > Mayuresh
> >     > > > > >     > > > Gharat <
> >     > > > > >     > > >     >     >         >> > >
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > > wrote:
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > That's a very good
> > point
> >     > > which I
> >     > > > > > might
> >     > > > > >     > have
> >     > > > > >     > > > not
> >     > > > > >     > > >     >     > considered earlier.
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Here is a plan that
> I
> > can
> >     > > think
> >     > > > > of:
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Stage 1) The broker
> > from now
> >     > > on,
> >     > > > > up
> >     > > > > >     > converts
> >     > > > > >     > > > the
> >     > > > > >     > > >     > message
> >     > > > > >     > > >     >     > to have the
> >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> The
> > log
> >     > > > > compaction
> >     > > > > >     > thread
> >     > > > > >     > > > does log
> >     > > > > >     > > >     >     > compaction
> >     > > > > >     > > >     >     >         >> based
> >     > > > > >     > > >     >     >         >> > on
> >     > > > > >     > > >     >     >         >> > > > both null and
> > tombstone
> >     > > marker.
> >     > > > > > This is
> >     > > > > >     > our
> >     > > > > >     > > >     > transition
> >     > > > > >     > > >     >     > period.
> >     > > > > >     > > >     >     >         >> > > > Stage 2) The next
> > release we
> >     > > > only
> >     > > > > > say that
> >     > > > > >     > > log
> >     > > > > >     > > >     > compaction
> >     > > > > >     > > >     >     > is based
> >     > > > > >     > > >     >     >         >> on
> >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> > (Open
> >     > source
> >     > > > > > kafka makes
> >     > > > > >     > > > this as a
> >     > > > > >     > > >     >     > policy). By
> >     > > > > >     > > >     >     >         >> this
> >     > > > > >     > > >     >     >         >> > > time,
> >     > > > > >     > > >     >     >         >> > > > the organization
> > which is
> >     > > moving
> >     > > > > to
> >     > > > > > this
> >     > > > > >     > > > release
> >     > > > > >     > > >     > will be
> >     > > > > >     > > >     >     > sure that
> >     > > > > >     > > >     >     >         >> they
> >     > > > > >     > > >     >     >         >> > > > have gone through
> the
> > entire
> >     > > > > > transition
> >     > > > > >     > > > period.
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > My only goal of
> doing
> > this
> >     > is
> >     > > > that
> >     > > > > > Kafka
> >     > > > > >     > > > clearly
> >     > > > > >     > > >     >     > specifies the end
> >     > > > > >     > > >     >     >         >> > state
> >     > > > > >     > > >     >     >         >> > > > about what log
> > compaction
> >     > > means
> >     > > > > (is
> >     > > > > > it
> >     > > > > >     > null
> >     > > > > >     > > > value
> >     > > > > >     > > >     > or a
> >     > > > > >     > > >     >     > tombstone
> >     > > > > >     > > >     >     >         >> > marker,
> >     > > > > >     > > >     >     >         >> > > > but not both).
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > What do you think?
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Thanks,
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Mayuresh
> >     > > > > >     > > >     >     >         >> > > > .
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016
> > at 9:17
> >     > > AM,
> >     > > > > > Ismael
> >     > > > > >     > > Juma <
> >     > > > > >     > > >     >     > ismael@juma.me.uk>
> >     > > > > >     > > >     >     >         >> > wrote:
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >> One comment below.
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16,
> 2016
> > at
> >     > 5:08
> >     > > > PM,
> >     > > > > > Mayuresh
> >     > > > > >     > > > Gharat <
> >     > > > > >     > > >     >     >         >> > > >>
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > >> > wrote:
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> >    - If we don't
> > bump up
> >     > > the
> >     > > > > > magic
> >     > > > > >     > byte,
> >     > > > > >     > > > on the
> >     > > > > >     > > >     > broker
> >     > > > > >     > > >     >     > side, the
> >     > > > > >     > > >     >     >         >> > > broker
> >     > > > > >     > > >     >     >         >> > > >> >    will always
> > have to
> >     > look
> >     > > > at
> >     > > > > > both
> >     > > > > >     > > > tombstone
> >     > > > > >     > > >     > bit and
> >     > > > > >     > > >     >     > the value
> >     > > > > >     > > >     >     >         >> when
> >     > > > > >     > > >     >     >         >> > > do
> >     > > > > >     > > >     >     >         >> > > >> the
> >     > > > > >     > > >     >     >         >> > > >> >    compaction.
> > Assuming
> >     > we
> >     > > do
> >     > > > > > not bump
> >     > > > > >     > up
> >     > > > > >     > > > the
> >     > > > > >     > > >     > magic
> >     > > > > >     > > >     >     > byte,
> >     > > > > >     > > >     >     >         >> > > >> >    imagine the
> > broker
> >     > sees
> >     > > a
> >     > > > > > message
> >     > > > > >     > > which
> >     > > > > >     > > > does
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > have a
> >     > > > > >     > > >     >     >         >> tombstone
> >     > > > > >     > > >     >     >         >> > > bit
> >     > > > > >     > > >     >     >         >> > > >> >    set. The
> broker
> > does
> >     > not
> >     > > > > know
> >     > > > > > when
> >     > > > > >     > the
> >     > > > > >     > > >     > message was
> >     > > > > >     > > >     >     > produced
> >     > > > > >     > > >     >     >         >> (i.e.
> >     > > > > >     > > >     >     >         >> > > >> > whether
> >     > > > > >     > > >     >     >         >> > > >> >    the message
> has
> > been
> >     > up
> >     > > > > > converted or
> >     > > > > >     > > > not), it
> >     > > > > >     > > >     > has
> >     > > > > >     > > >     >     > to take a
> >     > > > > >     > > >     >     >         >> > further
> >     > > > > >     > > >     >     >         >> > > >> > look at
> >     > > > > >     > > >     >     >         >> > > >> >    the value to
> > see if it
> >     > > is
> >     > > > > > null or
> >     > > > > >     > not
> >     > > > > >     > > in
> >     > > > > >     > > >     > order to
> >     > > > > >     > > >     >     > determine
> >     > > > > >     > > >     >     >         >> if it
> >     > > > > >     > > >     >     >         >> > > is
> >     > > > > >     > > >     >     >         >> > > >> a
> >     > > > > >     > > >     >     >         >> > > >> >    tombstone. The
> > same
> >     > > logic
> >     > > > > has
> >     > > > > > to be
> >     > > > > >     > > put
> >     > > > > >     > > > on the
> >     > > > > >     > > >     >     > consumer as
> >     > > > > >     > > >     >     >         >> well
> >     > > > > >     > > >     >     >         >> > > >> because
> >     > > > > >     > > >     >     >         >> > > >> > the
> >     > > > > >     > > >     >     >         >> > > >> >    consumer does
> > not know
> >     > > if
> >     > > > > the
> >     > > > > >     > message
> >     > > > > >     > > > has
> >     > > > > >     > > >     > been up
> >     > > > > >     > > >     >     > converted or
> >     > > > > >     > > >     >     >         >> > not.
> >     > > > > >     > > >     >     >         >> > > >> >       - If we
> > upconvert
> >     > > while
> >     > > > > >     > appending,
> >     > > > > >     > > > this is
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > the case,
> >     > > > > >     > > >     >     >         >> > right?
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> If I understand you
> >     > > correctly,
> >     > > > > > this is
> >     > > > > >     > not
> >     > > > > >     > > >     > sufficient
> >     > > > > >     > > >     >     > because the
> >     > > > > >     > > >     >     >         >> log
> >     > > > > >     > > >     >     >         >> > > may
> >     > > > > >     > > >     >     >         >> > > >> have messages
> > appended
> >     > before
> >     > > > it
> >     > > > > > was
> >     > > > > >     > > > upgraded to
> >     > > > > >     > > >     > include
> >     > > > > >     > > >     >     > KIP-87.
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> Ismael
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > --
> >     > > > > >     > > >     >     >         >> > > > -Regards,
> >     > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > --
> >     > > > > >     > > >     >     >         >> > > -Regards,
> >     > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> > > (862) 250-7125
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > The information
> contained
> > in
> >     > this
> >     > > > > email
> >     > > > > > is
> >     > > > > >     > > > strictly
> >     > > > > >     > > >     >     > confidential and for
> >     > > > > >     > > >     >     >         >> > the use of the addressee
> > only,
> >     > > > unless
> >     > > > > >     > otherwise
> >     > > > > >     > > >     > indicated. If
> >     > > > > >     > > >     >     > you are
> >     > > > > >     > > >     >     >         >> not
> >     > > > > >     > > >     >     >         >> > the intended recipient,
> > please
> >     > do
> >     > > > not
> >     > > > > > read,
> >     > > > > >     > > copy,
> >     > > > > >     > > > use or
> >     > > > > >     > > >     >     > disclose to
> >     > > > > >     > > >     >     >         >> others
> >     > > > > >     > > >     >     >         >> > this message or any
> > attachment.
> >     > > > Please
> >     > > > > > also
> >     > > > > >     > > > notify the
> >     > > > > >     > > >     > sender
> >     > > > > >     > > >     >     > by
> >     > > > > >     > > >     >     >         >> replying
> >     > > > > >     > > >     >     >         >> > to this email or by
> > telephone
> >     > > > (+44(020
> >     > > > > > 7896
> >     > > > > >     > > 0011)
> >     > > > > >     > > > and
> >     > > > > >     > > >     > then
> >     > > > > >     > > >     >     > delete the
> >     > > > > >     > > >     >     >         >> email
> >     > > > > >     > > >     >     >         >> > and any copies of it.
> > Opinions,
> >     > > > > > conclusion
> >     > > > > >     > (etc)
> >     > > > > >     > > > that
> >     > > > > >     > > >     > do not
> >     > > > > >     > > >     >     > relate to
> >     > > > > >     > > >     >     >         >> the
> >     > > > > >     > > >     >     >         >> > official business of
> this
> >     > company
> >     > > > > shall
> >     > > > > > be
> >     > > > > >     > > > understood as
> >     > > > > >     > > >     >     > neither given
> >     > > > > >     > > >     >     >         >> nor
> >     > > > > >     > > >     >     >         >> > endorsed by it. IG is a
> > trading
> >     > > name
> >     > > > > of
> >     > > > > > IG
> >     > > > > >     > > Markets
> >     > > > > >     > > >     > Limited (a
> >     > > > > >     > > >     >     > company
> >     > > > > >     > > >     >     >         >> > registered in England
> and
> > Wales,
> >     > > > > company
> >     > > > > >     > number
> >     > > > > >     > > >     > 04008957) and
> >     > > > > >     > > >     >     > IG Index
> >     > > > > >     > > >     >     >         >> > Limited (a company
> > registered in
> >     > > > > > England and
> >     > > > > >     > > > Wales,
> >     > > > > >     > > >     > company
> >     > > > > >     > > >     >     > number
> >     > > > > >     > > >     >     >         >> > 01190902). Registered
> > address at
> >     > > > > Cannon
> >     > > > > > Bridge
> >     > > > > >     > > > House, 25
> >     > > > > >     > > >     >     > Dowgate Hill,
> >     > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
> > Markets
> >     > > > > Limited
> >     > > > > >     > > (register
> >     > > > > >     > > >     > number
> >     > > > > >     > > >     >     > 195355) and IG
> >     > > > > >     > > >     >     >         >> > Index Limited (register
> > number
> >     > > > 114059)
> >     > > > > > are
> >     > > > > >     > > > authorised
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > regulated by
> >     > > > > >     > > >     >     >         >> the
> >     > > > > >     > > >     >     >         >> > Financial Conduct
> > Authority.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> --
> >     > > > > >     > > >     >     >         >> -Regards,
> >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> (862) 250-7125
> >     > > > > >     > > >     >     >         >> The information contained
> > in this
> >     > > > email
> >     > > > > is
> >     > > > > >     > > strictly
> >     > > > > >     > > >     >     > confidential and for
> >     > > > > >     > > >     >     >         >> the use of the addressee
> > only,
> >     > > unless
> >     > > > > > otherwise
> >     > > > > >     > > >     > indicated. If
> >     > > > > >     > > >     >     > you are not
> >     > > > > >     > > >     >     >         >> the intended recipient,
> > please do
> >     > > not
> >     > > > > > read,
> >     > > > > >     > copy,
> >     > > > > >     > > > use or
> >     > > > > >     > > >     >     > disclose to others
> >     > > > > >     > > >     >     >         >> this message or any
> > attachment.
> >     > > Please
> >     > > > > > also
> >     > > > > >     > notify
> >     > > > > >     > > > the
> >     > > > > >     > > >     > sender
> >     > > > > >     > > >     >     > by replying
> >     > > > > >     > > >     >     >         >> to this email or by
> > telephone
> >     > > (+44(020
> >     > > > > > 7896
> >     > > > > >     > 0011)
> >     > > > > >     > > > and then
> >     > > > > >     > > >     >     > delete the email
> >     > > > > >     > > >     >     >         >> and any copies of it.
> > Opinions,
> >     > > > > > conclusion (etc)
> >     > > > > >     > > > that do
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > relate to the
> >     > > > > >     > > >     >     >         >> official business of this
> > company
> >     > > > shall
> >     > > > > be
> >     > > > > >     > > > understood as
> >     > > > > >     > > >     >     > neither given nor
> >     > > > > >     > > >     >     >         >> endorsed by it. IG is a
> > trading
> >     > name
> >     > > > of
> >     > > > > IG
> >     > > > > >     > Markets
> >     > > > > >     > > >     > Limited (a
> >     > > > > >     > > >     >     > company
> >     > > > > >     > > >     >     >         >> registered in England and
> > Wales,
> >     > > > company
> >     > > > > > number
> >     > > > > >     > > > 04008957)
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > IG Index
> >     > > > > >     > > >     >     >         >> Limited (a company
> > registered in
> >     > > > England
> >     > > > > > and
> >     > > > > >     > > Wales,
> >     > > > > >     > > >     > company
> >     > > > > >     > > >     >     > number
> >     > > > > >     > > >     >     >         >> 01190902). Registered
> > address at
> >     > > > Cannon
> >     > > > > > Bridge
> >     > > > > >     > > > House, 25
> >     > > > > >     > > >     >     > Dowgate Hill,
> >     > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> > Markets
> >     > > > Limited
> >     > > > > >     > (register
> >     > > > > >     > > > number
> >     > > > > >     > > >     >     > 195355) and IG
> >     > > > > >     > > >     >     >         >> Index Limited (register
> > number
> >     > > 114059)
> >     > > > > are
> >     > > > > >     > > > authorised and
> >     > > > > >     > > >     >     > regulated by the
> >     > > > > >     > > >     >     >         >> Financial Conduct
> Authority.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > --
> >     > > > > >     > > >     >     >         > -Regards,
> >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         > (862) 250-7125
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         --
> >     > > > > >     > > >     >     >         -Regards,
> >     > > > > >     > > >     >     >         Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         (862) 250-7125
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     The information contained in this
> > email
> >     > is
> >     > > > > > strictly
> >     > > > > >     > > > confidential
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > for the use of the addressee only,
> > unless
> >     > > > otherwise
> >     > > > > >     > > indicated.
> >     > > > > >     > > > If
> >     > > > > >     > > >     > you are
> >     > > > > >     > > >     >     > not the intended recipient, please do
> > not
> >     > read,
> >     > > > > > copy, use
> >     > > > > >     > or
> >     > > > > >     > > >     > disclose to
> >     > > > > >     > > >     >     > others this message or any
> attachment.
> > Please
> >     > > > also
> >     > > > > > notify
> >     > > > > >     > the
> >     > > > > >     > > > sender
> >     > > > > >     > > >     > by
> >     > > > > >     > > >     >     > replying to this email or by
> telephone
> >     > (+44(020
> >     > > > > 7896
> >     > > > > > 0011)
> >     > > > > >     > > and
> >     > > > > >     > > > then
> >     > > > > >     > > >     > delete
> >     > > > > >     > > >     >     > the email and any copies of it.
> > Opinions,
> >     > > > > conclusion
> >     > > > > > (etc)
> >     > > > > >     > > > that do
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > relate to the official business of
> this
> >     > company
> >     > > > > > shall be
> >     > > > > >     > > > understood
> >     > > > > >     > > >     > as
> >     > > > > >     > > >     >     > neither given nor endorsed by it. IG
> > is a
> >     > > trading
> >     > > > > > name of
> >     > > > > >     > IG
> >     > > > > >     > > > Markets
> >     > > > > >     > > >     >     > Limited (a company registered in
> > England and
> >     > > > Wales,
> >     > > > > > company
> >     > > > > >     > > > number
> >     > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> > company
> >     > > > > registered
> >     > > > > > in
> >     > > > > >     > > > England and
> >     > > > > >     > > >     > Wales,
> >     > > > > >     > > >     >     > company number 01190902). Registered
> > address
> >     > at
> >     > > > > > Cannon
> >     > > > > >     > Bridge
> >     > > > > >     > > > House,
> >     > > > > >     > > >     > 25
> >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both
> IG
> >     > Markets
> >     > > > > > Limited
> >     > > > > >     > > > (register
> >     > > > > >     > > >     > number
> >     > > > > >     > > >     >     > 195355) and IG Index Limited
> (register
> > number
> >     > > > > > 114059) are
> >     > > > > >     > > > authorised
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > regulated by the Financial Conduct
> > Authority.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > The information contained in this email is
> > strictly
> >     > > > > > confidential
> >     > > > > >     > > and
> >     > > > > >     > > > for
> >     > > > > >     > > >     > the use of the addressee only, unless
> > otherwise
> >     > > > > indicated.
> >     > > > > > If you
> >     > > > > >     > > > are not
> >     > > > > >     > > >     > the intended recipient, please do not read,
> > copy,
> >     > use
> >     > > > or
> >     > > > > > disclose
> >     > > > > >     > > to
> >     > > > > >     > > > others
> >     > > > > >     > > >     > this message or any attachment. Please also
> > notify
> >     > > the
> >     > > > > > sender by
> >     > > > > >     > > > replying
> >     > > > > >     > > >     > to this email or by telephone (+44(020 7896
> > 0011
> >     > <020%207896%200011>) and
> >     > > > > then
> >     > > > > > delete
> >     > > > > >     > > > the email
> >     > > > > >     > > >     > and any copies of it. Opinions, conclusion
> > (etc)
> >     > that
> >     > > > do
> >     > > > > > not
> >     > > > > >     > relate
> >     > > > > >     > > > to the
> >     > > > > >     > > >     > official business of this company shall be
> >     > understood
> >     > > > as
> >     > > > > > neither
> >     > > > > >     > > > given nor
> >     > > > > >     > > >     > endorsed by it. IG is a trading name of IG
> > Markets
> >     > > > > Limited
> >     > > > > > (a
> >     > > > > >     > > company
> >     > > > > >     > > >     > registered in England and Wales, company
> > number
> >     > > > 04008957)
> >     > > > > > and IG
> >     > > > > >     > > > Index
> >     > > > > >     > > >     > Limited (a company registered in England
> and
> > Wales,
> >     > > > > company
> >     > > > > >     > number
> >     > > > > >     > > >     > 01190902). Registered address at Cannon
> > Bridge
> >     > House,
> >     > > > 25
> >     > > > > > Dowgate
> >     > > > > >     > > > Hill,
> >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > > > number
> >     > > > > > 195355)
> >     > > > > >     > > > and IG
> >     > > > > >     > > >     > Index Limited (register number 114059) are
> >     > authorised
> >     > > > and
> >     > > > > >     > regulated
> >     > > > > >     > > > by the
> >     > > > > >     > > >     > Financial Conduct Authority.
> >     > > > > >     > > >     >
> >     > > > > >     > > >
> >     > > > > >     > > >
> >     > > > > >     > > > The information contained in this email is
> strictly
> >     > > > > confidential
> >     > > > > > and
> >     > > > > >     > for
> >     > > > > >     > > > the use of the addressee only, unless otherwise
> >     > indicated.
> >     > > If
> >     > > > > > you are
> >     > > > > >     > not
> >     > > > > >     > > > the intended recipient, please do not read, copy,
> > use or
> >     > > > > > disclose to
> >     > > > > >     > > others
> >     > > > > >     > > > this message or any attachment. Please also
> notify
> > the
> >     > > sender
> >     > > > > by
> >     > > > > >     > replying
> >     > > > > >     > > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > > > > delete the
> >     > > > > >     > > email
> >     > > > > >     > > > and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > > not
> >     > > > > > relate to
> >     > > > > >     > > the
> >     > > > > >     > > > official business of this company shall be
> > understood as
> >     > > > > neither
> >     > > > > > given
> >     > > > > >     > > nor
> >     > > > > >     > > > endorsed by it. IG is a trading name of IG
> Markets
> >     > Limited
> >     > > (a
> >     > > > > > company
> >     > > > > >     > > > registered in England and Wales, company number
> > 04008957)
> >     > > and
> >     > > > > IG
> >     > > > > > Index
> >     > > > > >     > > > Limited (a company registered in England and
> Wales,
> >     > company
> >     > > > > > number
> >     > > > > >     > > > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     > > > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> >     > > > > > 195355) and
> >     > > > > >     > IG
> >     > > > > >     > > > Index Limited (register number 114059) are
> > authorised and
> >     > > > > > regulated by
> >     > > > > >     > > the
> >     > > > > >     > > > Financial Conduct Authority.
> >     > > > > >     > > >
> >     > > > > >     > > The information contained in this email is strictly
> >     > > > confidential
> >     > > > > > and for
> >     > > > > >     > > the use of the addressee only, unless otherwise
> > indicated.
> >     > If
> >     > > > you
> >     > > > > > are not
> >     > > > > >     > > the intended recipient, please do not read, copy,
> > use or
> >     > > > disclose
> >     > > > > > to
> >     > > > > >     > others
> >     > > > > >     > > this message or any attachment. Please also notify
> > the
> >     > sender
> >     > > > by
> >     > > > > > replying
> >     > > > > >     > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > > delete
> >     > > > > > the
> >     > > > > >     > email
> >     > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     > > > > > relate to
> >     > > > > >     > the
> >     > > > > >     > > official business of this company shall be
> > understood as
> >     > > > neither
> >     > > > > > given
> >     > > > > >     > nor
> >     > > > > >     > > endorsed by it. IG is a trading name of IG Markets
> > Limited
> >     > (a
> >     > > > > > company
> >     > > > > >     > > registered in England and Wales, company number
> > 04008957)
> >     > and
> >     > > > IG
> >     > > > > > Index
> >     > > > > >     > > Limited (a company registered in England and Wales,
> > company
> >     > > > > number
> >     > > > > >     > > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     > > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     > > > 195355)
> >     > > > > > and IG
> >     > > > > >     > > Index Limited (register number 114059) are
> > authorised and
> >     > > > > > regulated by
> >     > > > > >     > the
> >     > > > > >     > > Financial Conduct Authority.
> >     > > > > >     > >
> >     > > > > >     > >
> >     > > > > >     > The information contained in this email is strictly
> >     > > confidential
> >     > > > > and
> >     > > > > > for
> >     > > > > >     > the use of the addressee only, unless otherwise
> > indicated. If
> >     > > you
> >     > > > > > are not
> >     > > > > >     > the intended recipient, please do not read, copy, use
> > or
> >     > > disclose
> >     > > > > to
> >     > > > > > others
> >     > > > > >     > this message or any attachment. Please also notify
> the
> > sender
> >     > > by
> >     > > > > > replying
> >     > > > > >     > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > delete
> >     > > > > > the email
> >     > > > > >     > and any copies of it. Opinions, conclusion (etc) that
> > do not
> >     > > > relate
> >     > > > > > to the
> >     > > > > >     > official business of this company shall be understood
> > as
> >     > > neither
> >     > > > > > given nor
> >     > > > > >     > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> >     > > > > company
> >     > > > > >     > registered in England and Wales, company number
> > 04008957) and
> >     > > IG
> >     > > > > > Index
> >     > > > > >     > Limited (a company registered in England and Wales,
> > company
> >     > > > number
> >     > > > > >     > 01190902). Registered address at Cannon Bridge House,
> > 25
> >     > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     > > 195355)
> >     > > > > > and IG
> >     > > > > >     > Index Limited (register number 114059) are authorised
> > and
> >     > > > regulated
> >     > > > > > by the
> >     > > > > >     > Financial Conduct Authority.
> >     > > > > >     >
> >     > > > > >     >
> >     > > > > >
> >     > > > > >
> >     > > > > > The information contained in this email is strictly
> > confidential
> >     > and
> >     > > > for
> >     > > > > > the use of the addressee only, unless otherwise indicated.
> > If you
> >     > are
> >     > > > not
> >     > > > > > the intended recipient, please do not read, copy, use or
> > disclose
> >     > to
> >     > > > > others
> >     > > > > > this message or any attachment. Please also notify the
> > sender by
> >     > > > replying
> >     > > > > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then delete the
> >     > > > > email
> >     > > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> >     > > to
> >     > > > > the
> >     > > > > > official business of this company shall be understood as
> > neither
> >     > > given
> >     > > > > nor
> >     > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> >     > company
> >     > > > > > registered in England and Wales, company number 04008957)
> > and IG
> >     > > Index
> >     > > > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > > Hill,
> >     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> >     > and
> >     > > > IG
> >     > > > > > Index Limited (register number 114059) are authorised and
> > regulated
> >     > > by
> >     > > > > the
> >     > > > > > Financial Conduct Authority.
> >     > > > > >
> >     > > > > The information contained in this email is strictly
> > confidential and
> >     > > for
> >     > > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > > not
> >     > > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > > others
> >     > > > > this message or any attachment. Please also notify the sender
> > by
> >     > > replying
> >     > > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the
> >     > > > email
> >     > > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> >     > to
> >     > > > the
> >     > > > > official business of this company shall be understood as
> > neither
> >     > given
> >     > > > nor
> >     > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > > registered in England and Wales, company number 04008957) and
> > IG
> >     > Index
> >     > > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > Hill,
> >     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > > IG
> >     > > > > Index Limited (register number 114059) are authorised and
> > regulated
> >     > by
> >     > > > the
> >     > > > > Financial Conduct Authority.
> >     > > > >
> >     > > > The information contained in this email is strictly
> confidential
> > and
> >     > for
> >     > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > not
> >     > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > others
> >     > > > this message or any attachment. Please also notify the sender
> by
> >     > replying
> >     > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the
> >     > > email
> >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > > the
> >     > > > official business of this company shall be understood as
> neither
> > given
> >     > > nor
> >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> >     > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > IG
> >     > > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > > the
> >     > > > Financial Conduct Authority.
> >     > > >
> >     > > The information contained in this email is strictly confidential
> > and for
> >     > > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > > the intended recipient, please do not read, copy, use or disclose
> > to
> >     > others
> >     > > this message or any attachment. Please also notify the sender by
> > replying
> >     > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the email
> >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > the
> >     > > official business of this company shall be understood as neither
> > given
> >     > nor
> >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > > Limited (a company registered in England and Wales, company
> number
> >     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > the
> >     > > Financial Conduct Authority.
> >     > >
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jay Kreps <ja...@confluent.io>]

Hey Michael,

Sorry if it's frustrating. Definitely not a Confluent thing, I think you've
seen both sides from different Confluent folks. I'm just trying to
understand the compatibility implications to get on board myself. Given
you're saying there is no compatibility implication I think maybe I'm
confused with how the proposal works. Let me see if I can go back through
an un-confuse myself.

-Jay

On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jay
>
> I disagree here that we are breaking any compatibility, we went through
> this on the discussion thread in fact with the help of that thread is how
> the kip came to the solution.
>
> Also on the supported combinations front you mention, we are not taking
> anything away afaik.
>
> Currently supported are only are:
> Null value = delete
> Non-null value = non delete
>
> With this kip we would support
> Null value + tombstone = delete
> Non null value + tombstone = delete
> Non null value + no tombstone = non delete
>
> As for the alternative idea, this is simply a new policy, how can there be
> confusion here? For this policy it would be explicit that tombstone marker
> would need to be set for a delete.
>
> I'm going to vent a little now as starting to get quite frustrated.
>
> We are going round in circles on kip-82 as per use cases there is now many
> use cases, how many more are needed? just because confluent don't see these
> doesn't mean they aren't real use cases other have, this is the point of
> the Apache foundation, it shouldn't be the view of just one organisation.
> It really is getting a feeling of the NIH syndrome. Rather than it being
> constructive on discussion of the implementation detail.
>
> kip-87 spawned from as on the kip call we all agreed this was needed. And
> would at least allow a custom wrapper be supported in a compacted topic,
> allowing meta data. Which again now I feel we are spinning wheels, and
> simply finding reasons not support it.
>
> Cheers
> Mike
>
>
>
> Sent using OWA for iPad
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Friday, December 16, 2016 7:09:23 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hey Michael,
>
> I do think it might have been better had we started with a separate concept
> of null vs delete. Given that we are where we are, I'm not sure that the
> possible cures we explored so far are better than the disease.
>
> I apologize for coming to this late, but I didn't really understand the
> implications of the KIP and that we'd be breaking compatibility with
> existing apps until the vote had begun, and in my defense I'm not sure the
> other folks voting did either.
>
> I think we all agree there are many existing apps that are built with the
> assumption of "null value non-tombstone" and it isn't possible to
> disambiguate these from tombstones on the producer. It isn't that anyone is
> saying we have to support all four possibilities at once, it's that we
> simply can't orphan one of the existing combinations or our users will eat
> us!
>
> If I've understood your alternate solution of adding another setting for
> compaction, I think this does fix the compatibility problem, but it adds an
> odd mode the user has to add on all their topics. While the current state
> is easily explainable, the resulting state where the meaning of tombstone
> and null are overlapping and ambiguous and dependent on a compaction
> setting that could change out of band or not be in sync with your code in
> some environment seems worse to me then where we currently are. I think the
> question is how would this combination be explained to users and does it
> make sense?
>
> -Jay
>
>
>
> On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Chaps,
> >
> > Can we either get one more +1 binding (we have 2 already) on the
> existing?
> >
> > Or have some response on the below possible alternative. We are keen to
> > get working on this, so we make next feature release.
> >
> > Cheers
> > Mike
> >
> >
> > On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> >     Hi Ismael
> >
> >     Did you see our email this morning, what's your thoughts on this
> > approach to instead we simply have a brand new policy?
> >
> >     Cheers
> >     Mike
> >
> >
> >     Sent using OWA for iPhone
> >     ________________________________________
> >     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
> Juma <
> > ismael@juma.me.uk>
> >     Sent: Tuesday, December 13, 2016 11:30:05 AM
> >     To: dev@kafka.apache.org
> >     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> >     Yes, this is actually tricky to do in a way where we both have the
> > desired
> >     semantics and maintain compatibility. When someone creates a
> >     `ProducerRecord` with a `null` value today, the producer doesn't know
> > if
> >     it's meant to be a tombstone or not. For V3 messages, it's easy when
> > the
> >     constructor that takes a tombstone is used. However, if any other
> >     constructor is used, it's not clear. A couple of options I can think
> > of,
> >     none of them particularly nice:
> >
> >     1. Have a third state where tombstone = unknown and the broker would
> > set
> >     the tombstone bit if the value was null and the topic was compacted.
> > People
> >     that wanted to pass a non-null value for the tombstone would have to
> > use
> >     the constructor that takes a tombstone. The drawbacks: third state
> for
> >     tombstone in message format, message conversion at the broker for a
> > common
> >     case.
> >
> >     2. Extend MetadataResponse to optionally include topic configs, which
> > would
> >     make it possible for the producer to be smarter about setting the
> >     tombstone. It would only do it if a tombstone was not passed
> > explicitly,
> >     the value was null and the topic was compacted. The main drawback is
> > that
> >     the producer would be getting a bit more data for each topic even
> > though it
> >     probably won't use it most of the time. Extending MetadataResponse to
> >     return topic configs would be useful for other reasons as well, so
> that
> >     part seems OK.
> >
> >     In addition, for both proposals, we could consider adding warnings to
> > the
> >     documentation that the behaviour of the constructors that don't take
> a
> >     tombstone would change in the next major release so that tombstone =
> > false.
> >     Not sure if this would be worth it though.
> >
> >     Ismael
> >
> >     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> > ewen@confluent.io>
> >     wrote:
> >
> >     > Michael,
> >     >
> >     > It kind of depends on how you want to interpret the tombstone flag.
> > If it's
> >     > purely a producer-facing Kafka-level thing that we treat as
> internal
> > to the
> >     > broker and log cleaner once the record is sent, then your approach
> > makes
> >     > sense. You're just moving copying the null-indicates-delete
> behavior
> > of the
> >     > old constructor into the tombstone flag.
> >     >
> >     > However, if you want this change to more generally decouple the
> idea
> > of
> >     > deletion and null values, then you are sometimes converting what
> > might be a
> >     > completely valid null value that doesn't indicate deletion into a
> >     > tombstone. Downstream applications could potentially handle these
> > cases
> >     > differently given the separation of deletion from value.
> >     >
> >     > I guess the question is if we want to try to support the latter
> even
> > for
> >     > topics where we have older produce requests. An example where this
> > could
> >     > come up is in something like a CDC Connector. If we try to support
> > the
> >     > semantic difference, a connector might write changes to Kafka using
> > the
> >     > tombstone flag to indicate when a row was truly deleted (vs an
> > update that
> >     > sets it to null but still present; this probably makes more sense
> > for CDC
> >     > from document stores or extracting single columns). There are
> various
> >     > reasons we might want to maintain the full log and not turn
> > compaction on
> >     > (or just use a time-based retention policy), but downstream
> > applications
> >     > might care to know the difference between a delete and a null
> value.
> > In
> >     > fact both versions of the same log (compacted and time-retention)
> > could be
> >     > useful and I don't think it'll be uncommon to maintain both or use
> > KIP-71
> >     > to maintain a hybrid compacted/retention topic.
> >     >
> >     > -Ewen
> >     >
> >     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > wrote:
> >     >
> >     > > Hi Jay,
> >     > >
> >     > > Why wouldn't that work, the tombstone value is only looked at by
> > the
> >     > > broker, on a topic configured for compaction as such is benign on
> > non
> >     > > compacted topics. This is as much as sending a null value
> currently
> >     > >
> >     > >
> >     > > Regards
> >     > > Mike
> >     > >
> >     > >
> >     > >
> >     > > Sent using OWA for iPhone
> >     > > ________________________________________
> >     > > From: Jay Kreps <ja...@confluent.io>
> >     > > Sent: Sunday, December 11, 2016 8:58:53 PM
> >     > > To: dev@kafka.apache.org
> >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > >
> >     > > Hey Michael,
> >     > >
> >     > > I'm not quite sure that works as that would translate ALL null
> > values to
> >     > > tombstones, even for non-compacted topics that use null as an
> > acceptable
> >     > > value sent by the producer and expected by the consumer.
> >     > >
> >     > > -Jay
> >     > >
> >     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > > wrote:
> >     > >
> >     > > > Hi Ewen,
> >     > > >
> >     > > > I think the easiest way to show this is with code.
> >     > > >
> >     > > > As you can see we keep the existing behaviour for code/binaries
> > calling
> >     > > > the pre-existing constructors, whereby if the value is null the
> >     > tombstone
> >     > > > is set to true.
> >     > > >
> >     > > > Regards
> >     > > > Mike
> >     > > >
> >     > > >
> >     > > >
> >     > > >     /**
> >     > > >      * Creates a record with a specified timestamp to be sent
> to
> > a
> >     > > > specified topic and partition
> >     > > >      *
> >     > > >      * @param topic The topic the record will be appended to
> >     > > >      * @param partition The partition to which the record
> should
> > be
> >     > sent
> >     > > >      * @param timestamp The timestamp of the record
> >     > > >      * @param tombstone if the record should be treated as a
> > tombstone
> >     > if
> >     > > > the topic is compacted
> >     > > >      * @param key The key that will be included in the record
> >     > > >      * @param value The record contents
> >     > > >      */
> >     > > >     public ProducerRecord(String topic, Integer partition,
> > Boolean
> >     > > > tombstone, Long timestamp, K key, V value) {
> >     > > >         if (topic == null)
> >     > > >             throw new IllegalArgumentException("Topic cannot
> be
> >     > null.");
> >     > > >         if (timestamp != null && timestamp < 0)
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Invalid timestamp: %d.
> > Timestamp
> >     > > should
> >     > > > always be non-negative or null.", timestamp));
> >     > > >         if (partition != null && partition < 0)
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Invalid partition: %d.
> > Partition
> >     > > number
> >     > > > should always be non-negative or null.", partition));
> >     > > >         if (!tombstone && value == null){
> >     > > >             throw new IllegalArgumentException(
> >     > > >                     String.format("Tombstone must be true if
> null
> >     > > value"));
> >     > > >         }
> >     > > >         this.topic = topic;
> >     > > >         this.partition = partition;
> >     > > >         this.tombstone = tombstone;
> >     > > >         this.key = key;
> >     > > >         this.value = value;
> >     > > >         this.timestamp = timestamp;
> >     > > >     }
> >     > > >
> >     > > >     public ProducerRecord(String topic, Integer partition, Long
> >     > > timestamp,
> >     > > > K key, V value) {
> >     > > >         this(topic, partition, value == null, timestamp, key,
> > value);
> >     > > >     }
> >     > > > ________________________________________
> >     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> >     > > > Sent: Sunday, December 11, 2016 5:45 AM
> >     > > > To: dev@kafka.apache.org
> >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > >
> >     > > > It seemed like this was addressed in the migration section,
> > wasn't it?
> >     > > The
> >     > > > V2 vs V3 requests and the fact that the broker will downgrade
> the
> >     > message
> >     > > > format (losing zero copy) if you issues a V2 request to a
> broker
> > using
> >     > V3
> >     > > > format handles compatibility, does it not? The existing
> > consumers won't
> >     > > see
> >     > > > the extra metadata in the value, but they will get a null
> > instead and
> >     > > treat
> >     > > > it as a tombstone. Certainly there is a performance impact, but
> > it
> >     > seemed
> >     > > > compatible.
> >     > > >
> >     > > > I'm worried about this though:
> >     > > >
> >     > > >
> >     > > >    - *NOTE *: With the new version of producer client using
> >     > > ProduceRequest
> >     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set)
> > and
> >     > null
> >     > > > value
> >     > > >    should not be allowed as the older version of consumer using
> >     > > > FetchRequest
> >     > > >    V2 will think of this as a tombstone when its actually not.
> >     > > >
> >     > > >
> >     > > > Unless I'm misunderstanding, this ends up breaking binary
> > compatibility
> >     > > for
> >     > > > the Java API. It sounds like this suggests that passing a null
> > value to
> >     > > the
> >     > > > existing ProducerRecord constructors would generate an
> exception
> > since
> >     > > you
> >     > > > didn't explicitly enable the tombstone (via whatever new
> > constructor is
> >     > > > provided). But that means you can't swap in a newer client jar
> > without
> >     > > > recompiling and get the same behavior if your app deletes keys
> > using
> >     > the
> >     > > > current approach because your app will start throwing
> > exceptions. Maybe
> >     > > > this is a tradeoff we're ok with, but we've tried pretty hard
> to
> > avoid
> >     > > > breaking compatibility like this so far -- it makes picking up
> > bug
> >     > fixes
> >     > > in
> >     > > > the clients harder for users of frameworks that have to pin to
> > earlier
> >     > > > default versions for compatibility.
> >     > > >
> >     > > > But then later the KIP says:
> >     > > >
> >     > > >
> >     > > >    - The old constructors for ProducerRecord without this
> > parameter
> >     > will
> >     > > be
> >     > > >    kept but updated so that their default behaviour if setting
> > null
> >     > value
> >     > > > will
> >     > > >    be the tombstone will be set to true to keep existing
> > behaviour.
> >     > > >
> >     > > >
> >     > > > So maybe I am misinterpreting something.
> >     > > >
> >     > > > And just a nit re: motivation section, item 6 would be clearer
> > for a
> >     > > union
> >     > > > schema with null (or optional schemas in other formats), e.g.
> > [null,
> >     > > > string], in which case losing the schema truly is losing
> > information
> >     > > > (whereas null is already the only valid value for a pure null
> > schema).
> >     > > >
> >     > > > -Ewen
> >     > > >
> >     > > >
> >     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> >     > >
> >     > > > wrote:
> >     > > >
> >     > > > > Hi Jay,
> >     > > > >
> >     > > > > Good point this detail is missing in the KIP write up. Ive
> > added this
> >     > > > now.
> >     > > > >
> >     > > > > Essentially simply just upgrading the clients we do not
> expect
> > any
> >     > > client
> >     > > > > app code change needed.
> >     > > > >
> >     > > > > Cheers
> >     > > > > Mike
> >     > > > > ________________________________________
> >     > > > > From: Jay Kreps <ja...@confluent.io>
> >     > > > > Sent: Saturday, December 10, 2016 10:51 PM
> >     > > > > To: dev@kafka.apache.org
> >     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > > >
> >     > > > > Michael,
> >     > > > >
> >     > > > > The compatibility section goes through the migration path,
> but
> > isn't
> >     > > the
> >     > > > > bigger compatibility issue with existing apps? There are many
> >     > (probably
> >     > > > > thousands) of apps in production that use this feature and
> > send null
> >     > to
> >     > > > > mean delete. It seems like this would break compatibility
> with
> > them,
> >     > > and
> >     > > > > they would have to be rewritten to right?
> >     > > > >
> >     > > > > -Jay
> >     > > > >
> >     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> >     > Michael.Pearce@ig.com
> >     > > >
> >     > > > > wrote:
> >     > > > >
> >     > > > > > Hi Jun,
> >     > > > > >
> >     > > > > > 4) On v3 we honour the tombstone. As such we expect it to
> be
> > set
> >     > > > > correctly
> >     > > > > > as per the KIP.
> >     > > > > >
> >     > > > > > 4.1) why would we want to produce an error when its v3?
> This
> > is the
> >     > > > exact
> >     > > > > > purpose to support non-null tombstone’s
> >     > > > > > 4.2) again here im unclear on the question on the v3,
> produce
> >     > request
> >     > > > we
> >     > > > > > expect the tombstone flag to be set correctly.
> >     > > > > >
> >     > > > > > 4.4) compaction only occurs on compacted topics, the bit
> > makes no
> >     > > > > > difference and not looked at on un-compacted (time/size
> based
> >     > > > eviction).
> >     > > > > >
> >     > > > > >
> >     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> >     > > > > >
> >     > > > > >     Hi, Michael,
> >     > > > > >
> >     > > > > >     4. Then, I think I misunderstood this point. Could you
> > document
> >     > > the
> >     > > > > >     following points in the wiki?
> >     > > > > >     4.1 If producer V3 sets tombstone, but provides a
> > non-null
> >     > value,
> >     > > > > does
> >     > > > > > the
> >     > > > > >     send() get an error or does the producer automatically
> > set the
> >     > > > value
> >     > > > > to
> >     > > > > >     null?
> >     > > > > >     4.2 If producer V3 doesn't set tombstone, but provides
> a
> > null
> >     > > > value,
> >     > > > > > does
> >     > > > > >     the send() get an error or does the producer
> > automatically sets
> >     > > the
> >     > > > > >     tombstone?
> >     > > > > >     4.3 Does the broker only expect messages that (a) have
> no
> >     > > tombstone
> >     > > > > and
> >     > > > > >     non-null value; (b) have tombstone and null value and
> > reject
> >     > the
> >     > > > > > messages
> >     > > > > >     with an error code otherwise?
> >     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
> > compacted
> >     > on
> >     > > > > not?
> >     > > > > >
> >     > > > > >     Thanks,
> >     > > > > >
> >     > > > > >     Jun
> >     > > > > >
> >     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> >     > > > > Michael.Pearce@ig.com
> >     > > > > > >
> >     > > > > >     wrote:
> >     > > > > >
> >     > > > > >     > Not at all.  This only acts on compacted topics just
> > as what
> >     > > > occurs
> >     > > > > > today
> >     > > > > >     >
> >     > > > > >     > Sent using OWA for iPhone
> >     > > > > >     > ________________________________________
> >     > > > > >     > From: Jun Rao <ju...@confluent.io>
> >     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> >     > > > > >     > To: dev@kafka.apache.org
> >     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> > Flag
> >     > > > > >     >
> >     > > > > >     > Hi, Michael,
> >     > > > > >     >
> >     > > > > >     > 4. Hmm, does that mean the new client library can
> > never send
> >     > a
> >     > > > null
> >     > > > > > message
> >     > > > > >     > even to a regular topic? This seems like a change of
> > the
> >     > > existing
> >     > > > > > behavior.
> >     > > > > >     >
> >     > > > > >     > Thanks,
> >     > > > > >     >
> >     > > > > >     > Jun
> >     > > > > >     >
> >     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> >     > > > > > Michael.Pearce@ig.com>
> >     > > > > >     > wrote:
> >     > > > > >     >
> >     > > > > >     > > Hi Jun,
> >     > > > > >     > >
> >     > > > > >     > > Re 4) That's because we expect the tombstone value
> > to be
> >     > set
> >     > > > > > correctly if
> >     > > > > >     > > message bit is 2, as such if an older client sends
> > in on
> >     > old
> >     > > > > > message the
> >     > > > > >     > > message is upcast and the bit is set correctly. And
> > such no
> >     > > > > longer
> >     > > > > > need
> >     > > > > >     > to
> >     > > > > >     > > check the value. Mayuresh can you confirm my
> > thinking and
> >     > > > > > understanding
> >     > > > > >     > of
> >     > > > > >     > > what we've discussed?
> >     > > > > >     > >
> >     > > > > >     > > The second point I understand what you're getting
> at
> > now my
> >     > > > > > apologies.
> >     > > > > >     > Yes
> >     > > > > >     > > this makes sense to save on touching the message,
> if
> > we're
> >     > > the
> >     > > > > > only kip
> >     > > > > >     > > going in, in this release.
> >     > > > > >     > >
> >     > > > > >     > > Cheers
> >     > > > > >     > > Mike
> >     > > > > >     > >
> >     > > > > >     > > Sent using OWA for iPhone
> >     > > > > >     > > ________________________________________
> >     > > > > >     > > From: Jun Rao <ju...@confluent.io>
> >     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> >     > > > > >     > > To: dev@kafka.apache.org
> >     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone Flag
> >     > > > > >     > >
> >     > > > > >     > > Hi, Michael,
> >     > > > > >     > >
> >     > > > > >     > > 4. Is this updated in the wiki? The text "If the
> > magic byte
> >     > > on
> >     > > > > > message is
> >     > > > > >     > > 2, the broker should use the tombstone bit for log
> >     > > compaction."
> >     > > > > > doesn't
> >     > > > > >     > > seem to have changed.
> >     > > > > >     > >
> >     > > > > >     > > 2. My point is that if we change the message format
> > just
> >     > for
> >     > > > this
> >     > > > > > KIP, we
> >     > > > > >     > > should consider whether it's worth optimizing the
> > down
> >     > > > conversion
> >     > > > > > path
> >     > > > > >     > > (i.e., decide whether a conversion is needed by
> just
> >     > looking
> >     > > at
> >     > > > > the
> >     > > > > >     > > tombstone bit in the wrapper message) since
> > tombstone will
> >     > be
> >     > > > > used
> >     > > > > >     > rarely.
> >     > > > > >     > > However, if the message format change here is
> > combined with
> >     > > > other
> >     > > > > > KIPs,
> >     > > > > >     > > then this optimization likely won't be needed. The
> > latter
> >     > > > > probably
> >     > > > > > makes
> >     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> > think?
> >     > > > > >     > >
> >     > > > > >     > > Other than those, +1 from me,
> >     > > > > >     > >
> >     > > > > >     > > Thanks,
> >     > > > > >     > >
> >     > > > > >     > > Jun
> >     > > > > >     > >
> >     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> >     > > > > > Michael.Pearce@ig.com>
> >     > > > > >     > > wrote:
> >     > > > > >     > >
> >     > > > > >     > > > Hi Jun
> >     > > > > >     > > >
> >     > > > > >     > > > do we have your vote on this now?
> >     > > > > >     > > >
> >     > > > > >     > > > Any other concerns?
> >     > > > > >     > > >
> >     > > > > >     > > > Cheers
> >     > > > > >     > > > Mike
> >     > > > > >     > > >
> >     > > > > >     > > > Sent using OWA for iPhone
> >     > > > > >     > > > ________________________________________
> >     > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> >     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> >     > > > > >     > > > To: dev@kafka.apache.org
> >     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> > Tombstone
> >     > Flag
> >     > > > > >     > > >
> >     > > > > >     > > > Hi Jun,
> >     > > > > >     > > >
> >     > > > > >     > > > Have updated. Thanks again for the feedback.
> >     > > > > >     > > >
> >     > > > > >     > > > Agree yes we should align up when it gets to
> that,
> > I
> >     > assume
> >     > > > > > you’ve
> >     > > > > >     > > flagged
> >     > > > > >     > > > the same in the other KIP?
> >     > > > > >     > > >
> >     > > > > >     > > > I think for now let’s get this KIP approved, then
> > we can
> >     > > > start
> >     > > > > > the work
> >     > > > > >     > > to
> >     > > > > >     > > > get an initial PR, then we can discuss how to
> > align the
> >     > two
> >     > > > if
> >     > > > > > needed.
> >     > > > > >     > > >
> >     > > > > >     > > > Cheers,
> >     > > > > >     > > > Mike
> >     > > > > >     > > >
> >     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
> jun@confluent.io>
> >     > wrote:
> >     > > > > >     > > >
> >     > > > > >     > > >     Hi, Michael,
> >     > > > > >     > > >
> >     > > > > >     > > >     For 2), this is fine. Could you update the
> KIP
> > wiki
> >     > to
> >     > > > make
> >     > > > > > this
> >     > > > > >     > and
> >     > > > > >     > > > other
> >     > > > > >     > > >     points clearer? Other than that, the KIP
> looks
> > good
> >     > to
> >     > > > me.
> >     > > > > >     > > >
> >     > > > > >     > > >     An orthogonal thing is that there are other
> > KIPs such
> >     > > as
> >     > > > > > KIP-98
> >     > > > > >     > that
> >     > > > > >     > > > also
> >     > > > > >     > > >     intend to change the message format. If they
> > all get
> >     > > > > > approved, we
> >     > > > > >     > > > should
> >     > > > > >     > > >     think about whether it's better to just bump
> > up the
> >     > > magic
> >     > > > > > byte once
> >     > > > > >     > > to
> >     > > > > >     > > >     incorporate multiple format changes like we
> > did in
> >     > > > > > KIP-31/KIP-32.
> >     > > > > >     > > >
> >     > > > > >     > > >     Thanks,
> >     > > > > >     > > >
> >     > > > > >     > > >     Jun
> >     > > > > >     > > >
> >     > > > > >     > > >
> >     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
> Pearce
> > <
> >     > > > > >     > > Michael.Pearce@ig.com>
> >     > > > > >     > > >     wrote:
> >     > > > > >     > > >
> >     > > > > >     > > >     > Hi Jao
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > Thanks for the response. Sorry for slow
> > reply, both
> >     > > > with
> >     > > > > > personal
> >     > > > > >     > > > sickness
> >     > > > > >     > > >     > and also battling some critical issues
> > encountered
> >     > > > since
> >     > > > > >     > upgrading
> >     > > > > >     > > to
> >     > > > > >     > > >     > 0.10.1.0
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > 1) Thans for spotting, Document error where
> > we
> >     > > branched
> >     > > > > > this KIP
> >     > > > > >     > > from
> >     > > > > >     > > >     > KIP-82, will get that removed.
> >     > > > > >     > > >     > 2) Intent is to do this just at the record
> > message
> >     > > > level.
> >     > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> >     > > corrected.
> >     > > > > >     > > >     > 4) As per discussion thread we will support
> >     > > tombstone +
> >     > > > > > null
> >     > > > > >     > value,
> >     > > > > >     > > >     > tombstone + non null value, no tombstone +
> > null
> >     > > value.
> >     > > > > >     > > >     > 5) I believe this was in the discussion
> > thread,
> >     > > > @Mayuresh
> >     > > > > > is this
> >     > > > > >     > > >     > something we’ve overlooked? I thought we
> > would down
> >     > > > > > convert and
> >     > > > > >     > > > remove the
> >     > > > > >     > > >     > value so the old consumer had existing
> > behavior, or
> >     > > is
> >     > > > > > there
> >     > > > > >     > > > something we
> >     > > > > >     > > >     > haven’t thought about?
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > Cheers
> >     > > > > >     > > >     > Mike
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> > jun@confluent.io>
> >     > > > > wrote:
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Hi, Michael,
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Thanks for the KIP. A few comments
> below.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     1. The message format change contains
> >     > > > "HeadersLength
> >     > > > > >     > Headers".
> >     > > > > >     > > > Is that
> >     > > > > >     > > >     >     intended?
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     2. For compressed messageset, is the
> > tombstone
> >     > > bit
> >     > > > > > only set
> >     > > > > >     > at
> >     > > > > >     > > > the
> >     > > > > >     > > >     > shallow
> >     > > > > >     > > >     >     level? Do we always leave that bit in
> the
> >     > wrapper
> >     > > > > > message
> >     > > > > >     > > unset?
> >     > > > > >     > > > An
> >     > > > > >     > > >     >     alternative is to set the tombstone bit
> > in the
> >     > > > > wrapper
> >     > > > > > if at
> >     > > > > >     > > > least one
> >     > > > > >     > > >     >     inner message has the tombstone bit
> set.
> > This
> >     > > makes
> >     > > > > > things a
> >     > > > > >     > > bit
> >     > > > > >     > > > more
> >     > > > > >     > > >     >     complicated, but we could potentially
> > exploit
> >     > > that
> >     > > > > for
> >     > > > > >     > > > optimizing down
> >     > > > > >     > > >     >     conversion. For example, we only need
> to
> >     > convert
> >     > > > > > messages
> >     > > > > >     > with
> >     > > > > >     > > > magic 2
> >     > > > > >     > > >     > to
> >     > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit
> > is set
> >     > > > > > (conversion is
> >     > > > > >     > > > always
> >     > > > > >     > > >     > needed
> >     > > > > >     > > >     >     from magic 2 to magic 0). Not sure if
> the
> >     > > > > optimization
> >     > > > > > is
> >     > > > > >     > worth
> >     > > > > >     > > > the
> >     > > > > >     > > >     >     complexity though.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     3. The referencing of the new version
> of
> >     > > > > >     > > > ProducerRequest/FetchRequest
> >     > > > > >     > > >     > is
> >     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> > convention
> >     > > > starts
> >     > > > > at
> >     > > > > >     > version
> >     > > > > >     > > > at 0, I
> >     > > > > >     > > >     >     think the new version would be 3.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     4. "If the magic byte on message is 2,
> > the
> >     > broker
> >     > > > > > should use
> >     > > > > >     > > the
> >     > > > > >     > > >     > tombstone
> >     > > > > >     > > >     >     bit for log compaction." What about
> null
> > value?
> >     > > My
> >     > > > > >     > > understanding
> >     > > > > >     > > > is
> >     > > > > >     > > >     > that
> >     > > > > >     > > >     >     null value will be treated the same as
> > setting
> >     > > the
> >     > > > > > tombstone
> >     > > > > >     > > bit.
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     5. For the migration path, it would be
> > useful
> >     > to
> >     > > > > > describe the
> >     > > > > >     > > > down
> >     > > > > >     > > >     >     conversion path to consumers (i.e.,
> > brokers on
> >     > > > > message
> >     > > > > > format
> >     > > > > >     > > > 0.10.2
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     consumers on older version).
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Thanks,
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     Jun
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
> Michael
> >     > Pearce <
> >     > > > > >     > > > Michael.Pearce@ig.com
> >     > > > > >     > > >     > >
> >     > > > > >     > > >     >     wrote:
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >     > Hi All,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > We have been discussing in the below
> > thread
> >     > and
> >     > > > > final
> >     > > > > >     > changes
> >     > > > > >     > > > have
> >     > > > > >     > > >     > been
> >     > > > > >     > > >     >     > made to the KIP wiki based on these
> >     > > discussions.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > We would now like to put to the vote
> > the
> >     > > > following
> >     > > > > > KIP:
> >     > > > > >     > > >     >     > https://cwiki.apache.org/
> >     > > > > > confluence/display/KAFKA/KIP-
> >     > > > > >     > 87+-+
> >     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > This kip is for having a distinct
> > compaction
> >     > > > > > attribute
> >     > > > > >     > > > “tombstone”
> >     > > > > >     > > >     > flag
> >     > > > > >     > > >     >     > instead of relying on null value,
> > allowing
> >     > > > non-null
> >     > > > > > value
> >     > > > > >     > > > delete
> >     > > > > >     > > >     > messages.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > Many thanks,
> >     > > > > >     > > >     >     > Michael
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
> Pearce"
> > <
> >     > > > > >     > > Michael.Pearce@ig.com>
> >     > > > > >     > > >     > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Hi Mayuresh,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     LGTM. Ive just made one small
> > adjustment
> >     > > > > > updating the
> >     > > > > >     > > wire
> >     > > > > >     > > >     > protocol to
> >     > > > > >     > > >     >     > show the magic byte bump.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Do we think we’re good to put to
> a
> > vote?
> >     > Is
> >     > > > > > there any
> >     > > > > >     > > > other bits
> >     > > > > >     > > >     >     > needing discussion?
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     Cheers
> >     > > > > >     > > >     >     >     Mike
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
> > Gharat" <
> >     > > > > >     > > >     > gharatmayuresh15@gmail.com>
> >     > > > > >     > > >     >     > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Hi Michael,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         I have updated the migration
> > section
> >     > of
> >     > > > the
> >     > > > > > KIP.
> >     > > > > >     > Can
> >     > > > > >     > > > you
> >     > > > > >     > > >     > please
> >     > > > > >     > > >     >     > take a look?
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Thanks,
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         Mayuresh
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07
> > AM,
> >     > > Mayuresh
> >     > > > > > Gharat <
> >     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         > wrote:
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         > Hi Michael,
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > That whilst sending
> > tombstone and
> >     > non
> >     > > > > null
> >     > > > > > value,
> >     > > > > >     > > the
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     > can expect
> >     > > > > >     > > >     >     >         > only to receive the
> non-null
> >     > message
> >     > > > only
> >     > > > > > in step
> >     > > > > >     > > > (3) is
> >     > > > > >     > > >     > this
> >     > > > > >     > > >     >     > correct?
> >     > > > > >     > > >     >     >         > ---> I do agree with you
> > here.
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Becket, Ismael : can you
> guys
> >     > review
> >     > > > the
> >     > > > > >     > migration
> >     > > > > >     > > > plan
> >     > > > > >     > > >     > listed
> >     > > > > >     > > >     >     > above using
> >     > > > > >     > > >     >     >         > magic byte?
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Thanks,
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > Mayuresh
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at
> 8:58
> > AM,
> >     > > > Michael
> >     > > > > > Pearce <
> >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> >     > > > > >     > > >     >     >         > wrote:
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >> Many thanks for this
> > Mayuresh. I
> >     > > don't
> >     > > > > > have any
> >     > > > > >     > > >     > objections.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> I assume we should state:
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> That whilst sending
> > tombstone and
> >     > > non
> >     > > > > null
> >     > > > > >     > value,
> >     > > > > >     > > > the
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     > can expect
> >     > > > > >     > > >     >     >         >> only to receive the
> non-null
> >     > message
> >     > > > > only
> >     > > > > > in
> >     > > > > >     > step
> >     > > > > >     > > > (3) is
> >     > > > > >     > > >     > this
> >     > > > > >     > > >     >     > correct?
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Cheers
> >     > > > > >     > > >     >     >         >> Mike
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Sent using OWA for iPhone
> >     > > > > >     > > >     >     >         >>
> > ______________________________
> >     > > > > __________
> >     > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
> >     > > > > >     > gharatmayuresh15@gmail.com
> >     > > > > >     > > >
> >     > > > > >     > > >     >     >         >> Sent: Thursday, November
> > 17, 2016
> >     > > > > 5:18:41
> >     > > > > > PM
> >     > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
> >     > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> > KIP-87 -
> >     > Add
> >     > > > > > Compaction
> >     > > > > >     > > > Tombstone
> >     > > > > >     > > >     > Flag
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Hi Ismael,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Thanks for the
> explanation.
> >     > > > > >     > > >     >     >         >> Specially I like this part
> > where
> >     > in
> >     > > > you
> >     > > > > >     > mentioned
> >     > > > > >     > > > we can
> >     > > > > >     > > >     > get
> >     > > > > >     > > >     >     > rid of the
> >     > > > > >     > > >     >     >         >> older null value support
> > for log
> >     > > > > > compaction
> >     > > > > >     > later
> >     > > > > >     > > > on,
> >     > > > > >     > > >     > here :
> >     > > > > >     > > >     >     >         >> We can't change semantics
> > of the
> >     > > > message
> >     > > > > > format
> >     > > > > >     > > > without
> >     > > > > >     > > >     > having
> >     > > > > >     > > >     >     > a long
> >     > > > > >     > > >     >     >         >> transition period. And we
> > can't
> >     > rely
> >     > > > > >     > > >     >     >         >> on people reading
> > documentation or
> >     > > > > acting
> >     > > > > > on a
> >     > > > > >     > > > warning for
> >     > > > > >     > > >     >     > something so
> >     > > > > >     > > >     >     >         >> fundamental. As such, my
> > take is
> >     > > that
> >     > > > we
> >     > > > > > need to
> >     > > > > >     > > > bump the
> >     > > > > >     > > >     > magic
> >     > > > > >     > > >     >     > byte. The
> >     > > > > >     > > >     >     >         >> good news is
> >     > > > > >     > > >     >     >         >> that we don't have to
> > support all
> >     > > > > versions
> >     > > > > >     > > forever.
> >     > > > > >     > > > We
> >     > > > > >     > > >     > have
> >     > > > > >     > > >     >     > said that we
> >     > > > > >     > > >     >     >         >> will support direct
> > upgrades for 2
> >     > > > > years.
> >     > > > > > That
> >     > > > > >     > > > means that
> >     > > > > >     > > >     >     > message format
> >     > > > > >     > > >     >     >         >> version n could, in
> theory,
> > be
> >     > > > removed 2
> >     > > > > > years
> >     > > > > >     > > > after the
> >     > > > > >     > > >     > it's
> >     > > > > >     > > >     >     > introduced.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Just a heads up, I would
> > like to
> >     > > > mention
> >     > > > > > that
> >     > > > > >     > even
> >     > > > > >     > > > without
> >     > > > > >     > > >     >     > bumping magic
> >     > > > > >     > > >     >     >         >> byte, we will *NOT* loose
> > zero
> >     > copy
> >     > > as
> >     > > > > in
> >     > > > > > the
> >     > > > > >     > > > client(x+1)
> >     > > > > >     > > >     > in my
> >     > > > > >     > > >     >     >         >> explanation
> >     > > > > >     > > >     >     >         >> above will convert
> > internally a
> >     > null
> >     > > > > > value to
> >     > > > > >     > > have a
> >     > > > > >     > > >     > tombstone
> >     > > > > >     > > >     >     > bit set and
> >     > > > > >     > > >     >     >         >> a tombstone bit set to
> have
> > a null
> >     > > > value
> >     > > > > >     > > > automatically
> >     > > > > >     > > >     >     > internally and by
> >     > > > > >     > > >     >     >         >> the time we move to
> version
> > (x+2),
> >     > > the
> >     > > > > > clients
> >     > > > > >     > > > would have
> >     > > > > >     > > >     >     > upgraded.
> >     > > > > >     > > >     >     >         >> Obviously if we support a
> > request
> >     > > from
> >     > > > > >     > > consumer(x),
> >     > > > > >     > > > we
> >     > > > > >     > > >     > will
> >     > > > > >     > > >     >     > loose zero
> >     > > > > >     > > >     >     >         >> copy
> >     > > > > >     > > >     >     >         >> but that is the same case
> > with
> >     > magic
> >     > > > > byte.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> But if magic byte bump
> > makes life
> >     > > > easier
> >     > > > > > for
> >     > > > > >     > > > transition
> >     > > > > >     > > >     > for the
> >     > > > > >     > > >     >     > above
> >     > > > > >     > > >     >     >         >> reasons that you
> explained,
> > I am
> >     > OK
> >     > > > with
> >     > > > > > it
> >     > > > > >     > since
> >     > > > > >     > > > we are
> >     > > > > >     > > >     > going
> >     > > > > >     > > >     >     > to meet the
> >     > > > > >     > > >     >     >         >> end goal down the road :)
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> On a side note can we
> > update the
> >     > doc
> >     > > > > here
> >     > > > > > on
> >     > > > > >     > magic
> >     > > > > >     > > > byte
> >     > > > > >     > > >     > to say
> >     > > > > >     > > >     >     > that "*it
> >     > > > > >     > > >     >     >         >> should be bumped whenever
> > the
> >     > > message
> >     > > > > > format is
> >     > > > > >     > > > changed
> >     > > > > >     > > >     > or the
> >     > > > > >     > > >     >     >         >> interpretation of message
> > format
> >     > > > (usage
> >     > > > > > of the
> >     > > > > >     > > > reserved
> >     > > > > >     > > >     > bits as
> >     > > > > >     > > >     >     > well) is
> >     > > > > >     > > >     >     >         >> changed*".
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Hi Michael,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Here is the update plan
> > that we
> >     > > > > discussed
> >     > > > > >     > offline
> >     > > > > >     > > >     > yesterday :
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Currently the magic-byte
> > which
> >     > > > > > corresponds to
> >     > > > > >     > the
> >     > > > > >     > > >     >     > "message.format.version"
> >     > > > > >     > > >     >     >         >> is set to 1.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 1) On broker it will be
> set
> > to 1
> >     > > > > > initially.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 2) When a producer client
> > sends a
> >     > > > > message
> >     > > > > > with
> >     > > > > >     > > > magic-byte
> >     > > > > >     > > >     > = 2,
> >     > > > > >     > > >     >     > since the
> >     > > > > >     > > >     >     >         >> broker is on magic-byte =
> > 1, we
> >     > will
> >     > > > > down
> >     > > > > >     > convert
> >     > > > > >     > > > it,
> >     > > > > >     > > >     > which
> >     > > > > >     > > >     >     > means if the
> >     > > > > >     > > >     >     >         >> tombstone bit is set, the
> > value
> >     > will
> >     > > > be
> >     > > > > > set to
> >     > > > > >     > > > null. A
> >     > > > > >     > > >     > consumer
> >     > > > > >     > > >     >     >         >> understanding magic-byte =
> > 1, will
> >     > > > still
> >     > > > > > work
> >     > > > > >     > with
> >     > > > > >     > > > this. A
> >     > > > > >     > > >     >     > consumer
> >     > > > > >     > > >     >     >         >> working
> >     > > > > >     > > >     >     >         >> with magic-byte =2 will
> > also be
> >     > able
> >     > > > to
> >     > > > > >     > understand
> >     > > > > >     > > > this,
> >     > > > > >     > > >     > since
> >     > > > > >     > > >     >     > it
> >     > > > > >     > > >     >     >         >> understands the tombstone.
> >     > > > > >     > > >     >     >         >> Now there is still the
> > question of
> >     > > > > > supporting a
> >     > > > > >     > > >     > non-tombstone
> >     > > > > >     > > >     >     > and null
> >     > > > > >     > > >     >     >         >> value from producer client
> > with
> >     > > > > > magic-byte = 2.*
> >     > > > > >     > > (I
> >     > > > > >     > > > am
> >     > > > > >     > > >     > not sure
> >     > > > > >     > > >     >     > if we
> >     > > > > >     > > >     >     >         >> should support this.
> > Ismael/Becket
> >     > > can
> >     > > > > > comment
> >     > > > > >     > > > here)*
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> 3) When almost all the
> > clients
> >     > have
> >     > > > > > upgraded,
> >     > > > > >     > the
> >     > > > > >     > > >     >     > message.format.version
> >     > > > > >     > > >     >     >         >> on
> >     > > > > >     > > >     >     >         >> the broker can be changed
> > to 2,
> >     > > where
> >     > > > in
> >     > > > > > the
> >     > > > > >     > down
> >     > > > > >     > > >     > conversion in
> >     > > > > >     > > >     >     > the above
> >     > > > > >     > > >     >     >         >> step will not happen. If
> at
> > this
> >     > > point
> >     > > > > we
> >     > > > > > get a
> >     > > > > >     > > > consumer
> >     > > > > >     > > >     >     > request from a
> >     > > > > >     > > >     >     >         >> older consumer, we might
> > have to
> >     > > down
> >     > > > > > convert
> >     > > > > >     > > where
> >     > > > > >     > > > in we
> >     > > > > >     > > >     > loose
> >     > > > > >     > > >     >     > zero copy,
> >     > > > > >     > > >     >     >         >> but these cases should be
> > rare.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Becket can you review this
> > plan
> >     > and
> >     > > > add
> >     > > > > > more
> >     > > > > >     > > > details if I
> >     > > > > >     > > >     > have
> >     > > > > >     > > >     >     >         >> missed/wronged something,
> > before
> >     > we
> >     > > > put
> >     > > > > > it on
> >     > > > > >     > KIP.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Thanks,
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> Mayuresh
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> > 11:07 PM,
> >     > > > > Michael
> >     > > > > >     > Pearce <
> >     > > > > >     > > >     >     > Michael.Pearce@ig.com>
> >     > > > > >     > > >     >     >         >> wrote:
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> > Thanks guys, for
> > discussing this
> >     > > > > > offline and
> >     > > > > >     > > > getting
> >     > > > > >     > > >     > some
> >     > > > > >     > > >     >     > consensus.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > So its clear for myself
> > and
> >     > others
> >     > > > > what
> >     > > > > > is
> >     > > > > >     > > > proposed now
> >     > > > > >     > > >     > (i
> >     > > > > >     > > >     >     > think i
> >     > > > > >     > > >     >     >         >> > understand, but want to
> > make
> >     > sure)
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Could i ask either
> > directly
> >     > update
> >     > > > the
> >     > > > > > kip to
> >     > > > > >     > > > detail the
> >     > > > > >     > > >     >     > migration
> >     > > > > >     > > >     >     >         >> > strategy, or (re-)state
> > your
> >     > > offline
> >     > > > > > discussed
> >     > > > > >     > > and
> >     > > > > >     > > >     > agreed
> >     > > > > >     > > >     >     > migration
> >     > > > > >     > > >     >     >         >> > strategy based on a
> magic
> > byte
> >     > is
> >     > > in
> >     > > > > > this
> >     > > > > >     > > thread.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > The main original driver
> > for the
> >     > > KIP
> >     > > > > > was to
> >     > > > > >     > > > support
> >     > > > > >     > > >     >     > compaction where
> >     > > > > >     > > >     >     >         >> value
> >     > > > > >     > > >     >     >         >> > isn't null, based off
> the
> >     > > > discussions
> >     > > > > on
> >     > > > > >     > KIP-82
> >     > > > > >     > > > thread.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > We should be able to
> > support
> >     > > > > > non-tombstone +
> >     > > > > >     > > null
> >     > > > > >     > > > value
> >     > > > > >     > > >     > by the
> >     > > > > >     > > >     >     >         >> completion
> >     > > > > >     > > >     >     >         >> > of the KIP, as we noted
> > when
> >     > > > > discussing
> >     > > > > > this
> >     > > > > >     > > kip,
> >     > > > > >     > > > having
> >     > > > > >     > > >     >     > logic based on
> >     > > > > >     > > >     >     >         >> a
> >     > > > > >     > > >     >     >         >> > null value isn't very
> > clean and
> >     > > also
> >     > > > > > separates
> >     > > > > >     > > the
> >     > > > > >     > > >     > concerns.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > As discussed already
> > though we
> >     > can
> >     > > > > > split this
> >     > > > > >     > > into
> >     > > > > >     > > >     > KIP-87a
> >     > > > > >     > > >     >     > and KIP-87b
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Where we look to deliver
> > KIP-87a
> >     > > on
> >     > > > a
> >     > > > > >     > compacted
> >     > > > > >     > > > topic
> >     > > > > >     > > >     > (to
> >     > > > > >     > > >     >     > address the
> >     > > > > >     > > >     >     >         >> > immediate issues)
> >     > > > > >     > > >     >     >         >> > * tombstone + null value
> >     > > > > >     > > >     >     >         >> > * tombstone + non-null
> > value
> >     > > > > >     > > >     >     >         >> > * non-tombstone +
> > non-null value
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Then we can discuss once
> > KIP-87a
> >     > > is
> >     > > > > > completed
> >     > > > > >     > > > options
> >     > > > > >     > > >     > later
> >     > > > > >     > > >     >     > and how we
> >     > > > > >     > > >     >     >         >> > support the second part
> > KIP-87b
> >     > to
> >     > > > > > deliver:
> >     > > > > >     > > >     >     >         >> > * non-tombstone + null
> > value
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Cheers
> >     > > > > >     > > >     >     >         >> > Mike
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> > ______________________________
> >     > > > > > __________
> >     > > > > >     > > >     >     >         >> > From: Becket Qin <
> >     > > > > becket.qin@gmail.com>
> >     > > > > >     > > >     >     >         >> > Sent: Thursday, November
> > 17,
> >     > 2016
> >     > > > 1:43
> >     > > > > > AM
> >     > > > > >     > > >     >     >         >> > To:
> dev@kafka.apache.org
> >     > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> > KIP-87 -
> >     > > Add
> >     > > > > > Compaction
> >     > > > > >     > > >     > Tombstone Flag
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had
> > an
> >     > > offline
> >     > > > > >     > discussion,
> >     > > > > >     > > > and
> >     > > > > >     > > >     > following
> >     > > > > >     > > >     >     > is a brief
> >     > > > > >     > > >     >     >         >> > summary.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > 1. We agreed that not
> > bumping up
> >     > > > magic
> >     > > > > > value
> >     > > > > >     > may
> >     > > > > >     > > > result
> >     > > > > >     > > >     > in
> >     > > > > >     > > >     >     > losing zero
> >     > > > > >     > > >     >     >         >> copy
> >     > > > > >     > > >     >     >         >> > during migration.
> >     > > > > >     > > >     >     >         >> > 2. Given that bumping up
> > magic
> >     > > value
> >     > > > > is
> >     > > > > > almost
> >     > > > > >     > > > free and
> >     > > > > >     > > >     > has
> >     > > > > >     > > >     >     > benefit of
> >     > > > > >     > > >     >     >         >> > avoiding potential
> > performance
> >     > > > issue.
> >     > > > > > It is
> >     > > > > >     > > > probably
> >     > > > > >     > > >     > worth
> >     > > > > >     > > >     >     > doing.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > One issue we still need
> > to think
> >     > > > about
> >     > > > > > is
> >     > > > > >     > > whether
> >     > > > > >     > > > we
> >     > > > > >     > > >     > want to
> >     > > > > >     > > >     >     > support a
> >     > > > > >     > > >     >     >         >> > non-tombstone message
> > with null
> >     > > > value.
> >     > > > > >     > > >     >     >         >> > Currently it is not
> > supported by
> >     > > > > Kafka.
> >     > > > > > If we
> >     > > > > >     > > > allow a
> >     > > > > >     > > >     >     > non-tombstone null
> >     > > > > >     > > >     >     >         >> > value message to exist
> > after
> >     > > KIP-87.
> >     > > > > The
> >     > > > > >     > problem
> >     > > > > >     > > > is
> >     > > > > >     > > >     > that such
> >     > > > > >     > > >     >     > message
> >     > > > > >     > > >     >     >         >> will
> >     > > > > >     > > >     >     >         >> > not be supported by the
> >     > consumers
> >     > > > > prior
> >     > > > > > to
> >     > > > > >     > > KIP-87.
> >     > > > > >     > > >     > Because a
> >     > > > > >     > > >     >     > null value
> >     > > > > >     > > >     >     >         >> > will always be
> > interpreted to a
> >     > > > > > tombstone.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > One option is that we
> > keep the
> >     > > > current
> >     > > > > > way,
> >     > > > > >     > i.e.
> >     > > > > >     > > > do not
> >     > > > > >     > > >     >     > support such
> >     > > > > >     > > >     >     >         >> > message. It would be
> good
> > to
> >     > know
> >     > > if
> >     > > > > > there is
> >     > > > > >     > a
> >     > > > > >     > > >     > concrete use
> >     > > > > >     > > >     >     > case for
> >     > > > > >     > > >     >     >         >> such
> >     > > > > >     > > >     >     >         >> > message. If there is
> not,
> > we can
> >     > > > > > probably just
> >     > > > > >     > > not
> >     > > > > >     > > >     > support it.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > Thanks,
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at
> > 1:28 PM,
> >     > > > > > Mayuresh
> >     > > > > >     > > Gharat <
> >     > > > > >     > > >     >     >         >> >
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > wrote:
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >> > > Hi Ismael,
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > This is something I
> can
> > think
> >     > of
> >     > > > for
> >     > > > > >     > migration
> >     > > > > >     > > > plan:
> >     > > > > >     > > >     >     >         >> > > So the migration plan
> > can look
> >     > > > > > something
> >     > > > > >     > like
> >     > > > > >     > > > this,
> >     > > > > >     > > >     > with up
> >     > > > > >     > > >     >     >         >> conversion :
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > 1) Currently lets say
> > we have
> >     > > > Broker
> >     > > > > > at
> >     > > > > >     > > version
> >     > > > > >     > > > x.
> >     > > > > >     > > >     >     >         >> > > 2) Currently we have
> > clients
> >     > at
> >     > > > > > version x.
> >     > > > > >     > > >     >     >         >> > > 3) a) We move the
> > version to
> >     > > > > > Broker(x+1) :
> >     > > > > >     > > > supports
> >     > > > > >     > > >     > both
> >     > > > > >     > > >     >     > tombstone and
> >     > > > > >     > > >     >     >         >> > null
> >     > > > > >     > > >     >     >         >> > > for log compaction.
> >     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> > client
> >     > to
> >     > > > > > version
> >     > > > > >     > > > client(x+1) :
> >     > > > > >     > > >     > if in
> >     > > > > >     > > >     >     > the
> >     > > > > >     > > >     >     >         >> producer
> >     > > > > >     > > >     >     >         >> > > client(x+1) the value
> > is set
> >     > to
> >     > > > > null,
> >     > > > > > we
> >     > > > > >     > will
> >     > > > > >     > > >     > automatically
> >     > > > > >     > > >     >     > set the
> >     > > > > >     > > >     >     >         >> > > Tombstone bit
> > internally. If
> >     > the
> >     > > > > > producer
> >     > > > > >     > > > client(x+1)
> >     > > > > >     > > >     > sets
> >     > > > > >     > > >     >     > the
> >     > > > > >     > > >     >     >         >> tombstone
> >     > > > > >     > > >     >     >         >> > > itself, well and good.
> > For
> >     > > > producer
> >     > > > > >     > client(x),
> >     > > > > >     > > > the
> >     > > > > >     > > >     > broker
> >     > > > > >     > > >     >     > will up
> >     > > > > >     > > >     >     >         >> convert
> >     > > > > >     > > >     >     >         >> > > to have the tombstone
> > bit.
> >     > > > > > Broker(x+1) is
> >     > > > > >     > > > supporting
> >     > > > > >     > > >     > both.
> >     > > > > >     > > >     >     > Consumer
> >     > > > > >     > > >     >     >         >> > > client(x+1) will be
> > aware of
> >     > > this
> >     > > > > and
> >     > > > > > should
> >     > > > > >     > > be
> >     > > > > >     > > > able
> >     > > > > >     > > >     > to
> >     > > > > >     > > >     >     > handle this.
> >     > > > > >     > > >     >     >         >> For
> >     > > > > >     > > >     >     >         >> > > consumer client(x) we
> > will
> >     > down
> >     > > > > > convert the
> >     > > > > >     > > > message
> >     > > > > >     > > >     > on the
> >     > > > > >     > > >     >     > broker
> >     > > > > >     > > >     >     >         >> side.
> >     > > > > >     > > >     >     >         >> > >     c) At this point
> we
> > will
> >     > > have
> >     > > > to
> >     > > > > >     > specify a
> >     > > > > >     > > >     > warning or
> >     > > > > >     > > >     >     > clearly
> >     > > > > >     > > >     >     >         >> specify
> >     > > > > >     > > >     >     >         >> > > in docs that this
> > behavior is
> >     > > > about
> >     > > > > > to be
> >     > > > > >     > > > changed for
> >     > > > > >     > > >     > log
> >     > > > > >     > > >     >     > compaction.
> >     > > > > >     > > >     >     >         >> > > 4) a) In next release
> > of the
> >     > > > > > Broker(x+2), we
> >     > > > > >     > > > say that
> >     > > > > >     > > >     > only
> >     > > > > >     > > >     >     > Tombstone
> >     > > > > >     > > >     >     >         >> is
> >     > > > > >     > > >     >     >         >> > > used for log
> compaction
> > on the
> >     > > > > Broker
> >     > > > > > side.
> >     > > > > >     > > >     > Clients(x+1)
> >     > > > > >     > > >     >     > still is
> >     > > > > >     > > >     >     >         >> > > supported.
> >     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> > client
> >     > to
> >     > > > > > version
> >     > > > > >     > > > client(x+2) :
> >     > > > > >     > > >     > if
> >     > > > > >     > > >     >     > value is set
> >     > > > > >     > > >     >     >         >> to
> >     > > > > >     > > >     >     >         >> > > null, tombstone will
> > not be
> >     > set
> >     > > > > >     > automatically.
> >     > > > > >     > > > The
> >     > > > > >     > > >     > client
> >     > > > > >     > > >     >     > will have to
> >     > > > > >     > > >     >     >         >> > call
> >     > > > > >     > > >     >     >         >> > > setTombstone() to
> > actually set
> >     > > the
> >     > > > > >     > tombstone.
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > We should compare this
> >     > migration
> >     > > > > plan
> >     > > > > > with
> >     > > > > >     > the
> >     > > > > >     > > >     > migration
> >     > > > > >     > > >     >     > plan for
> >     > > > > >     > > >     >     >         >> magic
> >     > > > > >     > > >     >     >         >> > > byte bump and do
> > whatever
> >     > looks
> >     > > > > good.
> >     > > > > >     > > >     >     >         >> > > I am just worried that
> > if we
> >     > go
> >     > > > down
> >     > > > > > magic
> >     > > > > >     > > byte
> >     > > > > >     > > > route,
> >     > > > > >     > > >     >     > unless I am
> >     > > > > >     > > >     >     >         >> > missing
> >     > > > > >     > > >     >     >         >> > > something, it sounds
> > like
> >     > kafka
> >     > > > will
> >     > > > > > be
> >     > > > > >     > stuck
> >     > > > > >     > > > with
> >     > > > > >     > > >     >     > supporting both
> >     > > > > >     > > >     >     >         >> null
> >     > > > > >     > > >     >     >         >> > > value and tombstone
> bit
> > for
> >     > log
> >     > > > > > compaction
> >     > > > > >     > for
> >     > > > > >     > > > life
> >     > > > > >     > > >     > long,
> >     > > > > >     > > >     >     > which does
> >     > > > > >     > > >     >     >         >> not
> >     > > > > >     > > >     >     >         >> > > look like a good end
> > state.
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > Thanks,
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > Mayuresh
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016
> at
> > 9:32
> >     > AM,
> >     > > > > > Mayuresh
> >     > > > > >     > > > Gharat <
> >     > > > > >     > > >     >     >         >> > >
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > > wrote:
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > > Hi Ismael,
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > That's a very good
> > point
> >     > > which I
> >     > > > > > might
> >     > > > > >     > have
> >     > > > > >     > > > not
> >     > > > > >     > > >     >     > considered earlier.
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Here is a plan that
> I
> > can
> >     > > think
> >     > > > > of:
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Stage 1) The broker
> > from now
> >     > > on,
> >     > > > > up
> >     > > > > >     > converts
> >     > > > > >     > > > the
> >     > > > > >     > > >     > message
> >     > > > > >     > > >     >     > to have the
> >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> The
> > log
> >     > > > > compaction
> >     > > > > >     > thread
> >     > > > > >     > > > does log
> >     > > > > >     > > >     >     > compaction
> >     > > > > >     > > >     >     >         >> based
> >     > > > > >     > > >     >     >         >> > on
> >     > > > > >     > > >     >     >         >> > > > both null and
> > tombstone
> >     > > marker.
> >     > > > > > This is
> >     > > > > >     > our
> >     > > > > >     > > >     > transition
> >     > > > > >     > > >     >     > period.
> >     > > > > >     > > >     >     >         >> > > > Stage 2) The next
> > release we
> >     > > > only
> >     > > > > > say that
> >     > > > > >     > > log
> >     > > > > >     > > >     > compaction
> >     > > > > >     > > >     >     > is based
> >     > > > > >     > > >     >     >         >> on
> >     > > > > >     > > >     >     >         >> > > > tombstone marker.
> > (Open
> >     > source
> >     > > > > > kafka makes
> >     > > > > >     > > > this as a
> >     > > > > >     > > >     >     > policy). By
> >     > > > > >     > > >     >     >         >> this
> >     > > > > >     > > >     >     >         >> > > time,
> >     > > > > >     > > >     >     >         >> > > > the organization
> > which is
> >     > > moving
> >     > > > > to
> >     > > > > > this
> >     > > > > >     > > > release
> >     > > > > >     > > >     > will be
> >     > > > > >     > > >     >     > sure that
> >     > > > > >     > > >     >     >         >> they
> >     > > > > >     > > >     >     >         >> > > > have gone through
> the
> > entire
> >     > > > > > transition
> >     > > > > >     > > > period.
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > My only goal of
> doing
> > this
> >     > is
> >     > > > that
> >     > > > > > Kafka
> >     > > > > >     > > > clearly
> >     > > > > >     > > >     >     > specifies the end
> >     > > > > >     > > >     >     >         >> > state
> >     > > > > >     > > >     >     >         >> > > > about what log
> > compaction
> >     > > means
> >     > > > > (is
> >     > > > > > it
> >     > > > > >     > null
> >     > > > > >     > > > value
> >     > > > > >     > > >     > or a
> >     > > > > >     > > >     >     > tombstone
> >     > > > > >     > > >     >     >         >> > marker,
> >     > > > > >     > > >     >     >         >> > > > but not both).
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > What do you think?
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Thanks,
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > Mayuresh
> >     > > > > >     > > >     >     >         >> > > > .
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016
> > at 9:17
> >     > > AM,
> >     > > > > > Ismael
> >     > > > > >     > > Juma <
> >     > > > > >     > > >     >     > ismael@juma.me.uk>
> >     > > > > >     > > >     >     >         >> > wrote:
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >> One comment below.
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16,
> 2016
> > at
> >     > 5:08
> >     > > > PM,
> >     > > > > > Mayuresh
> >     > > > > >     > > > Gharat <
> >     > > > > >     > > >     >     >         >> > > >>
> > gharatmayuresh15@gmail.com
> >     > > > > >     > > >     >     >         >> > > >> > wrote:
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> >    - If we don't
> > bump up
> >     > > the
> >     > > > > > magic
> >     > > > > >     > byte,
> >     > > > > >     > > > on the
> >     > > > > >     > > >     > broker
> >     > > > > >     > > >     >     > side, the
> >     > > > > >     > > >     >     >         >> > > broker
> >     > > > > >     > > >     >     >         >> > > >> >    will always
> > have to
> >     > look
> >     > > > at
> >     > > > > > both
> >     > > > > >     > > > tombstone
> >     > > > > >     > > >     > bit and
> >     > > > > >     > > >     >     > the value
> >     > > > > >     > > >     >     >         >> when
> >     > > > > >     > > >     >     >         >> > > do
> >     > > > > >     > > >     >     >         >> > > >> the
> >     > > > > >     > > >     >     >         >> > > >> >    compaction.
> > Assuming
> >     > we
> >     > > do
> >     > > > > > not bump
> >     > > > > >     > up
> >     > > > > >     > > > the
> >     > > > > >     > > >     > magic
> >     > > > > >     > > >     >     > byte,
> >     > > > > >     > > >     >     >         >> > > >> >    imagine the
> > broker
> >     > sees
> >     > > a
> >     > > > > > message
> >     > > > > >     > > which
> >     > > > > >     > > > does
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > have a
> >     > > > > >     > > >     >     >         >> tombstone
> >     > > > > >     > > >     >     >         >> > > bit
> >     > > > > >     > > >     >     >         >> > > >> >    set. The
> broker
> > does
> >     > not
> >     > > > > know
> >     > > > > > when
> >     > > > > >     > the
> >     > > > > >     > > >     > message was
> >     > > > > >     > > >     >     > produced
> >     > > > > >     > > >     >     >         >> (i.e.
> >     > > > > >     > > >     >     >         >> > > >> > whether
> >     > > > > >     > > >     >     >         >> > > >> >    the message
> has
> > been
> >     > up
> >     > > > > > converted or
> >     > > > > >     > > > not), it
> >     > > > > >     > > >     > has
> >     > > > > >     > > >     >     > to take a
> >     > > > > >     > > >     >     >         >> > further
> >     > > > > >     > > >     >     >         >> > > >> > look at
> >     > > > > >     > > >     >     >         >> > > >> >    the value to
> > see if it
> >     > > is
> >     > > > > > null or
> >     > > > > >     > not
> >     > > > > >     > > in
> >     > > > > >     > > >     > order to
> >     > > > > >     > > >     >     > determine
> >     > > > > >     > > >     >     >         >> if it
> >     > > > > >     > > >     >     >         >> > > is
> >     > > > > >     > > >     >     >         >> > > >> a
> >     > > > > >     > > >     >     >         >> > > >> >    tombstone. The
> > same
> >     > > logic
> >     > > > > has
> >     > > > > > to be
> >     > > > > >     > > put
> >     > > > > >     > > > on the
> >     > > > > >     > > >     >     > consumer as
> >     > > > > >     > > >     >     >         >> well
> >     > > > > >     > > >     >     >         >> > > >> because
> >     > > > > >     > > >     >     >         >> > > >> > the
> >     > > > > >     > > >     >     >         >> > > >> >    consumer does
> > not know
> >     > > if
> >     > > > > the
> >     > > > > >     > message
> >     > > > > >     > > > has
> >     > > > > >     > > >     > been up
> >     > > > > >     > > >     >     > converted or
> >     > > > > >     > > >     >     >         >> > not.
> >     > > > > >     > > >     >     >         >> > > >> >       - If we
> > upconvert
> >     > > while
> >     > > > > >     > appending,
> >     > > > > >     > > > this is
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > the case,
> >     > > > > >     > > >     >     >         >> > right?
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> If I understand you
> >     > > correctly,
> >     > > > > > this is
> >     > > > > >     > not
> >     > > > > >     > > >     > sufficient
> >     > > > > >     > > >     >     > because the
> >     > > > > >     > > >     >     >         >> log
> >     > > > > >     > > >     >     >         >> > > may
> >     > > > > >     > > >     >     >         >> > > >> have messages
> > appended
> >     > before
> >     > > > it
> >     > > > > > was
> >     > > > > >     > > > upgraded to
> >     > > > > >     > > >     > include
> >     > > > > >     > > >     >     > KIP-87.
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >> Ismael
> >     > > > > >     > > >     >     >         >> > > >>
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > > > --
> >     > > > > >     > > >     >     >         >> > > > -Regards,
> >     > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> > > > (862) 250-7125
> >     > > > > >     > > >     >     >         >> > > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > > --
> >     > > > > >     > > >     >     >         >> > > -Regards,
> >     > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> > > (862) 250-7125
> >     > > > > >     > > >     >     >         >> > >
> >     > > > > >     > > >     >     >         >> > The information
> contained
> > in
> >     > this
> >     > > > > email
> >     > > > > > is
> >     > > > > >     > > > strictly
> >     > > > > >     > > >     >     > confidential and for
> >     > > > > >     > > >     >     >         >> > the use of the addressee
> > only,
> >     > > > unless
> >     > > > > >     > otherwise
> >     > > > > >     > > >     > indicated. If
> >     > > > > >     > > >     >     > you are
> >     > > > > >     > > >     >     >         >> not
> >     > > > > >     > > >     >     >         >> > the intended recipient,
> > please
> >     > do
> >     > > > not
> >     > > > > > read,
> >     > > > > >     > > copy,
> >     > > > > >     > > > use or
> >     > > > > >     > > >     >     > disclose to
> >     > > > > >     > > >     >     >         >> others
> >     > > > > >     > > >     >     >         >> > this message or any
> > attachment.
> >     > > > Please
> >     > > > > > also
> >     > > > > >     > > > notify the
> >     > > > > >     > > >     > sender
> >     > > > > >     > > >     >     > by
> >     > > > > >     > > >     >     >         >> replying
> >     > > > > >     > > >     >     >         >> > to this email or by
> > telephone
> >     > > > (+44(020
> >     > > > > > 7896
> >     > > > > >     > > 0011)
> >     > > > > >     > > > and
> >     > > > > >     > > >     > then
> >     > > > > >     > > >     >     > delete the
> >     > > > > >     > > >     >     >         >> email
> >     > > > > >     > > >     >     >         >> > and any copies of it.
> > Opinions,
> >     > > > > > conclusion
> >     > > > > >     > (etc)
> >     > > > > >     > > > that
> >     > > > > >     > > >     > do not
> >     > > > > >     > > >     >     > relate to
> >     > > > > >     > > >     >     >         >> the
> >     > > > > >     > > >     >     >         >> > official business of
> this
> >     > company
> >     > > > > shall
> >     > > > > > be
> >     > > > > >     > > > understood as
> >     > > > > >     > > >     >     > neither given
> >     > > > > >     > > >     >     >         >> nor
> >     > > > > >     > > >     >     >         >> > endorsed by it. IG is a
> > trading
> >     > > name
> >     > > > > of
> >     > > > > > IG
> >     > > > > >     > > Markets
> >     > > > > >     > > >     > Limited (a
> >     > > > > >     > > >     >     > company
> >     > > > > >     > > >     >     >         >> > registered in England
> and
> > Wales,
> >     > > > > company
> >     > > > > >     > number
> >     > > > > >     > > >     > 04008957) and
> >     > > > > >     > > >     >     > IG Index
> >     > > > > >     > > >     >     >         >> > Limited (a company
> > registered in
> >     > > > > > England and
> >     > > > > >     > > > Wales,
> >     > > > > >     > > >     > company
> >     > > > > >     > > >     >     > number
> >     > > > > >     > > >     >     >         >> > 01190902). Registered
> > address at
> >     > > > > Cannon
> >     > > > > > Bridge
> >     > > > > >     > > > House, 25
> >     > > > > >     > > >     >     > Dowgate Hill,
> >     > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
> > Markets
> >     > > > > Limited
> >     > > > > >     > > (register
> >     > > > > >     > > >     > number
> >     > > > > >     > > >     >     > 195355) and IG
> >     > > > > >     > > >     >     >         >> > Index Limited (register
> > number
> >     > > > 114059)
> >     > > > > > are
> >     > > > > >     > > > authorised
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > regulated by
> >     > > > > >     > > >     >     >         >> the
> >     > > > > >     > > >     >     >         >> > Financial Conduct
> > Authority.
> >     > > > > >     > > >     >     >         >> >
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >> --
> >     > > > > >     > > >     >     >         >> -Regards,
> >     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         >> (862) 250-7125
> >     > > > > >     > > >     >     >         >> The information contained
> > in this
> >     > > > email
> >     > > > > is
> >     > > > > >     > > strictly
> >     > > > > >     > > >     >     > confidential and for
> >     > > > > >     > > >     >     >         >> the use of the addressee
> > only,
> >     > > unless
> >     > > > > > otherwise
> >     > > > > >     > > >     > indicated. If
> >     > > > > >     > > >     >     > you are not
> >     > > > > >     > > >     >     >         >> the intended recipient,
> > please do
> >     > > not
> >     > > > > > read,
> >     > > > > >     > copy,
> >     > > > > >     > > > use or
> >     > > > > >     > > >     >     > disclose to others
> >     > > > > >     > > >     >     >         >> this message or any
> > attachment.
> >     > > Please
> >     > > > > > also
> >     > > > > >     > notify
> >     > > > > >     > > > the
> >     > > > > >     > > >     > sender
> >     > > > > >     > > >     >     > by replying
> >     > > > > >     > > >     >     >         >> to this email or by
> > telephone
> >     > > (+44(020
> >     > > > > > 7896
> >     > > > > >     > 0011)
> >     > > > > >     > > > and then
> >     > > > > >     > > >     >     > delete the email
> >     > > > > >     > > >     >     >         >> and any copies of it.
> > Opinions,
> >     > > > > > conclusion (etc)
> >     > > > > >     > > > that do
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > relate to the
> >     > > > > >     > > >     >     >         >> official business of this
> > company
> >     > > > shall
> >     > > > > be
> >     > > > > >     > > > understood as
> >     > > > > >     > > >     >     > neither given nor
> >     > > > > >     > > >     >     >         >> endorsed by it. IG is a
> > trading
> >     > name
> >     > > > of
> >     > > > > IG
> >     > > > > >     > Markets
> >     > > > > >     > > >     > Limited (a
> >     > > > > >     > > >     >     > company
> >     > > > > >     > > >     >     >         >> registered in England and
> > Wales,
> >     > > > company
> >     > > > > > number
> >     > > > > >     > > > 04008957)
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > IG Index
> >     > > > > >     > > >     >     >         >> Limited (a company
> > registered in
> >     > > > England
> >     > > > > > and
> >     > > > > >     > > Wales,
> >     > > > > >     > > >     > company
> >     > > > > >     > > >     >     > number
> >     > > > > >     > > >     >     >         >> 01190902). Registered
> > address at
> >     > > > Cannon
> >     > > > > > Bridge
> >     > > > > >     > > > House, 25
> >     > > > > >     > > >     >     > Dowgate Hill,
> >     > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> > Markets
> >     > > > Limited
> >     > > > > >     > (register
> >     > > > > >     > > > number
> >     > > > > >     > > >     >     > 195355) and IG
> >     > > > > >     > > >     >     >         >> Index Limited (register
> > number
> >     > > 114059)
> >     > > > > are
> >     > > > > >     > > > authorised and
> >     > > > > >     > > >     >     > regulated by the
> >     > > > > >     > > >     >     >         >> Financial Conduct
> Authority.
> >     > > > > >     > > >     >     >         >>
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >         > --
> >     > > > > >     > > >     >     >         > -Regards,
> >     > > > > >     > > >     >     >         > Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         > (862) 250-7125
> >     > > > > >     > > >     >     >         >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >         --
> >     > > > > >     > > >     >     >         -Regards,
> >     > > > > >     > > >     >     >         Mayuresh R. Gharat
> >     > > > > >     > > >     >     >         (862) 250-7125
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >     The information contained in this
> > email
> >     > is
> >     > > > > > strictly
> >     > > > > >     > > > confidential
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > for the use of the addressee only,
> > unless
> >     > > > otherwise
> >     > > > > >     > > indicated.
> >     > > > > >     > > > If
> >     > > > > >     > > >     > you are
> >     > > > > >     > > >     >     > not the intended recipient, please do
> > not
> >     > read,
> >     > > > > > copy, use
> >     > > > > >     > or
> >     > > > > >     > > >     > disclose to
> >     > > > > >     > > >     >     > others this message or any
> attachment.
> > Please
> >     > > > also
> >     > > > > > notify
> >     > > > > >     > the
> >     > > > > >     > > > sender
> >     > > > > >     > > >     > by
> >     > > > > >     > > >     >     > replying to this email or by
> telephone
> >     > (+44(020
> >     > > > > 7896
> >     > > > > > 0011)
> >     > > > > >     > > and
> >     > > > > >     > > > then
> >     > > > > >     > > >     > delete
> >     > > > > >     > > >     >     > the email and any copies of it.
> > Opinions,
> >     > > > > conclusion
> >     > > > > > (etc)
> >     > > > > >     > > > that do
> >     > > > > >     > > >     > not
> >     > > > > >     > > >     >     > relate to the official business of
> this
> >     > company
> >     > > > > > shall be
> >     > > > > >     > > > understood
> >     > > > > >     > > >     > as
> >     > > > > >     > > >     >     > neither given nor endorsed by it. IG
> > is a
> >     > > trading
> >     > > > > > name of
> >     > > > > >     > IG
> >     > > > > >     > > > Markets
> >     > > > > >     > > >     >     > Limited (a company registered in
> > England and
> >     > > > Wales,
> >     > > > > > company
> >     > > > > >     > > > number
> >     > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> > company
> >     > > > > registered
> >     > > > > > in
> >     > > > > >     > > > England and
> >     > > > > >     > > >     > Wales,
> >     > > > > >     > > >     >     > company number 01190902). Registered
> > address
> >     > at
> >     > > > > > Cannon
> >     > > > > >     > Bridge
> >     > > > > >     > > > House,
> >     > > > > >     > > >     > 25
> >     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both
> IG
> >     > Markets
> >     > > > > > Limited
> >     > > > > >     > > > (register
> >     > > > > >     > > >     > number
> >     > > > > >     > > >     >     > 195355) and IG Index Limited
> (register
> > number
> >     > > > > > 114059) are
> >     > > > > >     > > > authorised
> >     > > > > >     > > >     > and
> >     > > > > >     > > >     >     > regulated by the Financial Conduct
> > Authority.
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     >
> >     > > > > >     > > >     > The information contained in this email is
> > strictly
> >     > > > > > confidential
> >     > > > > >     > > and
> >     > > > > >     > > > for
> >     > > > > >     > > >     > the use of the addressee only, unless
> > otherwise
> >     > > > > indicated.
> >     > > > > > If you
> >     > > > > >     > > > are not
> >     > > > > >     > > >     > the intended recipient, please do not read,
> > copy,
> >     > use
> >     > > > or
> >     > > > > > disclose
> >     > > > > >     > > to
> >     > > > > >     > > > others
> >     > > > > >     > > >     > this message or any attachment. Please also
> > notify
> >     > > the
> >     > > > > > sender by
> >     > > > > >     > > > replying
> >     > > > > >     > > >     > to this email or by telephone (+44(020 7896
> > 0011
> >     > <020%207896%200011>) and
> >     > > > > then
> >     > > > > > delete
> >     > > > > >     > > > the email
> >     > > > > >     > > >     > and any copies of it. Opinions, conclusion
> > (etc)
> >     > that
> >     > > > do
> >     > > > > > not
> >     > > > > >     > relate
> >     > > > > >     > > > to the
> >     > > > > >     > > >     > official business of this company shall be
> >     > understood
> >     > > > as
> >     > > > > > neither
> >     > > > > >     > > > given nor
> >     > > > > >     > > >     > endorsed by it. IG is a trading name of IG
> > Markets
> >     > > > > Limited
> >     > > > > > (a
> >     > > > > >     > > company
> >     > > > > >     > > >     > registered in England and Wales, company
> > number
> >     > > > 04008957)
> >     > > > > > and IG
> >     > > > > >     > > > Index
> >     > > > > >     > > >     > Limited (a company registered in England
> and
> > Wales,
> >     > > > > company
> >     > > > > >     > number
> >     > > > > >     > > >     > 01190902). Registered address at Cannon
> > Bridge
> >     > House,
> >     > > > 25
> >     > > > > > Dowgate
> >     > > > > >     > > > Hill,
> >     > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > > > number
> >     > > > > > 195355)
> >     > > > > >     > > > and IG
> >     > > > > >     > > >     > Index Limited (register number 114059) are
> >     > authorised
> >     > > > and
> >     > > > > >     > regulated
> >     > > > > >     > > > by the
> >     > > > > >     > > >     > Financial Conduct Authority.
> >     > > > > >     > > >     >
> >     > > > > >     > > >
> >     > > > > >     > > >
> >     > > > > >     > > > The information contained in this email is
> strictly
> >     > > > > confidential
> >     > > > > > and
> >     > > > > >     > for
> >     > > > > >     > > > the use of the addressee only, unless otherwise
> >     > indicated.
> >     > > If
> >     > > > > > you are
> >     > > > > >     > not
> >     > > > > >     > > > the intended recipient, please do not read, copy,
> > use or
> >     > > > > > disclose to
> >     > > > > >     > > others
> >     > > > > >     > > > this message or any attachment. Please also
> notify
> > the
> >     > > sender
> >     > > > > by
> >     > > > > >     > replying
> >     > > > > >     > > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > > > > delete the
> >     > > > > >     > > email
> >     > > > > >     > > > and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > > not
> >     > > > > > relate to
> >     > > > > >     > > the
> >     > > > > >     > > > official business of this company shall be
> > understood as
> >     > > > > neither
> >     > > > > > given
> >     > > > > >     > > nor
> >     > > > > >     > > > endorsed by it. IG is a trading name of IG
> Markets
> >     > Limited
> >     > > (a
> >     > > > > > company
> >     > > > > >     > > > registered in England and Wales, company number
> > 04008957)
> >     > > and
> >     > > > > IG
> >     > > > > > Index
> >     > > > > >     > > > Limited (a company registered in England and
> Wales,
> >     > company
> >     > > > > > number
> >     > > > > >     > > > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     > > > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> >     > > > > > 195355) and
> >     > > > > >     > IG
> >     > > > > >     > > > Index Limited (register number 114059) are
> > authorised and
> >     > > > > > regulated by
> >     > > > > >     > > the
> >     > > > > >     > > > Financial Conduct Authority.
> >     > > > > >     > > >
> >     > > > > >     > > The information contained in this email is strictly
> >     > > > confidential
> >     > > > > > and for
> >     > > > > >     > > the use of the addressee only, unless otherwise
> > indicated.
> >     > If
> >     > > > you
> >     > > > > > are not
> >     > > > > >     > > the intended recipient, please do not read, copy,
> > use or
> >     > > > disclose
> >     > > > > > to
> >     > > > > >     > others
> >     > > > > >     > > this message or any attachment. Please also notify
> > the
> >     > sender
> >     > > > by
> >     > > > > > replying
> >     > > > > >     > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > > delete
> >     > > > > > the
> >     > > > > >     > email
> >     > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     > > > > > relate to
> >     > > > > >     > the
> >     > > > > >     > > official business of this company shall be
> > understood as
> >     > > > neither
> >     > > > > > given
> >     > > > > >     > nor
> >     > > > > >     > > endorsed by it. IG is a trading name of IG Markets
> > Limited
> >     > (a
> >     > > > > > company
> >     > > > > >     > > registered in England and Wales, company number
> > 04008957)
> >     > and
> >     > > > IG
> >     > > > > > Index
> >     > > > > >     > > Limited (a company registered in England and Wales,
> > company
> >     > > > > number
> >     > > > > >     > > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     > > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     > > > 195355)
> >     > > > > > and IG
> >     > > > > >     > > Index Limited (register number 114059) are
> > authorised and
> >     > > > > > regulated by
> >     > > > > >     > the
> >     > > > > >     > > Financial Conduct Authority.
> >     > > > > >     > >
> >     > > > > >     > >
> >     > > > > >     > The information contained in this email is strictly
> >     > > confidential
> >     > > > > and
> >     > > > > > for
> >     > > > > >     > the use of the addressee only, unless otherwise
> > indicated. If
> >     > > you
> >     > > > > > are not
> >     > > > > >     > the intended recipient, please do not read, copy, use
> > or
> >     > > disclose
> >     > > > > to
> >     > > > > > others
> >     > > > > >     > this message or any attachment. Please also notify
> the
> > sender
> >     > > by
> >     > > > > > replying
> >     > > > > >     > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then
> >     > > delete
> >     > > > > > the email
> >     > > > > >     > and any copies of it. Opinions, conclusion (etc) that
> > do not
> >     > > > relate
> >     > > > > > to the
> >     > > > > >     > official business of this company shall be understood
> > as
> >     > > neither
> >     > > > > > given nor
> >     > > > > >     > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> >     > > > > company
> >     > > > > >     > registered in England and Wales, company number
> > 04008957) and
> >     > > IG
> >     > > > > > Index
> >     > > > > >     > Limited (a company registered in England and Wales,
> > company
> >     > > > number
> >     > > > > >     > 01190902). Registered address at Cannon Bridge House,
> > 25
> >     > > Dowgate
> >     > > > > > Hill,
> >     > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     > > 195355)
> >     > > > > > and IG
> >     > > > > >     > Index Limited (register number 114059) are authorised
> > and
> >     > > > regulated
> >     > > > > > by the
> >     > > > > >     > Financial Conduct Authority.
> >     > > > > >     >
> >     > > > > >     >
> >     > > > > >
> >     > > > > >
> >     > > > > > The information contained in this email is strictly
> > confidential
> >     > and
> >     > > > for
> >     > > > > > the use of the addressee only, unless otherwise indicated.
> > If you
> >     > are
> >     > > > not
> >     > > > > > the intended recipient, please do not read, copy, use or
> > disclose
> >     > to
> >     > > > > others
> >     > > > > > this message or any attachment. Please also notify the
> > sender by
> >     > > > replying
> >     > > > > > to this email or by telephone (+44(020 7896 0011
> >     > <020%207896%200011>) and then delete the
> >     > > > > email
> >     > > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> >     > > to
> >     > > > > the
> >     > > > > > official business of this company shall be understood as
> > neither
> >     > > given
> >     > > > > nor
> >     > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> >     > company
> >     > > > > > registered in England and Wales, company number 04008957)
> > and IG
> >     > > Index
> >     > > > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > > Hill,
> >     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> >     > and
> >     > > > IG
> >     > > > > > Index Limited (register number 114059) are authorised and
> > regulated
> >     > > by
> >     > > > > the
> >     > > > > > Financial Conduct Authority.
> >     > > > > >
> >     > > > > The information contained in this email is strictly
> > confidential and
> >     > > for
> >     > > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > > not
> >     > > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > > others
> >     > > > > this message or any attachment. Please also notify the sender
> > by
> >     > > replying
> >     > > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the
> >     > > > email
> >     > > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> >     > to
> >     > > > the
> >     > > > > official business of this company shall be understood as
> > neither
> >     > given
> >     > > > nor
> >     > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > > registered in England and Wales, company number 04008957) and
> > IG
> >     > Index
> >     > > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > Hill,
> >     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > > IG
> >     > > > > Index Limited (register number 114059) are authorised and
> > regulated
> >     > by
> >     > > > the
> >     > > > > Financial Conduct Authority.
> >     > > > >
> >     > > > The information contained in this email is strictly
> confidential
> > and
> >     > for
> >     > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > not
> >     > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > others
> >     > > > this message or any attachment. Please also notify the sender
> by
> >     > replying
> >     > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the
> >     > > email
> >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > > the
> >     > > > official business of this company shall be understood as
> neither
> > given
> >     > > nor
> >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> >     > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > IG
> >     > > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > > the
> >     > > > Financial Conduct Authority.
> >     > > >
> >     > > The information contained in this email is strictly confidential
> > and for
> >     > > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > > the intended recipient, please do not read, copy, use or disclose
> > to
> >     > others
> >     > > this message or any attachment. Please also notify the sender by
> > replying
> >     > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>)
> >     > and then delete the email
> >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > the
> >     > > official business of this company shall be understood as neither
> > given
> >     > nor
> >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > > Limited (a company registered in England and Wales, company
> number
> >     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > the
> >     > > Financial Conduct Authority.
> >     > >
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jay

I disagree here that we are breaking any compatibility, we went through this on the discussion thread in fact with the help of that thread is how the kip came to the solution.

Also on the supported combinations front you mention, we are not taking anything away afaik.

Currently supported are only are:
Null value = delete
Non-null value = non delete

With this kip we would support
Null value + tombstone = delete
Non null value + tombstone = delete
Non null value + no tombstone = non delete

As for the alternative idea, this is simply a new policy, how can there be confusion here? For this policy it would be explicit that tombstone marker would need to be set for a delete.

I'm going to vent a little now as starting to get quite frustrated.

We are going round in circles on kip-82 as per use cases there is now many use cases, how many more are needed? just because confluent don't see these doesn't mean they aren't real use cases other have, this is the point of the Apache foundation, it shouldn't be the view of just one organisation. It really is getting a feeling of the NIH syndrome. Rather than it being constructive on discussion of the implementation detail.

kip-87 spawned from as on the kip call we all agreed this was needed. And would at least allow a custom wrapper be supported in a compacted topic, allowing meta data. Which again now I feel we are spinning wheels, and simply finding reasons not support it.

Cheers
Mike



Sent using OWA for iPad
________________________________________
From: Jay Kreps <ja...@confluent.io>
Sent: Friday, December 16, 2016 7:09:23 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hey Michael,

I do think it might have been better had we started with a separate concept
of null vs delete. Given that we are where we are, I'm not sure that the
possible cures we explored so far are better than the disease.

I apologize for coming to this late, but I didn't really understand the
implications of the KIP and that we'd be breaking compatibility with
existing apps until the vote had begun, and in my defense I'm not sure the
other folks voting did either.

I think we all agree there are many existing apps that are built with the
assumption of "null value non-tombstone" and it isn't possible to
disambiguate these from tombstones on the producer. It isn't that anyone is
saying we have to support all four possibilities at once, it's that we
simply can't orphan one of the existing combinations or our users will eat
us!

If I've understood your alternate solution of adding another setting for
compaction, I think this does fix the compatibility problem, but it adds an
odd mode the user has to add on all their topics. While the current state
is easily explainable, the resulting state where the meaning of tombstone
and null are overlapping and ambiguous and dependent on a compaction
setting that could change out of band or not be in sync with your code in
some environment seems worse to me then where we currently are. I think the
question is how would this combination be explained to users and does it
make sense?

-Jay



On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Chaps,
>
> Can we either get one more +1 binding (we have 2 already) on the existing?
>
> Or have some response on the below possible alternative. We are keen to
> get working on this, so we make next feature release.
>
> Cheers
> Mike
>
>
> On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com> wrote:
>
>     Hi Ismael
>
>     Did you see our email this morning, what's your thoughts on this
> approach to instead we simply have a brand new policy?
>
>     Cheers
>     Mike
>
>
>     Sent using OWA for iPhone
>     ________________________________________
>     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <
> ismael@juma.me.uk>
>     Sent: Tuesday, December 13, 2016 11:30:05 AM
>     To: dev@kafka.apache.org
>     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
>     Yes, this is actually tricky to do in a way where we both have the
> desired
>     semantics and maintain compatibility. When someone creates a
>     `ProducerRecord` with a `null` value today, the producer doesn't know
> if
>     it's meant to be a tombstone or not. For V3 messages, it's easy when
> the
>     constructor that takes a tombstone is used. However, if any other
>     constructor is used, it's not clear. A couple of options I can think
> of,
>     none of them particularly nice:
>
>     1. Have a third state where tombstone = unknown and the broker would
> set
>     the tombstone bit if the value was null and the topic was compacted.
> People
>     that wanted to pass a non-null value for the tombstone would have to
> use
>     the constructor that takes a tombstone. The drawbacks: third state for
>     tombstone in message format, message conversion at the broker for a
> common
>     case.
>
>     2. Extend MetadataResponse to optionally include topic configs, which
> would
>     make it possible for the producer to be smarter about setting the
>     tombstone. It would only do it if a tombstone was not passed
> explicitly,
>     the value was null and the topic was compacted. The main drawback is
> that
>     the producer would be getting a bit more data for each topic even
> though it
>     probably won't use it most of the time. Extending MetadataResponse to
>     return topic configs would be useful for other reasons as well, so that
>     part seems OK.
>
>     In addition, for both proposals, we could consider adding warnings to
> the
>     documentation that the behaviour of the constructors that don't take a
>     tombstone would change in the next major release so that tombstone =
> false.
>     Not sure if this would be worth it though.
>
>     Ismael
>
>     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> ewen@confluent.io>
>     wrote:
>
>     > Michael,
>     >
>     > It kind of depends on how you want to interpret the tombstone flag.
> If it's
>     > purely a producer-facing Kafka-level thing that we treat as internal
> to the
>     > broker and log cleaner once the record is sent, then your approach
> makes
>     > sense. You're just moving copying the null-indicates-delete behavior
> of the
>     > old constructor into the tombstone flag.
>     >
>     > However, if you want this change to more generally decouple the idea
> of
>     > deletion and null values, then you are sometimes converting what
> might be a
>     > completely valid null value that doesn't indicate deletion into a
>     > tombstone. Downstream applications could potentially handle these
> cases
>     > differently given the separation of deletion from value.
>     >
>     > I guess the question is if we want to try to support the latter even
> for
>     > topics where we have older produce requests. An example where this
> could
>     > come up is in something like a CDC Connector. If we try to support
> the
>     > semantic difference, a connector might write changes to Kafka using
> the
>     > tombstone flag to indicate when a row was truly deleted (vs an
> update that
>     > sets it to null but still present; this probably makes more sense
> for CDC
>     > from document stores or extracting single columns). There are various
>     > reasons we might want to maintain the full log and not turn
> compaction on
>     > (or just use a time-based retention policy), but downstream
> applications
>     > might care to know the difference between a delete and a null value.
> In
>     > fact both versions of the same log (compacted and time-retention)
> could be
>     > useful and I don't think it'll be uncommon to maintain both or use
> KIP-71
>     > to maintain a hybrid compacted/retention topic.
>     >
>     > -Ewen
>     >
>     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > wrote:
>     >
>     > > Hi Jay,
>     > >
>     > > Why wouldn't that work, the tombstone value is only looked at by
> the
>     > > broker, on a topic configured for compaction as such is benign on
> non
>     > > compacted topics. This is as much as sending a null value currently
>     > >
>     > >
>     > > Regards
>     > > Mike
>     > >
>     > >
>     > >
>     > > Sent using OWA for iPhone
>     > > ________________________________________
>     > > From: Jay Kreps <ja...@confluent.io>
>     > > Sent: Sunday, December 11, 2016 8:58:53 PM
>     > > To: dev@kafka.apache.org
>     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > >
>     > > Hey Michael,
>     > >
>     > > I'm not quite sure that works as that would translate ALL null
> values to
>     > > tombstones, even for non-compacted topics that use null as an
> acceptable
>     > > value sent by the producer and expected by the consumer.
>     > >
>     > > -Jay
>     > >
>     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > > wrote:
>     > >
>     > > > Hi Ewen,
>     > > >
>     > > > I think the easiest way to show this is with code.
>     > > >
>     > > > As you can see we keep the existing behaviour for code/binaries
> calling
>     > > > the pre-existing constructors, whereby if the value is null the
>     > tombstone
>     > > > is set to true.
>     > > >
>     > > > Regards
>     > > > Mike
>     > > >
>     > > >
>     > > >
>     > > >     /**
>     > > >      * Creates a record with a specified timestamp to be sent to
> a
>     > > > specified topic and partition
>     > > >      *
>     > > >      * @param topic The topic the record will be appended to
>     > > >      * @param partition The partition to which the record should
> be
>     > sent
>     > > >      * @param timestamp The timestamp of the record
>     > > >      * @param tombstone if the record should be treated as a
> tombstone
>     > if
>     > > > the topic is compacted
>     > > >      * @param key The key that will be included in the record
>     > > >      * @param value The record contents
>     > > >      */
>     > > >     public ProducerRecord(String topic, Integer partition,
> Boolean
>     > > > tombstone, Long timestamp, K key, V value) {
>     > > >         if (topic == null)
>     > > >             throw new IllegalArgumentException("Topic cannot be
>     > null.");
>     > > >         if (timestamp != null && timestamp < 0)
>     > > >             throw new IllegalArgumentException(
>     > > >                     String.format("Invalid timestamp: %d.
> Timestamp
>     > > should
>     > > > always be non-negative or null.", timestamp));
>     > > >         if (partition != null && partition < 0)
>     > > >             throw new IllegalArgumentException(
>     > > >                     String.format("Invalid partition: %d.
> Partition
>     > > number
>     > > > should always be non-negative or null.", partition));
>     > > >         if (!tombstone && value == null){
>     > > >             throw new IllegalArgumentException(
>     > > >                     String.format("Tombstone must be true if null
>     > > value"));
>     > > >         }
>     > > >         this.topic = topic;
>     > > >         this.partition = partition;
>     > > >         this.tombstone = tombstone;
>     > > >         this.key = key;
>     > > >         this.value = value;
>     > > >         this.timestamp = timestamp;
>     > > >     }
>     > > >
>     > > >     public ProducerRecord(String topic, Integer partition, Long
>     > > timestamp,
>     > > > K key, V value) {
>     > > >         this(topic, partition, value == null, timestamp, key,
> value);
>     > > >     }
>     > > > ________________________________________
>     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
>     > > > Sent: Sunday, December 11, 2016 5:45 AM
>     > > > To: dev@kafka.apache.org
>     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > >
>     > > > It seemed like this was addressed in the migration section,
> wasn't it?
>     > > The
>     > > > V2 vs V3 requests and the fact that the broker will downgrade the
>     > message
>     > > > format (losing zero copy) if you issues a V2 request to a broker
> using
>     > V3
>     > > > format handles compatibility, does it not? The existing
> consumers won't
>     > > see
>     > > > the extra metadata in the value, but they will get a null
> instead and
>     > > treat
>     > > > it as a tombstone. Certainly there is a performance impact, but
> it
>     > seemed
>     > > > compatible.
>     > > >
>     > > > I'm worried about this though:
>     > > >
>     > > >
>     > > >    - *NOTE *: With the new version of producer client using
>     > > ProduceRequest
>     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set)
> and
>     > null
>     > > > value
>     > > >    should not be allowed as the older version of consumer using
>     > > > FetchRequest
>     > > >    V2 will think of this as a tombstone when its actually not.
>     > > >
>     > > >
>     > > > Unless I'm misunderstanding, this ends up breaking binary
> compatibility
>     > > for
>     > > > the Java API. It sounds like this suggests that passing a null
> value to
>     > > the
>     > > > existing ProducerRecord constructors would generate an exception
> since
>     > > you
>     > > > didn't explicitly enable the tombstone (via whatever new
> constructor is
>     > > > provided). But that means you can't swap in a newer client jar
> without
>     > > > recompiling and get the same behavior if your app deletes keys
> using
>     > the
>     > > > current approach because your app will start throwing
> exceptions. Maybe
>     > > > this is a tradeoff we're ok with, but we've tried pretty hard to
> avoid
>     > > > breaking compatibility like this so far -- it makes picking up
> bug
>     > fixes
>     > > in
>     > > > the clients harder for users of frameworks that have to pin to
> earlier
>     > > > default versions for compatibility.
>     > > >
>     > > > But then later the KIP says:
>     > > >
>     > > >
>     > > >    - The old constructors for ProducerRecord without this
> parameter
>     > will
>     > > be
>     > > >    kept but updated so that their default behaviour if setting
> null
>     > value
>     > > > will
>     > > >    be the tombstone will be set to true to keep existing
> behaviour.
>     > > >
>     > > >
>     > > > So maybe I am misinterpreting something.
>     > > >
>     > > > And just a nit re: motivation section, item 6 would be clearer
> for a
>     > > union
>     > > > schema with null (or optional schemas in other formats), e.g.
> [null,
>     > > > string], in which case losing the schema truly is losing
> information
>     > > > (whereas null is already the only valid value for a pure null
> schema).
>     > > >
>     > > > -Ewen
>     > > >
>     > > >
>     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> Michael.Pearce@ig.com
>     > >
>     > > > wrote:
>     > > >
>     > > > > Hi Jay,
>     > > > >
>     > > > > Good point this detail is missing in the KIP write up. Ive
> added this
>     > > > now.
>     > > > >
>     > > > > Essentially simply just upgrading the clients we do not expect
> any
>     > > client
>     > > > > app code change needed.
>     > > > >
>     > > > > Cheers
>     > > > > Mike
>     > > > > ________________________________________
>     > > > > From: Jay Kreps <ja...@confluent.io>
>     > > > > Sent: Saturday, December 10, 2016 10:51 PM
>     > > > > To: dev@kafka.apache.org
>     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > > >
>     > > > > Michael,
>     > > > >
>     > > > > The compatibility section goes through the migration path, but
> isn't
>     > > the
>     > > > > bigger compatibility issue with existing apps? There are many
>     > (probably
>     > > > > thousands) of apps in production that use this feature and
> send null
>     > to
>     > > > > mean delete. It seems like this would break compatibility with
> them,
>     > > and
>     > > > > they would have to be rewritten to right?
>     > > > >
>     > > > > -Jay
>     > > > >
>     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
>     > Michael.Pearce@ig.com
>     > > >
>     > > > > wrote:
>     > > > >
>     > > > > > Hi Jun,
>     > > > > >
>     > > > > > 4) On v3 we honour the tombstone. As such we expect it to be
> set
>     > > > > correctly
>     > > > > > as per the KIP.
>     > > > > >
>     > > > > > 4.1) why would we want to produce an error when its v3? This
> is the
>     > > > exact
>     > > > > > purpose to support non-null tombstone’s
>     > > > > > 4.2) again here im unclear on the question on the v3, produce
>     > request
>     > > > we
>     > > > > > expect the tombstone flag to be set correctly.
>     > > > > >
>     > > > > > 4.4) compaction only occurs on compacted topics, the bit
> makes no
>     > > > > > difference and not looked at on un-compacted (time/size based
>     > > > eviction).
>     > > > > >
>     > > > > >
>     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
>     > > > > >
>     > > > > >     Hi, Michael,
>     > > > > >
>     > > > > >     4. Then, I think I misunderstood this point. Could you
> document
>     > > the
>     > > > > >     following points in the wiki?
>     > > > > >     4.1 If producer V3 sets tombstone, but provides a
> non-null
>     > value,
>     > > > > does
>     > > > > > the
>     > > > > >     send() get an error or does the producer automatically
> set the
>     > > > value
>     > > > > to
>     > > > > >     null?
>     > > > > >     4.2 If producer V3 doesn't set tombstone, but provides a
> null
>     > > > value,
>     > > > > > does
>     > > > > >     the send() get an error or does the producer
> automatically sets
>     > > the
>     > > > > >     tombstone?
>     > > > > >     4.3 Does the broker only expect messages that (a) have no
>     > > tombstone
>     > > > > and
>     > > > > >     non-null value; (b) have tombstone and null value and
> reject
>     > the
>     > > > > > messages
>     > > > > >     with an error code otherwise?
>     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
> compacted
>     > on
>     > > > > not?
>     > > > > >
>     > > > > >     Thanks,
>     > > > > >
>     > > > > >     Jun
>     > > > > >
>     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
>     > > > > Michael.Pearce@ig.com
>     > > > > > >
>     > > > > >     wrote:
>     > > > > >
>     > > > > >     > Not at all.  This only acts on compacted topics just
> as what
>     > > > occurs
>     > > > > > today
>     > > > > >     >
>     > > > > >     > Sent using OWA for iPhone
>     > > > > >     > ________________________________________
>     > > > > >     > From: Jun Rao <ju...@confluent.io>
>     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
>     > > > > >     > To: dev@kafka.apache.org
>     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> Flag
>     > > > > >     >
>     > > > > >     > Hi, Michael,
>     > > > > >     >
>     > > > > >     > 4. Hmm, does that mean the new client library can
> never send
>     > a
>     > > > null
>     > > > > > message
>     > > > > >     > even to a regular topic? This seems like a change of
> the
>     > > existing
>     > > > > > behavior.
>     > > > > >     >
>     > > > > >     > Thanks,
>     > > > > >     >
>     > > > > >     > Jun
>     > > > > >     >
>     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
>     > > > > > Michael.Pearce@ig.com>
>     > > > > >     > wrote:
>     > > > > >     >
>     > > > > >     > > Hi Jun,
>     > > > > >     > >
>     > > > > >     > > Re 4) That's because we expect the tombstone value
> to be
>     > set
>     > > > > > correctly if
>     > > > > >     > > message bit is 2, as such if an older client sends
> in on
>     > old
>     > > > > > message the
>     > > > > >     > > message is upcast and the bit is set correctly. And
> such no
>     > > > > longer
>     > > > > > need
>     > > > > >     > to
>     > > > > >     > > check the value. Mayuresh can you confirm my
> thinking and
>     > > > > > understanding
>     > > > > >     > of
>     > > > > >     > > what we've discussed?
>     > > > > >     > >
>     > > > > >     > > The second point I understand what you're getting at
> now my
>     > > > > > apologies.
>     > > > > >     > Yes
>     > > > > >     > > this makes sense to save on touching the message, if
> we're
>     > > the
>     > > > > > only kip
>     > > > > >     > > going in, in this release.
>     > > > > >     > >
>     > > > > >     > > Cheers
>     > > > > >     > > Mike
>     > > > > >     > >
>     > > > > >     > > Sent using OWA for iPhone
>     > > > > >     > > ________________________________________
>     > > > > >     > > From: Jun Rao <ju...@confluent.io>
>     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>     > > > > >     > > To: dev@kafka.apache.org
>     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone Flag
>     > > > > >     > >
>     > > > > >     > > Hi, Michael,
>     > > > > >     > >
>     > > > > >     > > 4. Is this updated in the wiki? The text "If the
> magic byte
>     > > on
>     > > > > > message is
>     > > > > >     > > 2, the broker should use the tombstone bit for log
>     > > compaction."
>     > > > > > doesn't
>     > > > > >     > > seem to have changed.
>     > > > > >     > >
>     > > > > >     > > 2. My point is that if we change the message format
> just
>     > for
>     > > > this
>     > > > > > KIP, we
>     > > > > >     > > should consider whether it's worth optimizing the
> down
>     > > > conversion
>     > > > > > path
>     > > > > >     > > (i.e., decide whether a conversion is needed by just
>     > looking
>     > > at
>     > > > > the
>     > > > > >     > > tombstone bit in the wrapper message) since
> tombstone will
>     > be
>     > > > > used
>     > > > > >     > rarely.
>     > > > > >     > > However, if the message format change here is
> combined with
>     > > > other
>     > > > > > KIPs,
>     > > > > >     > > then this optimization likely won't be needed. The
> latter
>     > > > > probably
>     > > > > > makes
>     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> think?
>     > > > > >     > >
>     > > > > >     > > Other than those, +1 from me,
>     > > > > >     > >
>     > > > > >     > > Thanks,
>     > > > > >     > >
>     > > > > >     > > Jun
>     > > > > >     > >
>     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
>     > > > > > Michael.Pearce@ig.com>
>     > > > > >     > > wrote:
>     > > > > >     > >
>     > > > > >     > > > Hi Jun
>     > > > > >     > > >
>     > > > > >     > > > do we have your vote on this now?
>     > > > > >     > > >
>     > > > > >     > > > Any other concerns?
>     > > > > >     > > >
>     > > > > >     > > > Cheers
>     > > > > >     > > > Mike
>     > > > > >     > > >
>     > > > > >     > > > Sent using OWA for iPhone
>     > > > > >     > > > ________________________________________
>     > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
>     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>     > > > > >     > > > To: dev@kafka.apache.org
>     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone
>     > Flag
>     > > > > >     > > >
>     > > > > >     > > > Hi Jun,
>     > > > > >     > > >
>     > > > > >     > > > Have updated. Thanks again for the feedback.
>     > > > > >     > > >
>     > > > > >     > > > Agree yes we should align up when it gets to that,
> I
>     > assume
>     > > > > > you’ve
>     > > > > >     > > flagged
>     > > > > >     > > > the same in the other KIP?
>     > > > > >     > > >
>     > > > > >     > > > I think for now let’s get this KIP approved, then
> we can
>     > > > start
>     > > > > > the work
>     > > > > >     > > to
>     > > > > >     > > > get an initial PR, then we can discuss how to
> align the
>     > two
>     > > > if
>     > > > > > needed.
>     > > > > >     > > >
>     > > > > >     > > > Cheers,
>     > > > > >     > > > Mike
>     > > > > >     > > >
>     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io>
>     > wrote:
>     > > > > >     > > >
>     > > > > >     > > >     Hi, Michael,
>     > > > > >     > > >
>     > > > > >     > > >     For 2), this is fine. Could you update the KIP
> wiki
>     > to
>     > > > make
>     > > > > > this
>     > > > > >     > and
>     > > > > >     > > > other
>     > > > > >     > > >     points clearer? Other than that, the KIP looks
> good
>     > to
>     > > > me.
>     > > > > >     > > >
>     > > > > >     > > >     An orthogonal thing is that there are other
> KIPs such
>     > > as
>     > > > > > KIP-98
>     > > > > >     > that
>     > > > > >     > > > also
>     > > > > >     > > >     intend to change the message format. If they
> all get
>     > > > > > approved, we
>     > > > > >     > > > should
>     > > > > >     > > >     think about whether it's better to just bump
> up the
>     > > magic
>     > > > > > byte once
>     > > > > >     > > to
>     > > > > >     > > >     incorporate multiple format changes like we
> did in
>     > > > > > KIP-31/KIP-32.
>     > > > > >     > > >
>     > > > > >     > > >     Thanks,
>     > > > > >     > > >
>     > > > > >     > > >     Jun
>     > > > > >     > > >
>     > > > > >     > > >
>     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce
> <
>     > > > > >     > > Michael.Pearce@ig.com>
>     > > > > >     > > >     wrote:
>     > > > > >     > > >
>     > > > > >     > > >     > Hi Jao
>     > > > > >     > > >     >
>     > > > > >     > > >     > Thanks for the response. Sorry for slow
> reply, both
>     > > > with
>     > > > > > personal
>     > > > > >     > > > sickness
>     > > > > >     > > >     > and also battling some critical issues
> encountered
>     > > > since
>     > > > > >     > upgrading
>     > > > > >     > > to
>     > > > > >     > > >     > 0.10.1.0
>     > > > > >     > > >     >
>     > > > > >     > > >     > 1) Thans for spotting, Document error where
> we
>     > > branched
>     > > > > > this KIP
>     > > > > >     > > from
>     > > > > >     > > >     > KIP-82, will get that removed.
>     > > > > >     > > >     > 2) Intent is to do this just at the record
> message
>     > > > level.
>     > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
>     > > corrected.
>     > > > > >     > > >     > 4) As per discussion thread we will support
>     > > tombstone +
>     > > > > > null
>     > > > > >     > value,
>     > > > > >     > > >     > tombstone + non null value, no tombstone +
> null
>     > > value.
>     > > > > >     > > >     > 5) I believe this was in the discussion
> thread,
>     > > > @Mayuresh
>     > > > > > is this
>     > > > > >     > > >     > something we’ve overlooked? I thought we
> would down
>     > > > > > convert and
>     > > > > >     > > > remove the
>     > > > > >     > > >     > value so the old consumer had existing
> behavior, or
>     > > is
>     > > > > > there
>     > > > > >     > > > something we
>     > > > > >     > > >     > haven’t thought about?
>     > > > > >     > > >     >
>     > > > > >     > > >     > Cheers
>     > > > > >     > > >     > Mike
>     > > > > >     > > >     >
>     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> jun@confluent.io>
>     > > > > wrote:
>     > > > > >     > > >     >
>     > > > > >     > > >     >     Hi, Michael,
>     > > > > >     > > >     >
>     > > > > >     > > >     >     Thanks for the KIP. A few comments below.
>     > > > > >     > > >     >
>     > > > > >     > > >     >     1. The message format change contains
>     > > > "HeadersLength
>     > > > > >     > Headers".
>     > > > > >     > > > Is that
>     > > > > >     > > >     >     intended?
>     > > > > >     > > >     >
>     > > > > >     > > >     >     2. For compressed messageset, is the
> tombstone
>     > > bit
>     > > > > > only set
>     > > > > >     > at
>     > > > > >     > > > the
>     > > > > >     > > >     > shallow
>     > > > > >     > > >     >     level? Do we always leave that bit in the
>     > wrapper
>     > > > > > message
>     > > > > >     > > unset?
>     > > > > >     > > > An
>     > > > > >     > > >     >     alternative is to set the tombstone bit
> in the
>     > > > > wrapper
>     > > > > > if at
>     > > > > >     > > > least one
>     > > > > >     > > >     >     inner message has the tombstone bit set.
> This
>     > > makes
>     > > > > > things a
>     > > > > >     > > bit
>     > > > > >     > > > more
>     > > > > >     > > >     >     complicated, but we could potentially
> exploit
>     > > that
>     > > > > for
>     > > > > >     > > > optimizing down
>     > > > > >     > > >     >     conversion. For example, we only need to
>     > convert
>     > > > > > messages
>     > > > > >     > with
>     > > > > >     > > > magic 2
>     > > > > >     > > >     > to
>     > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit
> is set
>     > > > > > (conversion is
>     > > > > >     > > > always
>     > > > > >     > > >     > needed
>     > > > > >     > > >     >     from magic 2 to magic 0). Not sure if the
>     > > > > optimization
>     > > > > > is
>     > > > > >     > worth
>     > > > > >     > > > the
>     > > > > >     > > >     >     complexity though.
>     > > > > >     > > >     >
>     > > > > >     > > >     >     3. The referencing of the new version of
>     > > > > >     > > > ProducerRequest/FetchRequest
>     > > > > >     > > >     > is
>     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> convention
>     > > > starts
>     > > > > at
>     > > > > >     > version
>     > > > > >     > > > at 0, I
>     > > > > >     > > >     >     think the new version would be 3.
>     > > > > >     > > >     >
>     > > > > >     > > >     >     4. "If the magic byte on message is 2,
> the
>     > broker
>     > > > > > should use
>     > > > > >     > > the
>     > > > > >     > > >     > tombstone
>     > > > > >     > > >     >     bit for log compaction." What about null
> value?
>     > > My
>     > > > > >     > > understanding
>     > > > > >     > > > is
>     > > > > >     > > >     > that
>     > > > > >     > > >     >     null value will be treated the same as
> setting
>     > > the
>     > > > > > tombstone
>     > > > > >     > > bit.
>     > > > > >     > > >     >
>     > > > > >     > > >     >     5. For the migration path, it would be
> useful
>     > to
>     > > > > > describe the
>     > > > > >     > > > down
>     > > > > >     > > >     >     conversion path to consumers (i.e.,
> brokers on
>     > > > > message
>     > > > > > format
>     > > > > >     > > > 0.10.2
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     consumers on older version).
>     > > > > >     > > >     >
>     > > > > >     > > >     >     Thanks,
>     > > > > >     > > >     >
>     > > > > >     > > >     >     Jun
>     > > > > >     > > >     >
>     > > > > >     > > >     >
>     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael
>     > Pearce <
>     > > > > >     > > > Michael.Pearce@ig.com
>     > > > > >     > > >     > >
>     > > > > >     > > >     >     wrote:
>     > > > > >     > > >     >
>     > > > > >     > > >     >     > Hi All,
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > We have been discussing in the below
> thread
>     > and
>     > > > > final
>     > > > > >     > changes
>     > > > > >     > > > have
>     > > > > >     > > >     > been
>     > > > > >     > > >     >     > made to the KIP wiki based on these
>     > > discussions.
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > We would now like to put to the vote
> the
>     > > > following
>     > > > > > KIP:
>     > > > > >     > > >     >     > https://cwiki.apache.org/
>     > > > > > confluence/display/KAFKA/KIP-
>     > > > > >     > 87+-+
>     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > This kip is for having a distinct
> compaction
>     > > > > > attribute
>     > > > > >     > > > “tombstone”
>     > > > > >     > > >     > flag
>     > > > > >     > > >     >     > instead of relying on null value,
> allowing
>     > > > non-null
>     > > > > > value
>     > > > > >     > > > delete
>     > > > > >     > > >     > messages.
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > Many thanks,
>     > > > > >     > > >     >     > Michael
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce"
> <
>     > > > > >     > > Michael.Pearce@ig.com>
>     > > > > >     > > >     > wrote:
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     Hi Mayuresh,
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     LGTM. Ive just made one small
> adjustment
>     > > > > > updating the
>     > > > > >     > > wire
>     > > > > >     > > >     > protocol to
>     > > > > >     > > >     >     > show the magic byte bump.
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     Do we think we’re good to put to a
> vote?
>     > Is
>     > > > > > there any
>     > > > > >     > > > other bits
>     > > > > >     > > >     >     > needing discussion?
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     Cheers
>     > > > > >     > > >     >     >     Mike
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
> Gharat" <
>     > > > > >     > > >     > gharatmayuresh15@gmail.com>
>     > > > > >     > > >     >     > wrote:
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         Hi Michael,
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         I have updated the migration
> section
>     > of
>     > > > the
>     > > > > > KIP.
>     > > > > >     > Can
>     > > > > >     > > > you
>     > > > > >     > > >     > please
>     > > > > >     > > >     >     > take a look?
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         Thanks,
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         Mayuresh
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07
> AM,
>     > > Mayuresh
>     > > > > > Gharat <
>     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
>     > > > > >     > > >     >     >         > wrote:
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         > Hi Michael,
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > That whilst sending
> tombstone and
>     > non
>     > > > > null
>     > > > > > value,
>     > > > > >     > > the
>     > > > > >     > > >     > consumer
>     > > > > >     > > >     >     > can expect
>     > > > > >     > > >     >     >         > only to receive the non-null
>     > message
>     > > > only
>     > > > > > in step
>     > > > > >     > > > (3) is
>     > > > > >     > > >     > this
>     > > > > >     > > >     >     > correct?
>     > > > > >     > > >     >     >         > ---> I do agree with you
> here.
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > Becket, Ismael : can you guys
>     > review
>     > > > the
>     > > > > >     > migration
>     > > > > >     > > > plan
>     > > > > >     > > >     > listed
>     > > > > >     > > >     >     > above using
>     > > > > >     > > >     >     >         > magic byte?
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > Thanks,
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > Mayuresh
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58
> AM,
>     > > > Michael
>     > > > > > Pearce <
>     > > > > >     > > >     >     > Michael.Pearce@ig.com>
>     > > > > >     > > >     >     >         > wrote:
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         >> Many thanks for this
> Mayuresh. I
>     > > don't
>     > > > > > have any
>     > > > > >     > > >     > objections.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> I assume we should state:
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> That whilst sending
> tombstone and
>     > > non
>     > > > > null
>     > > > > >     > value,
>     > > > > >     > > > the
>     > > > > >     > > >     > consumer
>     > > > > >     > > >     >     > can expect
>     > > > > >     > > >     >     >         >> only to receive the non-null
>     > message
>     > > > > only
>     > > > > > in
>     > > > > >     > step
>     > > > > >     > > > (3) is
>     > > > > >     > > >     > this
>     > > > > >     > > >     >     > correct?
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Cheers
>     > > > > >     > > >     >     >         >> Mike
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Sent using OWA for iPhone
>     > > > > >     > > >     >     >         >>
> ______________________________
>     > > > > __________
>     > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
>     > > > > >     > gharatmayuresh15@gmail.com
>     > > > > >     > > >
>     > > > > >     > > >     >     >         >> Sent: Thursday, November
> 17, 2016
>     > > > > 5:18:41
>     > > > > > PM
>     > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
>     > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> KIP-87 -
>     > Add
>     > > > > > Compaction
>     > > > > >     > > > Tombstone
>     > > > > >     > > >     > Flag
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Hi Ismael,
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Thanks for the explanation.
>     > > > > >     > > >     >     >         >> Specially I like this part
> where
>     > in
>     > > > you
>     > > > > >     > mentioned
>     > > > > >     > > > we can
>     > > > > >     > > >     > get
>     > > > > >     > > >     >     > rid of the
>     > > > > >     > > >     >     >         >> older null value support
> for log
>     > > > > > compaction
>     > > > > >     > later
>     > > > > >     > > > on,
>     > > > > >     > > >     > here :
>     > > > > >     > > >     >     >         >> We can't change semantics
> of the
>     > > > message
>     > > > > > format
>     > > > > >     > > > without
>     > > > > >     > > >     > having
>     > > > > >     > > >     >     > a long
>     > > > > >     > > >     >     >         >> transition period. And we
> can't
>     > rely
>     > > > > >     > > >     >     >         >> on people reading
> documentation or
>     > > > > acting
>     > > > > > on a
>     > > > > >     > > > warning for
>     > > > > >     > > >     >     > something so
>     > > > > >     > > >     >     >         >> fundamental. As such, my
> take is
>     > > that
>     > > > we
>     > > > > > need to
>     > > > > >     > > > bump the
>     > > > > >     > > >     > magic
>     > > > > >     > > >     >     > byte. The
>     > > > > >     > > >     >     >         >> good news is
>     > > > > >     > > >     >     >         >> that we don't have to
> support all
>     > > > > versions
>     > > > > >     > > forever.
>     > > > > >     > > > We
>     > > > > >     > > >     > have
>     > > > > >     > > >     >     > said that we
>     > > > > >     > > >     >     >         >> will support direct
> upgrades for 2
>     > > > > years.
>     > > > > > That
>     > > > > >     > > > means that
>     > > > > >     > > >     >     > message format
>     > > > > >     > > >     >     >         >> version n could, in theory,
> be
>     > > > removed 2
>     > > > > > years
>     > > > > >     > > > after the
>     > > > > >     > > >     > it's
>     > > > > >     > > >     >     > introduced.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Just a heads up, I would
> like to
>     > > > mention
>     > > > > > that
>     > > > > >     > even
>     > > > > >     > > > without
>     > > > > >     > > >     >     > bumping magic
>     > > > > >     > > >     >     >         >> byte, we will *NOT* loose
> zero
>     > copy
>     > > as
>     > > > > in
>     > > > > > the
>     > > > > >     > > > client(x+1)
>     > > > > >     > > >     > in my
>     > > > > >     > > >     >     >         >> explanation
>     > > > > >     > > >     >     >         >> above will convert
> internally a
>     > null
>     > > > > > value to
>     > > > > >     > > have a
>     > > > > >     > > >     > tombstone
>     > > > > >     > > >     >     > bit set and
>     > > > > >     > > >     >     >         >> a tombstone bit set to have
> a null
>     > > > value
>     > > > > >     > > > automatically
>     > > > > >     > > >     >     > internally and by
>     > > > > >     > > >     >     >         >> the time we move to version
> (x+2),
>     > > the
>     > > > > > clients
>     > > > > >     > > > would have
>     > > > > >     > > >     >     > upgraded.
>     > > > > >     > > >     >     >         >> Obviously if we support a
> request
>     > > from
>     > > > > >     > > consumer(x),
>     > > > > >     > > > we
>     > > > > >     > > >     > will
>     > > > > >     > > >     >     > loose zero
>     > > > > >     > > >     >     >         >> copy
>     > > > > >     > > >     >     >         >> but that is the same case
> with
>     > magic
>     > > > > byte.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> But if magic byte bump
> makes life
>     > > > easier
>     > > > > > for
>     > > > > >     > > > transition
>     > > > > >     > > >     > for the
>     > > > > >     > > >     >     > above
>     > > > > >     > > >     >     >         >> reasons that you explained,
> I am
>     > OK
>     > > > with
>     > > > > > it
>     > > > > >     > since
>     > > > > >     > > > we are
>     > > > > >     > > >     > going
>     > > > > >     > > >     >     > to meet the
>     > > > > >     > > >     >     >         >> end goal down the road :)
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> On a side note can we
> update the
>     > doc
>     > > > > here
>     > > > > > on
>     > > > > >     > magic
>     > > > > >     > > > byte
>     > > > > >     > > >     > to say
>     > > > > >     > > >     >     > that "*it
>     > > > > >     > > >     >     >         >> should be bumped whenever
> the
>     > > message
>     > > > > > format is
>     > > > > >     > > > changed
>     > > > > >     > > >     > or the
>     > > > > >     > > >     >     >         >> interpretation of message
> format
>     > > > (usage
>     > > > > > of the
>     > > > > >     > > > reserved
>     > > > > >     > > >     > bits as
>     > > > > >     > > >     >     > well) is
>     > > > > >     > > >     >     >         >> changed*".
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Hi Michael,
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Here is the update plan
> that we
>     > > > > discussed
>     > > > > >     > offline
>     > > > > >     > > >     > yesterday :
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Currently the magic-byte
> which
>     > > > > > corresponds to
>     > > > > >     > the
>     > > > > >     > > >     >     > "message.format.version"
>     > > > > >     > > >     >     >         >> is set to 1.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> 1) On broker it will be set
> to 1
>     > > > > > initially.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> 2) When a producer client
> sends a
>     > > > > message
>     > > > > > with
>     > > > > >     > > > magic-byte
>     > > > > >     > > >     > = 2,
>     > > > > >     > > >     >     > since the
>     > > > > >     > > >     >     >         >> broker is on magic-byte =
> 1, we
>     > will
>     > > > > down
>     > > > > >     > convert
>     > > > > >     > > > it,
>     > > > > >     > > >     > which
>     > > > > >     > > >     >     > means if the
>     > > > > >     > > >     >     >         >> tombstone bit is set, the
> value
>     > will
>     > > > be
>     > > > > > set to
>     > > > > >     > > > null. A
>     > > > > >     > > >     > consumer
>     > > > > >     > > >     >     >         >> understanding magic-byte =
> 1, will
>     > > > still
>     > > > > > work
>     > > > > >     > with
>     > > > > >     > > > this. A
>     > > > > >     > > >     >     > consumer
>     > > > > >     > > >     >     >         >> working
>     > > > > >     > > >     >     >         >> with magic-byte =2 will
> also be
>     > able
>     > > > to
>     > > > > >     > understand
>     > > > > >     > > > this,
>     > > > > >     > > >     > since
>     > > > > >     > > >     >     > it
>     > > > > >     > > >     >     >         >> understands the tombstone.
>     > > > > >     > > >     >     >         >> Now there is still the
> question of
>     > > > > > supporting a
>     > > > > >     > > >     > non-tombstone
>     > > > > >     > > >     >     > and null
>     > > > > >     > > >     >     >         >> value from producer client
> with
>     > > > > > magic-byte = 2.*
>     > > > > >     > > (I
>     > > > > >     > > > am
>     > > > > >     > > >     > not sure
>     > > > > >     > > >     >     > if we
>     > > > > >     > > >     >     >         >> should support this.
> Ismael/Becket
>     > > can
>     > > > > > comment
>     > > > > >     > > > here)*
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> 3) When almost all the
> clients
>     > have
>     > > > > > upgraded,
>     > > > > >     > the
>     > > > > >     > > >     >     > message.format.version
>     > > > > >     > > >     >     >         >> on
>     > > > > >     > > >     >     >         >> the broker can be changed
> to 2,
>     > > where
>     > > > in
>     > > > > > the
>     > > > > >     > down
>     > > > > >     > > >     > conversion in
>     > > > > >     > > >     >     > the above
>     > > > > >     > > >     >     >         >> step will not happen. If at
> this
>     > > point
>     > > > > we
>     > > > > > get a
>     > > > > >     > > > consumer
>     > > > > >     > > >     >     > request from a
>     > > > > >     > > >     >     >         >> older consumer, we might
> have to
>     > > down
>     > > > > > convert
>     > > > > >     > > where
>     > > > > >     > > > in we
>     > > > > >     > > >     > loose
>     > > > > >     > > >     >     > zero copy,
>     > > > > >     > > >     >     >         >> but these cases should be
> rare.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Becket can you review this
> plan
>     > and
>     > > > add
>     > > > > > more
>     > > > > >     > > > details if I
>     > > > > >     > > >     > have
>     > > > > >     > > >     >     >         >> missed/wronged something,
> before
>     > we
>     > > > put
>     > > > > > it on
>     > > > > >     > KIP.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Thanks,
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Mayuresh
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> 11:07 PM,
>     > > > > Michael
>     > > > > >     > Pearce <
>     > > > > >     > > >     >     > Michael.Pearce@ig.com>
>     > > > > >     > > >     >     >         >> wrote:
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> > Thanks guys, for
> discussing this
>     > > > > > offline and
>     > > > > >     > > > getting
>     > > > > >     > > >     > some
>     > > > > >     > > >     >     > consensus.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > So its clear for myself
> and
>     > others
>     > > > > what
>     > > > > > is
>     > > > > >     > > > proposed now
>     > > > > >     > > >     > (i
>     > > > > >     > > >     >     > think i
>     > > > > >     > > >     >     >         >> > understand, but want to
> make
>     > sure)
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Could i ask either
> directly
>     > update
>     > > > the
>     > > > > > kip to
>     > > > > >     > > > detail the
>     > > > > >     > > >     >     > migration
>     > > > > >     > > >     >     >         >> > strategy, or (re-)state
> your
>     > > offline
>     > > > > > discussed
>     > > > > >     > > and
>     > > > > >     > > >     > agreed
>     > > > > >     > > >     >     > migration
>     > > > > >     > > >     >     >         >> > strategy based on a magic
> byte
>     > is
>     > > in
>     > > > > > this
>     > > > > >     > > thread.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > The main original driver
> for the
>     > > KIP
>     > > > > > was to
>     > > > > >     > > > support
>     > > > > >     > > >     >     > compaction where
>     > > > > >     > > >     >     >         >> value
>     > > > > >     > > >     >     >         >> > isn't null, based off the
>     > > > discussions
>     > > > > on
>     > > > > >     > KIP-82
>     > > > > >     > > > thread.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > We should be able to
> support
>     > > > > > non-tombstone +
>     > > > > >     > > null
>     > > > > >     > > > value
>     > > > > >     > > >     > by the
>     > > > > >     > > >     >     >         >> completion
>     > > > > >     > > >     >     >         >> > of the KIP, as we noted
> when
>     > > > > discussing
>     > > > > > this
>     > > > > >     > > kip,
>     > > > > >     > > > having
>     > > > > >     > > >     >     > logic based on
>     > > > > >     > > >     >     >         >> a
>     > > > > >     > > >     >     >         >> > null value isn't very
> clean and
>     > > also
>     > > > > > separates
>     > > > > >     > > the
>     > > > > >     > > >     > concerns.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > As discussed already
> though we
>     > can
>     > > > > > split this
>     > > > > >     > > into
>     > > > > >     > > >     > KIP-87a
>     > > > > >     > > >     >     > and KIP-87b
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Where we look to deliver
> KIP-87a
>     > > on
>     > > > a
>     > > > > >     > compacted
>     > > > > >     > > > topic
>     > > > > >     > > >     > (to
>     > > > > >     > > >     >     > address the
>     > > > > >     > > >     >     >         >> > immediate issues)
>     > > > > >     > > >     >     >         >> > * tombstone + null value
>     > > > > >     > > >     >     >         >> > * tombstone + non-null
> value
>     > > > > >     > > >     >     >         >> > * non-tombstone +
> non-null value
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Then we can discuss once
> KIP-87a
>     > > is
>     > > > > > completed
>     > > > > >     > > > options
>     > > > > >     > > >     > later
>     > > > > >     > > >     >     > and how we
>     > > > > >     > > >     >     >         >> > support the second part
> KIP-87b
>     > to
>     > > > > > deliver:
>     > > > > >     > > >     >     >         >> > * non-tombstone + null
> value
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Cheers
>     > > > > >     > > >     >     >         >> > Mike
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
> ______________________________
>     > > > > > __________
>     > > > > >     > > >     >     >         >> > From: Becket Qin <
>     > > > > becket.qin@gmail.com>
>     > > > > >     > > >     >     >         >> > Sent: Thursday, November
> 17,
>     > 2016
>     > > > 1:43
>     > > > > > AM
>     > > > > >     > > >     >     >         >> > To: dev@kafka.apache.org
>     > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> KIP-87 -
>     > > Add
>     > > > > > Compaction
>     > > > > >     > > >     > Tombstone Flag
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had
> an
>     > > offline
>     > > > > >     > discussion,
>     > > > > >     > > > and
>     > > > > >     > > >     > following
>     > > > > >     > > >     >     > is a brief
>     > > > > >     > > >     >     >         >> > summary.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > 1. We agreed that not
> bumping up
>     > > > magic
>     > > > > > value
>     > > > > >     > may
>     > > > > >     > > > result
>     > > > > >     > > >     > in
>     > > > > >     > > >     >     > losing zero
>     > > > > >     > > >     >     >         >> copy
>     > > > > >     > > >     >     >         >> > during migration.
>     > > > > >     > > >     >     >         >> > 2. Given that bumping up
> magic
>     > > value
>     > > > > is
>     > > > > > almost
>     > > > > >     > > > free and
>     > > > > >     > > >     > has
>     > > > > >     > > >     >     > benefit of
>     > > > > >     > > >     >     >         >> > avoiding potential
> performance
>     > > > issue.
>     > > > > > It is
>     > > > > >     > > > probably
>     > > > > >     > > >     > worth
>     > > > > >     > > >     >     > doing.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > One issue we still need
> to think
>     > > > about
>     > > > > > is
>     > > > > >     > > whether
>     > > > > >     > > > we
>     > > > > >     > > >     > want to
>     > > > > >     > > >     >     > support a
>     > > > > >     > > >     >     >         >> > non-tombstone message
> with null
>     > > > value.
>     > > > > >     > > >     >     >         >> > Currently it is not
> supported by
>     > > > > Kafka.
>     > > > > > If we
>     > > > > >     > > > allow a
>     > > > > >     > > >     >     > non-tombstone null
>     > > > > >     > > >     >     >         >> > value message to exist
> after
>     > > KIP-87.
>     > > > > The
>     > > > > >     > problem
>     > > > > >     > > > is
>     > > > > >     > > >     > that such
>     > > > > >     > > >     >     > message
>     > > > > >     > > >     >     >         >> will
>     > > > > >     > > >     >     >         >> > not be supported by the
>     > consumers
>     > > > > prior
>     > > > > > to
>     > > > > >     > > KIP-87.
>     > > > > >     > > >     > Because a
>     > > > > >     > > >     >     > null value
>     > > > > >     > > >     >     >         >> > will always be
> interpreted to a
>     > > > > > tombstone.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > One option is that we
> keep the
>     > > > current
>     > > > > > way,
>     > > > > >     > i.e.
>     > > > > >     > > > do not
>     > > > > >     > > >     >     > support such
>     > > > > >     > > >     >     >         >> > message. It would be good
> to
>     > know
>     > > if
>     > > > > > there is
>     > > > > >     > a
>     > > > > >     > > >     > concrete use
>     > > > > >     > > >     >     > case for
>     > > > > >     > > >     >     >         >> such
>     > > > > >     > > >     >     >         >> > message. If there is not,
> we can
>     > > > > > probably just
>     > > > > >     > > not
>     > > > > >     > > >     > support it.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Thanks,
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at
> 1:28 PM,
>     > > > > > Mayuresh
>     > > > > >     > > Gharat <
>     > > > > >     > > >     >     >         >> >
> gharatmayuresh15@gmail.com
>     > > > > >     > > >     >     >         >> > > wrote:
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > > Hi Ismael,
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > This is something I can
> think
>     > of
>     > > > for
>     > > > > >     > migration
>     > > > > >     > > > plan:
>     > > > > >     > > >     >     >         >> > > So the migration plan
> can look
>     > > > > > something
>     > > > > >     > like
>     > > > > >     > > > this,
>     > > > > >     > > >     > with up
>     > > > > >     > > >     >     >         >> conversion :
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > 1) Currently lets say
> we have
>     > > > Broker
>     > > > > > at
>     > > > > >     > > version
>     > > > > >     > > > x.
>     > > > > >     > > >     >     >         >> > > 2) Currently we have
> clients
>     > at
>     > > > > > version x.
>     > > > > >     > > >     >     >         >> > > 3) a) We move the
> version to
>     > > > > > Broker(x+1) :
>     > > > > >     > > > supports
>     > > > > >     > > >     > both
>     > > > > >     > > >     >     > tombstone and
>     > > > > >     > > >     >     >         >> > null
>     > > > > >     > > >     >     >         >> > > for log compaction.
>     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> client
>     > to
>     > > > > > version
>     > > > > >     > > > client(x+1) :
>     > > > > >     > > >     > if in
>     > > > > >     > > >     >     > the
>     > > > > >     > > >     >     >         >> producer
>     > > > > >     > > >     >     >         >> > > client(x+1) the value
> is set
>     > to
>     > > > > null,
>     > > > > > we
>     > > > > >     > will
>     > > > > >     > > >     > automatically
>     > > > > >     > > >     >     > set the
>     > > > > >     > > >     >     >         >> > > Tombstone bit
> internally. If
>     > the
>     > > > > > producer
>     > > > > >     > > > client(x+1)
>     > > > > >     > > >     > sets
>     > > > > >     > > >     >     > the
>     > > > > >     > > >     >     >         >> tombstone
>     > > > > >     > > >     >     >         >> > > itself, well and good.
> For
>     > > > producer
>     > > > > >     > client(x),
>     > > > > >     > > > the
>     > > > > >     > > >     > broker
>     > > > > >     > > >     >     > will up
>     > > > > >     > > >     >     >         >> convert
>     > > > > >     > > >     >     >         >> > > to have the tombstone
> bit.
>     > > > > > Broker(x+1) is
>     > > > > >     > > > supporting
>     > > > > >     > > >     > both.
>     > > > > >     > > >     >     > Consumer
>     > > > > >     > > >     >     >         >> > > client(x+1) will be
> aware of
>     > > this
>     > > > > and
>     > > > > > should
>     > > > > >     > > be
>     > > > > >     > > > able
>     > > > > >     > > >     > to
>     > > > > >     > > >     >     > handle this.
>     > > > > >     > > >     >     >         >> For
>     > > > > >     > > >     >     >         >> > > consumer client(x) we
> will
>     > down
>     > > > > > convert the
>     > > > > >     > > > message
>     > > > > >     > > >     > on the
>     > > > > >     > > >     >     > broker
>     > > > > >     > > >     >     >         >> side.
>     > > > > >     > > >     >     >         >> > >     c) At this point we
> will
>     > > have
>     > > > to
>     > > > > >     > specify a
>     > > > > >     > > >     > warning or
>     > > > > >     > > >     >     > clearly
>     > > > > >     > > >     >     >         >> specify
>     > > > > >     > > >     >     >         >> > > in docs that this
> behavior is
>     > > > about
>     > > > > > to be
>     > > > > >     > > > changed for
>     > > > > >     > > >     > log
>     > > > > >     > > >     >     > compaction.
>     > > > > >     > > >     >     >         >> > > 4) a) In next release
> of the
>     > > > > > Broker(x+2), we
>     > > > > >     > > > say that
>     > > > > >     > > >     > only
>     > > > > >     > > >     >     > Tombstone
>     > > > > >     > > >     >     >         >> is
>     > > > > >     > > >     >     >         >> > > used for log compaction
> on the
>     > > > > Broker
>     > > > > > side.
>     > > > > >     > > >     > Clients(x+1)
>     > > > > >     > > >     >     > still is
>     > > > > >     > > >     >     >         >> > > supported.
>     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> client
>     > to
>     > > > > > version
>     > > > > >     > > > client(x+2) :
>     > > > > >     > > >     > if
>     > > > > >     > > >     >     > value is set
>     > > > > >     > > >     >     >         >> to
>     > > > > >     > > >     >     >         >> > > null, tombstone will
> not be
>     > set
>     > > > > >     > automatically.
>     > > > > >     > > > The
>     > > > > >     > > >     > client
>     > > > > >     > > >     >     > will have to
>     > > > > >     > > >     >     >         >> > call
>     > > > > >     > > >     >     >         >> > > setTombstone() to
> actually set
>     > > the
>     > > > > >     > tombstone.
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > We should compare this
>     > migration
>     > > > > plan
>     > > > > > with
>     > > > > >     > the
>     > > > > >     > > >     > migration
>     > > > > >     > > >     >     > plan for
>     > > > > >     > > >     >     >         >> magic
>     > > > > >     > > >     >     >         >> > > byte bump and do
> whatever
>     > looks
>     > > > > good.
>     > > > > >     > > >     >     >         >> > > I am just worried that
> if we
>     > go
>     > > > down
>     > > > > > magic
>     > > > > >     > > byte
>     > > > > >     > > > route,
>     > > > > >     > > >     >     > unless I am
>     > > > > >     > > >     >     >         >> > missing
>     > > > > >     > > >     >     >         >> > > something, it sounds
> like
>     > kafka
>     > > > will
>     > > > > > be
>     > > > > >     > stuck
>     > > > > >     > > > with
>     > > > > >     > > >     >     > supporting both
>     > > > > >     > > >     >     >         >> null
>     > > > > >     > > >     >     >         >> > > value and tombstone bit
> for
>     > log
>     > > > > > compaction
>     > > > > >     > for
>     > > > > >     > > > life
>     > > > > >     > > >     > long,
>     > > > > >     > > >     >     > which does
>     > > > > >     > > >     >     >         >> not
>     > > > > >     > > >     >     >         >> > > look like a good end
> state.
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > Thanks,
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > Mayuresh
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at
> 9:32
>     > AM,
>     > > > > > Mayuresh
>     > > > > >     > > > Gharat <
>     > > > > >     > > >     >     >         >> > >
> gharatmayuresh15@gmail.com
>     > > > > >     > > >     >     >         >> > > > wrote:
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > > Hi Ismael,
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > That's a very good
> point
>     > > which I
>     > > > > > might
>     > > > > >     > have
>     > > > > >     > > > not
>     > > > > >     > > >     >     > considered earlier.
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > Here is a plan that I
> can
>     > > think
>     > > > > of:
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > Stage 1) The broker
> from now
>     > > on,
>     > > > > up
>     > > > > >     > converts
>     > > > > >     > > > the
>     > > > > >     > > >     > message
>     > > > > >     > > >     >     > to have the
>     > > > > >     > > >     >     >         >> > > > tombstone marker. The
> log
>     > > > > compaction
>     > > > > >     > thread
>     > > > > >     > > > does log
>     > > > > >     > > >     >     > compaction
>     > > > > >     > > >     >     >         >> based
>     > > > > >     > > >     >     >         >> > on
>     > > > > >     > > >     >     >         >> > > > both null and
> tombstone
>     > > marker.
>     > > > > > This is
>     > > > > >     > our
>     > > > > >     > > >     > transition
>     > > > > >     > > >     >     > period.
>     > > > > >     > > >     >     >         >> > > > Stage 2) The next
> release we
>     > > > only
>     > > > > > say that
>     > > > > >     > > log
>     > > > > >     > > >     > compaction
>     > > > > >     > > >     >     > is based
>     > > > > >     > > >     >     >         >> on
>     > > > > >     > > >     >     >         >> > > > tombstone marker.
> (Open
>     > source
>     > > > > > kafka makes
>     > > > > >     > > > this as a
>     > > > > >     > > >     >     > policy). By
>     > > > > >     > > >     >     >         >> this
>     > > > > >     > > >     >     >         >> > > time,
>     > > > > >     > > >     >     >         >> > > > the organization
> which is
>     > > moving
>     > > > > to
>     > > > > > this
>     > > > > >     > > > release
>     > > > > >     > > >     > will be
>     > > > > >     > > >     >     > sure that
>     > > > > >     > > >     >     >         >> they
>     > > > > >     > > >     >     >         >> > > > have gone through the
> entire
>     > > > > > transition
>     > > > > >     > > > period.
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > My only goal of doing
> this
>     > is
>     > > > that
>     > > > > > Kafka
>     > > > > >     > > > clearly
>     > > > > >     > > >     >     > specifies the end
>     > > > > >     > > >     >     >         >> > state
>     > > > > >     > > >     >     >         >> > > > about what log
> compaction
>     > > means
>     > > > > (is
>     > > > > > it
>     > > > > >     > null
>     > > > > >     > > > value
>     > > > > >     > > >     > or a
>     > > > > >     > > >     >     > tombstone
>     > > > > >     > > >     >     >         >> > marker,
>     > > > > >     > > >     >     >         >> > > > but not both).
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > What do you think?
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > Thanks,
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > Mayuresh
>     > > > > >     > > >     >     >         >> > > > .
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016
> at 9:17
>     > > AM,
>     > > > > > Ismael
>     > > > > >     > > Juma <
>     > > > > >     > > >     >     > ismael@juma.me.uk>
>     > > > > >     > > >     >     >         >> > wrote:
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > >> One comment below.
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016
> at
>     > 5:08
>     > > > PM,
>     > > > > > Mayuresh
>     > > > > >     > > > Gharat <
>     > > > > >     > > >     >     >         >> > > >>
> gharatmayuresh15@gmail.com
>     > > > > >     > > >     >     >         >> > > >> > wrote:
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >> >    - If we don't
> bump up
>     > > the
>     > > > > > magic
>     > > > > >     > byte,
>     > > > > >     > > > on the
>     > > > > >     > > >     > broker
>     > > > > >     > > >     >     > side, the
>     > > > > >     > > >     >     >         >> > > broker
>     > > > > >     > > >     >     >         >> > > >> >    will always
> have to
>     > look
>     > > > at
>     > > > > > both
>     > > > > >     > > > tombstone
>     > > > > >     > > >     > bit and
>     > > > > >     > > >     >     > the value
>     > > > > >     > > >     >     >         >> when
>     > > > > >     > > >     >     >         >> > > do
>     > > > > >     > > >     >     >         >> > > >> the
>     > > > > >     > > >     >     >         >> > > >> >    compaction.
> Assuming
>     > we
>     > > do
>     > > > > > not bump
>     > > > > >     > up
>     > > > > >     > > > the
>     > > > > >     > > >     > magic
>     > > > > >     > > >     >     > byte,
>     > > > > >     > > >     >     >         >> > > >> >    imagine the
> broker
>     > sees
>     > > a
>     > > > > > message
>     > > > > >     > > which
>     > > > > >     > > > does
>     > > > > >     > > >     > not
>     > > > > >     > > >     >     > have a
>     > > > > >     > > >     >     >         >> tombstone
>     > > > > >     > > >     >     >         >> > > bit
>     > > > > >     > > >     >     >         >> > > >> >    set. The broker
> does
>     > not
>     > > > > know
>     > > > > > when
>     > > > > >     > the
>     > > > > >     > > >     > message was
>     > > > > >     > > >     >     > produced
>     > > > > >     > > >     >     >         >> (i.e.
>     > > > > >     > > >     >     >         >> > > >> > whether
>     > > > > >     > > >     >     >         >> > > >> >    the message has
> been
>     > up
>     > > > > > converted or
>     > > > > >     > > > not), it
>     > > > > >     > > >     > has
>     > > > > >     > > >     >     > to take a
>     > > > > >     > > >     >     >         >> > further
>     > > > > >     > > >     >     >         >> > > >> > look at
>     > > > > >     > > >     >     >         >> > > >> >    the value to
> see if it
>     > > is
>     > > > > > null or
>     > > > > >     > not
>     > > > > >     > > in
>     > > > > >     > > >     > order to
>     > > > > >     > > >     >     > determine
>     > > > > >     > > >     >     >         >> if it
>     > > > > >     > > >     >     >         >> > > is
>     > > > > >     > > >     >     >         >> > > >> a
>     > > > > >     > > >     >     >         >> > > >> >    tombstone. The
> same
>     > > logic
>     > > > > has
>     > > > > > to be
>     > > > > >     > > put
>     > > > > >     > > > on the
>     > > > > >     > > >     >     > consumer as
>     > > > > >     > > >     >     >         >> well
>     > > > > >     > > >     >     >         >> > > >> because
>     > > > > >     > > >     >     >         >> > > >> > the
>     > > > > >     > > >     >     >         >> > > >> >    consumer does
> not know
>     > > if
>     > > > > the
>     > > > > >     > message
>     > > > > >     > > > has
>     > > > > >     > > >     > been up
>     > > > > >     > > >     >     > converted or
>     > > > > >     > > >     >     >         >> > not.
>     > > > > >     > > >     >     >         >> > > >> >       - If we
> upconvert
>     > > while
>     > > > > >     > appending,
>     > > > > >     > > > this is
>     > > > > >     > > >     > not
>     > > > > >     > > >     >     > the case,
>     > > > > >     > > >     >     >         >> > right?
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >> If I understand you
>     > > correctly,
>     > > > > > this is
>     > > > > >     > not
>     > > > > >     > > >     > sufficient
>     > > > > >     > > >     >     > because the
>     > > > > >     > > >     >     >         >> log
>     > > > > >     > > >     >     >         >> > > may
>     > > > > >     > > >     >     >         >> > > >> have messages
> appended
>     > before
>     > > > it
>     > > > > > was
>     > > > > >     > > > upgraded to
>     > > > > >     > > >     > include
>     > > > > >     > > >     >     > KIP-87.
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >> Ismael
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > --
>     > > > > >     > > >     >     >         >> > > > -Regards,
>     > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
>     > > > > >     > > >     >     >         >> > > > (862) 250-7125
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > --
>     > > > > >     > > >     >     >         >> > > -Regards,
>     > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
>     > > > > >     > > >     >     >         >> > > (862) 250-7125
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > The information contained
> in
>     > this
>     > > > > email
>     > > > > > is
>     > > > > >     > > > strictly
>     > > > > >     > > >     >     > confidential and for
>     > > > > >     > > >     >     >         >> > the use of the addressee
> only,
>     > > > unless
>     > > > > >     > otherwise
>     > > > > >     > > >     > indicated. If
>     > > > > >     > > >     >     > you are
>     > > > > >     > > >     >     >         >> not
>     > > > > >     > > >     >     >         >> > the intended recipient,
> please
>     > do
>     > > > not
>     > > > > > read,
>     > > > > >     > > copy,
>     > > > > >     > > > use or
>     > > > > >     > > >     >     > disclose to
>     > > > > >     > > >     >     >         >> others
>     > > > > >     > > >     >     >         >> > this message or any
> attachment.
>     > > > Please
>     > > > > > also
>     > > > > >     > > > notify the
>     > > > > >     > > >     > sender
>     > > > > >     > > >     >     > by
>     > > > > >     > > >     >     >         >> replying
>     > > > > >     > > >     >     >         >> > to this email or by
> telephone
>     > > > (+44(020
>     > > > > > 7896
>     > > > > >     > > 0011)
>     > > > > >     > > > and
>     > > > > >     > > >     > then
>     > > > > >     > > >     >     > delete the
>     > > > > >     > > >     >     >         >> email
>     > > > > >     > > >     >     >         >> > and any copies of it.
> Opinions,
>     > > > > > conclusion
>     > > > > >     > (etc)
>     > > > > >     > > > that
>     > > > > >     > > >     > do not
>     > > > > >     > > >     >     > relate to
>     > > > > >     > > >     >     >         >> the
>     > > > > >     > > >     >     >         >> > official business of this
>     > company
>     > > > > shall
>     > > > > > be
>     > > > > >     > > > understood as
>     > > > > >     > > >     >     > neither given
>     > > > > >     > > >     >     >         >> nor
>     > > > > >     > > >     >     >         >> > endorsed by it. IG is a
> trading
>     > > name
>     > > > > of
>     > > > > > IG
>     > > > > >     > > Markets
>     > > > > >     > > >     > Limited (a
>     > > > > >     > > >     >     > company
>     > > > > >     > > >     >     >         >> > registered in England and
> Wales,
>     > > > > company
>     > > > > >     > number
>     > > > > >     > > >     > 04008957) and
>     > > > > >     > > >     >     > IG Index
>     > > > > >     > > >     >     >         >> > Limited (a company
> registered in
>     > > > > > England and
>     > > > > >     > > > Wales,
>     > > > > >     > > >     > company
>     > > > > >     > > >     >     > number
>     > > > > >     > > >     >     >         >> > 01190902). Registered
> address at
>     > > > > Cannon
>     > > > > > Bridge
>     > > > > >     > > > House, 25
>     > > > > >     > > >     >     > Dowgate Hill,
>     > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
> Markets
>     > > > > Limited
>     > > > > >     > > (register
>     > > > > >     > > >     > number
>     > > > > >     > > >     >     > 195355) and IG
>     > > > > >     > > >     >     >         >> > Index Limited (register
> number
>     > > > 114059)
>     > > > > > are
>     > > > > >     > > > authorised
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     > regulated by
>     > > > > >     > > >     >     >         >> the
>     > > > > >     > > >     >     >         >> > Financial Conduct
> Authority.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> --
>     > > > > >     > > >     >     >         >> -Regards,
>     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
>     > > > > >     > > >     >     >         >> (862) 250-7125
>     > > > > >     > > >     >     >         >> The information contained
> in this
>     > > > email
>     > > > > is
>     > > > > >     > > strictly
>     > > > > >     > > >     >     > confidential and for
>     > > > > >     > > >     >     >         >> the use of the addressee
> only,
>     > > unless
>     > > > > > otherwise
>     > > > > >     > > >     > indicated. If
>     > > > > >     > > >     >     > you are not
>     > > > > >     > > >     >     >         >> the intended recipient,
> please do
>     > > not
>     > > > > > read,
>     > > > > >     > copy,
>     > > > > >     > > > use or
>     > > > > >     > > >     >     > disclose to others
>     > > > > >     > > >     >     >         >> this message or any
> attachment.
>     > > Please
>     > > > > > also
>     > > > > >     > notify
>     > > > > >     > > > the
>     > > > > >     > > >     > sender
>     > > > > >     > > >     >     > by replying
>     > > > > >     > > >     >     >         >> to this email or by
> telephone
>     > > (+44(020
>     > > > > > 7896
>     > > > > >     > 0011)
>     > > > > >     > > > and then
>     > > > > >     > > >     >     > delete the email
>     > > > > >     > > >     >     >         >> and any copies of it.
> Opinions,
>     > > > > > conclusion (etc)
>     > > > > >     > > > that do
>     > > > > >     > > >     > not
>     > > > > >     > > >     >     > relate to the
>     > > > > >     > > >     >     >         >> official business of this
> company
>     > > > shall
>     > > > > be
>     > > > > >     > > > understood as
>     > > > > >     > > >     >     > neither given nor
>     > > > > >     > > >     >     >         >> endorsed by it. IG is a
> trading
>     > name
>     > > > of
>     > > > > IG
>     > > > > >     > Markets
>     > > > > >     > > >     > Limited (a
>     > > > > >     > > >     >     > company
>     > > > > >     > > >     >     >         >> registered in England and
> Wales,
>     > > > company
>     > > > > > number
>     > > > > >     > > > 04008957)
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     > IG Index
>     > > > > >     > > >     >     >         >> Limited (a company
> registered in
>     > > > England
>     > > > > > and
>     > > > > >     > > Wales,
>     > > > > >     > > >     > company
>     > > > > >     > > >     >     > number
>     > > > > >     > > >     >     >         >> 01190902). Registered
> address at
>     > > > Cannon
>     > > > > > Bridge
>     > > > > >     > > > House, 25
>     > > > > >     > > >     >     > Dowgate Hill,
>     > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> Markets
>     > > > Limited
>     > > > > >     > (register
>     > > > > >     > > > number
>     > > > > >     > > >     >     > 195355) and IG
>     > > > > >     > > >     >     >         >> Index Limited (register
> number
>     > > 114059)
>     > > > > are
>     > > > > >     > > > authorised and
>     > > > > >     > > >     >     > regulated by the
>     > > > > >     > > >     >     >         >> Financial Conduct Authority.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > --
>     > > > > >     > > >     >     >         > -Regards,
>     > > > > >     > > >     >     >         > Mayuresh R. Gharat
>     > > > > >     > > >     >     >         > (862) 250-7125
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         --
>     > > > > >     > > >     >     >         -Regards,
>     > > > > >     > > >     >     >         Mayuresh R. Gharat
>     > > > > >     > > >     >     >         (862) 250-7125
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     The information contained in this
> email
>     > is
>     > > > > > strictly
>     > > > > >     > > > confidential
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     > for the use of the addressee only,
> unless
>     > > > otherwise
>     > > > > >     > > indicated.
>     > > > > >     > > > If
>     > > > > >     > > >     > you are
>     > > > > >     > > >     >     > not the intended recipient, please do
> not
>     > read,
>     > > > > > copy, use
>     > > > > >     > or
>     > > > > >     > > >     > disclose to
>     > > > > >     > > >     >     > others this message or any attachment.
> Please
>     > > > also
>     > > > > > notify
>     > > > > >     > the
>     > > > > >     > > > sender
>     > > > > >     > > >     > by
>     > > > > >     > > >     >     > replying to this email or by telephone
>     > (+44(020
>     > > > > 7896
>     > > > > > 0011)
>     > > > > >     > > and
>     > > > > >     > > > then
>     > > > > >     > > >     > delete
>     > > > > >     > > >     >     > the email and any copies of it.
> Opinions,
>     > > > > conclusion
>     > > > > > (etc)
>     > > > > >     > > > that do
>     > > > > >     > > >     > not
>     > > > > >     > > >     >     > relate to the official business of this
>     > company
>     > > > > > shall be
>     > > > > >     > > > understood
>     > > > > >     > > >     > as
>     > > > > >     > > >     >     > neither given nor endorsed by it. IG
> is a
>     > > trading
>     > > > > > name of
>     > > > > >     > IG
>     > > > > >     > > > Markets
>     > > > > >     > > >     >     > Limited (a company registered in
> England and
>     > > > Wales,
>     > > > > > company
>     > > > > >     > > > number
>     > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> company
>     > > > > registered
>     > > > > > in
>     > > > > >     > > > England and
>     > > > > >     > > >     > Wales,
>     > > > > >     > > >     >     > company number 01190902). Registered
> address
>     > at
>     > > > > > Cannon
>     > > > > >     > Bridge
>     > > > > >     > > > House,
>     > > > > >     > > >     > 25
>     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG
>     > Markets
>     > > > > > Limited
>     > > > > >     > > > (register
>     > > > > >     > > >     > number
>     > > > > >     > > >     >     > 195355) and IG Index Limited (register
> number
>     > > > > > 114059) are
>     > > > > >     > > > authorised
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     > regulated by the Financial Conduct
> Authority.
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >
>     > > > > >     > > >     >
>     > > > > >     > > >     > The information contained in this email is
> strictly
>     > > > > > confidential
>     > > > > >     > > and
>     > > > > >     > > > for
>     > > > > >     > > >     > the use of the addressee only, unless
> otherwise
>     > > > > indicated.
>     > > > > > If you
>     > > > > >     > > > are not
>     > > > > >     > > >     > the intended recipient, please do not read,
> copy,
>     > use
>     > > > or
>     > > > > > disclose
>     > > > > >     > > to
>     > > > > >     > > > others
>     > > > > >     > > >     > this message or any attachment. Please also
> notify
>     > > the
>     > > > > > sender by
>     > > > > >     > > > replying
>     > > > > >     > > >     > to this email or by telephone (+44(020 7896
> 0011
>     > <020%207896%200011>) and
>     > > > > then
>     > > > > > delete
>     > > > > >     > > > the email
>     > > > > >     > > >     > and any copies of it. Opinions, conclusion
> (etc)
>     > that
>     > > > do
>     > > > > > not
>     > > > > >     > relate
>     > > > > >     > > > to the
>     > > > > >     > > >     > official business of this company shall be
>     > understood
>     > > > as
>     > > > > > neither
>     > > > > >     > > > given nor
>     > > > > >     > > >     > endorsed by it. IG is a trading name of IG
> Markets
>     > > > > Limited
>     > > > > > (a
>     > > > > >     > > company
>     > > > > >     > > >     > registered in England and Wales, company
> number
>     > > > 04008957)
>     > > > > > and IG
>     > > > > >     > > > Index
>     > > > > >     > > >     > Limited (a company registered in England and
> Wales,
>     > > > > company
>     > > > > >     > number
>     > > > > >     > > >     > 01190902). Registered address at Cannon
> Bridge
>     > House,
>     > > > 25
>     > > > > > Dowgate
>     > > > > >     > > > Hill,
>     > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
> (register
>     > > > number
>     > > > > > 195355)
>     > > > > >     > > > and IG
>     > > > > >     > > >     > Index Limited (register number 114059) are
>     > authorised
>     > > > and
>     > > > > >     > regulated
>     > > > > >     > > > by the
>     > > > > >     > > >     > Financial Conduct Authority.
>     > > > > >     > > >     >
>     > > > > >     > > >
>     > > > > >     > > >
>     > > > > >     > > > The information contained in this email is strictly
>     > > > > confidential
>     > > > > > and
>     > > > > >     > for
>     > > > > >     > > > the use of the addressee only, unless otherwise
>     > indicated.
>     > > If
>     > > > > > you are
>     > > > > >     > not
>     > > > > >     > > > the intended recipient, please do not read, copy,
> use or
>     > > > > > disclose to
>     > > > > >     > > others
>     > > > > >     > > > this message or any attachment. Please also notify
> the
>     > > sender
>     > > > > by
>     > > > > >     > replying
>     > > > > >     > > > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>) and then
>     > > > > > delete the
>     > > > > >     > > email
>     > > > > >     > > > and any copies of it. Opinions, conclusion (etc)
> that do
>     > > not
>     > > > > > relate to
>     > > > > >     > > the
>     > > > > >     > > > official business of this company shall be
> understood as
>     > > > > neither
>     > > > > > given
>     > > > > >     > > nor
>     > > > > >     > > > endorsed by it. IG is a trading name of IG Markets
>     > Limited
>     > > (a
>     > > > > > company
>     > > > > >     > > > registered in England and Wales, company number
> 04008957)
>     > > and
>     > > > > IG
>     > > > > > Index
>     > > > > >     > > > Limited (a company registered in England and Wales,
>     > company
>     > > > > > number
>     > > > > >     > > > 01190902). Registered address at Cannon Bridge
> House, 25
>     > > > > Dowgate
>     > > > > > Hill,
>     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > > > > 195355) and
>     > > > > >     > IG
>     > > > > >     > > > Index Limited (register number 114059) are
> authorised and
>     > > > > > regulated by
>     > > > > >     > > the
>     > > > > >     > > > Financial Conduct Authority.
>     > > > > >     > > >
>     > > > > >     > > The information contained in this email is strictly
>     > > > confidential
>     > > > > > and for
>     > > > > >     > > the use of the addressee only, unless otherwise
> indicated.
>     > If
>     > > > you
>     > > > > > are not
>     > > > > >     > > the intended recipient, please do not read, copy,
> use or
>     > > > disclose
>     > > > > > to
>     > > > > >     > others
>     > > > > >     > > this message or any attachment. Please also notify
> the
>     > sender
>     > > > by
>     > > > > > replying
>     > > > > >     > > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>) and then
>     > > > delete
>     > > > > > the
>     > > > > >     > email
>     > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> that do
>     > not
>     > > > > > relate to
>     > > > > >     > the
>     > > > > >     > > official business of this company shall be
> understood as
>     > > > neither
>     > > > > > given
>     > > > > >     > nor
>     > > > > >     > > endorsed by it. IG is a trading name of IG Markets
> Limited
>     > (a
>     > > > > > company
>     > > > > >     > > registered in England and Wales, company number
> 04008957)
>     > and
>     > > > IG
>     > > > > > Index
>     > > > > >     > > Limited (a company registered in England and Wales,
> company
>     > > > > number
>     > > > > >     > > 01190902). Registered address at Cannon Bridge
> House, 25
>     > > > Dowgate
>     > > > > > Hill,
>     > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > > 195355)
>     > > > > > and IG
>     > > > > >     > > Index Limited (register number 114059) are
> authorised and
>     > > > > > regulated by
>     > > > > >     > the
>     > > > > >     > > Financial Conduct Authority.
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > The information contained in this email is strictly
>     > > confidential
>     > > > > and
>     > > > > > for
>     > > > > >     > the use of the addressee only, unless otherwise
> indicated. If
>     > > you
>     > > > > > are not
>     > > > > >     > the intended recipient, please do not read, copy, use
> or
>     > > disclose
>     > > > > to
>     > > > > > others
>     > > > > >     > this message or any attachment. Please also notify the
> sender
>     > > by
>     > > > > > replying
>     > > > > >     > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>) and then
>     > > delete
>     > > > > > the email
>     > > > > >     > and any copies of it. Opinions, conclusion (etc) that
> do not
>     > > > relate
>     > > > > > to the
>     > > > > >     > official business of this company shall be understood
> as
>     > > neither
>     > > > > > given nor
>     > > > > >     > endorsed by it. IG is a trading name of IG Markets
> Limited (a
>     > > > > company
>     > > > > >     > registered in England and Wales, company number
> 04008957) and
>     > > IG
>     > > > > > Index
>     > > > > >     > Limited (a company registered in England and Wales,
> company
>     > > > number
>     > > > > >     > 01190902). Registered address at Cannon Bridge House,
> 25
>     > > Dowgate
>     > > > > > Hill,
>     > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > 195355)
>     > > > > > and IG
>     > > > > >     > Index Limited (register number 114059) are authorised
> and
>     > > > regulated
>     > > > > > by the
>     > > > > >     > Financial Conduct Authority.
>     > > > > >     >
>     > > > > >     >
>     > > > > >
>     > > > > >
>     > > > > > The information contained in this email is strictly
> confidential
>     > and
>     > > > for
>     > > > > > the use of the addressee only, unless otherwise indicated.
> If you
>     > are
>     > > > not
>     > > > > > the intended recipient, please do not read, copy, use or
> disclose
>     > to
>     > > > > others
>     > > > > > this message or any attachment. Please also notify the
> sender by
>     > > > replying
>     > > > > > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>) and then delete the
>     > > > > email
>     > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > > to
>     > > > > the
>     > > > > > official business of this company shall be understood as
> neither
>     > > given
>     > > > > nor
>     > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>     > company
>     > > > > > registered in England and Wales, company number 04008957)
> and IG
>     > > Index
>     > > > > > Limited (a company registered in England and Wales, company
> number
>     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > > Hill,
>     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
>     > and
>     > > > IG
>     > > > > > Index Limited (register number 114059) are authorised and
> regulated
>     > > by
>     > > > > the
>     > > > > > Financial Conduct Authority.
>     > > > > >
>     > > > > The information contained in this email is strictly
> confidential and
>     > > for
>     > > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > > not
>     > > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > > others
>     > > > > this message or any attachment. Please also notify the sender
> by
>     > > replying
>     > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > and then delete the
>     > > > email
>     > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > to
>     > > > the
>     > > > > official business of this company shall be understood as
> neither
>     > given
>     > > > nor
>     > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > > registered in England and Wales, company number 04008957) and
> IG
>     > Index
>     > > > > Limited (a company registered in England and Wales, company
> number
>     > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > Hill,
>     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > > IG
>     > > > > Index Limited (register number 114059) are authorised and
> regulated
>     > by
>     > > > the
>     > > > > Financial Conduct Authority.
>     > > > >
>     > > > The information contained in this email is strictly confidential
> and
>     > for
>     > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > not
>     > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > others
>     > > > this message or any attachment. Please also notify the sender by
>     > replying
>     > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > and then delete the
>     > > email
>     > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > > the
>     > > > official business of this company shall be understood as neither
> given
>     > > nor
>     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > > Limited (a company registered in England and Wales, company
> number
>     > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > IG
>     > > > Index Limited (register number 114059) are authorised and
> regulated by
>     > > the
>     > > > Financial Conduct Authority.
>     > > >
>     > > The information contained in this email is strictly confidential
> and for
>     > > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > > the intended recipient, please do not read, copy, use or disclose
> to
>     > others
>     > > this message or any attachment. Please also notify the sender by
> replying
>     > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > and then delete the email
>     > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > the
>     > > official business of this company shall be understood as neither
> given
>     > nor
>     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > Limited (a company registered in England and Wales, company number
>     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > > Index Limited (register number 114059) are authorised and
> regulated by
>     > the
>     > > Financial Conduct Authority.
>     > >
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jay Kreps <ja...@confluent.io>]

Hey Michael,

I do think it might have been better had we started with a separate concept
of null vs delete. Given that we are where we are, I'm not sure that the
possible cures we explored so far are better than the disease.

I apologize for coming to this late, but I didn't really understand the
implications of the KIP and that we'd be breaking compatibility with
existing apps until the vote had begun, and in my defense I'm not sure the
other folks voting did either.

I think we all agree there are many existing apps that are built with the
assumption of "null value non-tombstone" and it isn't possible to
disambiguate these from tombstones on the producer. It isn't that anyone is
saying we have to support all four possibilities at once, it's that we
simply can't orphan one of the existing combinations or our users will eat
us!

If I've understood your alternate solution of adding another setting for
compaction, I think this does fix the compatibility problem, but it adds an
odd mode the user has to add on all their topics. While the current state
is easily explainable, the resulting state where the meaning of tombstone
and null are overlapping and ambiguous and dependent on a compaction
setting that could change out of band or not be in sync with your code in
some environment seems worse to me then where we currently are. I think the
question is how would this combination be explained to users and does it
make sense?

-Jay



On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Chaps,
>
> Can we either get one more +1 binding (we have 2 already) on the existing?
>
> Or have some response on the below possible alternative. We are keen to
> get working on this, so we make next feature release.
>
> Cheers
> Mike
>
>
> On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com> wrote:
>
>     Hi Ismael
>
>     Did you see our email this morning, what's your thoughts on this
> approach to instead we simply have a brand new policy?
>
>     Cheers
>     Mike
>
>
>     Sent using OWA for iPhone
>     ________________________________________
>     From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <
> ismael@juma.me.uk>
>     Sent: Tuesday, December 13, 2016 11:30:05 AM
>     To: dev@kafka.apache.org
>     Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
>     Yes, this is actually tricky to do in a way where we both have the
> desired
>     semantics and maintain compatibility. When someone creates a
>     `ProducerRecord` with a `null` value today, the producer doesn't know
> if
>     it's meant to be a tombstone or not. For V3 messages, it's easy when
> the
>     constructor that takes a tombstone is used. However, if any other
>     constructor is used, it's not clear. A couple of options I can think
> of,
>     none of them particularly nice:
>
>     1. Have a third state where tombstone = unknown and the broker would
> set
>     the tombstone bit if the value was null and the topic was compacted.
> People
>     that wanted to pass a non-null value for the tombstone would have to
> use
>     the constructor that takes a tombstone. The drawbacks: third state for
>     tombstone in message format, message conversion at the broker for a
> common
>     case.
>
>     2. Extend MetadataResponse to optionally include topic configs, which
> would
>     make it possible for the producer to be smarter about setting the
>     tombstone. It would only do it if a tombstone was not passed
> explicitly,
>     the value was null and the topic was compacted. The main drawback is
> that
>     the producer would be getting a bit more data for each topic even
> though it
>     probably won't use it most of the time. Extending MetadataResponse to
>     return topic configs would be useful for other reasons as well, so that
>     part seems OK.
>
>     In addition, for both proposals, we could consider adding warnings to
> the
>     documentation that the behaviour of the constructors that don't take a
>     tombstone would change in the next major release so that tombstone =
> false.
>     Not sure if this would be worth it though.
>
>     Ismael
>
>     On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> ewen@confluent.io>
>     wrote:
>
>     > Michael,
>     >
>     > It kind of depends on how you want to interpret the tombstone flag.
> If it's
>     > purely a producer-facing Kafka-level thing that we treat as internal
> to the
>     > broker and log cleaner once the record is sent, then your approach
> makes
>     > sense. You're just moving copying the null-indicates-delete behavior
> of the
>     > old constructor into the tombstone flag.
>     >
>     > However, if you want this change to more generally decouple the idea
> of
>     > deletion and null values, then you are sometimes converting what
> might be a
>     > completely valid null value that doesn't indicate deletion into a
>     > tombstone. Downstream applications could potentially handle these
> cases
>     > differently given the separation of deletion from value.
>     >
>     > I guess the question is if we want to try to support the latter even
> for
>     > topics where we have older produce requests. An example where this
> could
>     > come up is in something like a CDC Connector. If we try to support
> the
>     > semantic difference, a connector might write changes to Kafka using
> the
>     > tombstone flag to indicate when a row was truly deleted (vs an
> update that
>     > sets it to null but still present; this probably makes more sense
> for CDC
>     > from document stores or extracting single columns). There are various
>     > reasons we might want to maintain the full log and not turn
> compaction on
>     > (or just use a time-based retention policy), but downstream
> applications
>     > might care to know the difference between a delete and a null value.
> In
>     > fact both versions of the same log (compacted and time-retention)
> could be
>     > useful and I don't think it'll be uncommon to maintain both or use
> KIP-71
>     > to maintain a hybrid compacted/retention topic.
>     >
>     > -Ewen
>     >
>     > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > wrote:
>     >
>     > > Hi Jay,
>     > >
>     > > Why wouldn't that work, the tombstone value is only looked at by
> the
>     > > broker, on a topic configured for compaction as such is benign on
> non
>     > > compacted topics. This is as much as sending a null value currently
>     > >
>     > >
>     > > Regards
>     > > Mike
>     > >
>     > >
>     > >
>     > > Sent using OWA for iPhone
>     > > ________________________________________
>     > > From: Jay Kreps <ja...@confluent.io>
>     > > Sent: Sunday, December 11, 2016 8:58:53 PM
>     > > To: dev@kafka.apache.org
>     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > >
>     > > Hey Michael,
>     > >
>     > > I'm not quite sure that works as that would translate ALL null
> values to
>     > > tombstones, even for non-compacted topics that use null as an
> acceptable
>     > > value sent by the producer and expected by the consumer.
>     > >
>     > > -Jay
>     > >
>     > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > > wrote:
>     > >
>     > > > Hi Ewen,
>     > > >
>     > > > I think the easiest way to show this is with code.
>     > > >
>     > > > As you can see we keep the existing behaviour for code/binaries
> calling
>     > > > the pre-existing constructors, whereby if the value is null the
>     > tombstone
>     > > > is set to true.
>     > > >
>     > > > Regards
>     > > > Mike
>     > > >
>     > > >
>     > > >
>     > > >     /**
>     > > >      * Creates a record with a specified timestamp to be sent to
> a
>     > > > specified topic and partition
>     > > >      *
>     > > >      * @param topic The topic the record will be appended to
>     > > >      * @param partition The partition to which the record should
> be
>     > sent
>     > > >      * @param timestamp The timestamp of the record
>     > > >      * @param tombstone if the record should be treated as a
> tombstone
>     > if
>     > > > the topic is compacted
>     > > >      * @param key The key that will be included in the record
>     > > >      * @param value The record contents
>     > > >      */
>     > > >     public ProducerRecord(String topic, Integer partition,
> Boolean
>     > > > tombstone, Long timestamp, K key, V value) {
>     > > >         if (topic == null)
>     > > >             throw new IllegalArgumentException("Topic cannot be
>     > null.");
>     > > >         if (timestamp != null && timestamp < 0)
>     > > >             throw new IllegalArgumentException(
>     > > >                     String.format("Invalid timestamp: %d.
> Timestamp
>     > > should
>     > > > always be non-negative or null.", timestamp));
>     > > >         if (partition != null && partition < 0)
>     > > >             throw new IllegalArgumentException(
>     > > >                     String.format("Invalid partition: %d.
> Partition
>     > > number
>     > > > should always be non-negative or null.", partition));
>     > > >         if (!tombstone && value == null){
>     > > >             throw new IllegalArgumentException(
>     > > >                     String.format("Tombstone must be true if null
>     > > value"));
>     > > >         }
>     > > >         this.topic = topic;
>     > > >         this.partition = partition;
>     > > >         this.tombstone = tombstone;
>     > > >         this.key = key;
>     > > >         this.value = value;
>     > > >         this.timestamp = timestamp;
>     > > >     }
>     > > >
>     > > >     public ProducerRecord(String topic, Integer partition, Long
>     > > timestamp,
>     > > > K key, V value) {
>     > > >         this(topic, partition, value == null, timestamp, key,
> value);
>     > > >     }
>     > > > ________________________________________
>     > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
>     > > > Sent: Sunday, December 11, 2016 5:45 AM
>     > > > To: dev@kafka.apache.org
>     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > >
>     > > > It seemed like this was addressed in the migration section,
> wasn't it?
>     > > The
>     > > > V2 vs V3 requests and the fact that the broker will downgrade the
>     > message
>     > > > format (losing zero copy) if you issues a V2 request to a broker
> using
>     > V3
>     > > > format handles compatibility, does it not? The existing
> consumers won't
>     > > see
>     > > > the extra metadata in the value, but they will get a null
> instead and
>     > > treat
>     > > > it as a tombstone. Certainly there is a performance impact, but
> it
>     > seemed
>     > > > compatible.
>     > > >
>     > > > I'm worried about this though:
>     > > >
>     > > >
>     > > >    - *NOTE *: With the new version of producer client using
>     > > ProduceRequest
>     > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set)
> and
>     > null
>     > > > value
>     > > >    should not be allowed as the older version of consumer using
>     > > > FetchRequest
>     > > >    V2 will think of this as a tombstone when its actually not.
>     > > >
>     > > >
>     > > > Unless I'm misunderstanding, this ends up breaking binary
> compatibility
>     > > for
>     > > > the Java API. It sounds like this suggests that passing a null
> value to
>     > > the
>     > > > existing ProducerRecord constructors would generate an exception
> since
>     > > you
>     > > > didn't explicitly enable the tombstone (via whatever new
> constructor is
>     > > > provided). But that means you can't swap in a newer client jar
> without
>     > > > recompiling and get the same behavior if your app deletes keys
> using
>     > the
>     > > > current approach because your app will start throwing
> exceptions. Maybe
>     > > > this is a tradeoff we're ok with, but we've tried pretty hard to
> avoid
>     > > > breaking compatibility like this so far -- it makes picking up
> bug
>     > fixes
>     > > in
>     > > > the clients harder for users of frameworks that have to pin to
> earlier
>     > > > default versions for compatibility.
>     > > >
>     > > > But then later the KIP says:
>     > > >
>     > > >
>     > > >    - The old constructors for ProducerRecord without this
> parameter
>     > will
>     > > be
>     > > >    kept but updated so that their default behaviour if setting
> null
>     > value
>     > > > will
>     > > >    be the tombstone will be set to true to keep existing
> behaviour.
>     > > >
>     > > >
>     > > > So maybe I am misinterpreting something.
>     > > >
>     > > > And just a nit re: motivation section, item 6 would be clearer
> for a
>     > > union
>     > > > schema with null (or optional schemas in other formats), e.g.
> [null,
>     > > > string], in which case losing the schema truly is losing
> information
>     > > > (whereas null is already the only valid value for a pure null
> schema).
>     > > >
>     > > > -Ewen
>     > > >
>     > > >
>     > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> Michael.Pearce@ig.com
>     > >
>     > > > wrote:
>     > > >
>     > > > > Hi Jay,
>     > > > >
>     > > > > Good point this detail is missing in the KIP write up. Ive
> added this
>     > > > now.
>     > > > >
>     > > > > Essentially simply just upgrading the clients we do not expect
> any
>     > > client
>     > > > > app code change needed.
>     > > > >
>     > > > > Cheers
>     > > > > Mike
>     > > > > ________________________________________
>     > > > > From: Jay Kreps <ja...@confluent.io>
>     > > > > Sent: Saturday, December 10, 2016 10:51 PM
>     > > > > To: dev@kafka.apache.org
>     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > > >
>     > > > > Michael,
>     > > > >
>     > > > > The compatibility section goes through the migration path, but
> isn't
>     > > the
>     > > > > bigger compatibility issue with existing apps? There are many
>     > (probably
>     > > > > thousands) of apps in production that use this feature and
> send null
>     > to
>     > > > > mean delete. It seems like this would break compatibility with
> them,
>     > > and
>     > > > > they would have to be rewritten to right?
>     > > > >
>     > > > > -Jay
>     > > > >
>     > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
>     > Michael.Pearce@ig.com
>     > > >
>     > > > > wrote:
>     > > > >
>     > > > > > Hi Jun,
>     > > > > >
>     > > > > > 4) On v3 we honour the tombstone. As such we expect it to be
> set
>     > > > > correctly
>     > > > > > as per the KIP.
>     > > > > >
>     > > > > > 4.1) why would we want to produce an error when its v3? This
> is the
>     > > > exact
>     > > > > > purpose to support non-null tombstone’s
>     > > > > > 4.2) again here im unclear on the question on the v3, produce
>     > request
>     > > > we
>     > > > > > expect the tombstone flag to be set correctly.
>     > > > > >
>     > > > > > 4.4) compaction only occurs on compacted topics, the bit
> makes no
>     > > > > > difference and not looked at on un-compacted (time/size based
>     > > > eviction).
>     > > > > >
>     > > > > >
>     > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
>     > > > > >
>     > > > > >     Hi, Michael,
>     > > > > >
>     > > > > >     4. Then, I think I misunderstood this point. Could you
> document
>     > > the
>     > > > > >     following points in the wiki?
>     > > > > >     4.1 If producer V3 sets tombstone, but provides a
> non-null
>     > value,
>     > > > > does
>     > > > > > the
>     > > > > >     send() get an error or does the producer automatically
> set the
>     > > > value
>     > > > > to
>     > > > > >     null?
>     > > > > >     4.2 If producer V3 doesn't set tombstone, but provides a
> null
>     > > > value,
>     > > > > > does
>     > > > > >     the send() get an error or does the producer
> automatically sets
>     > > the
>     > > > > >     tombstone?
>     > > > > >     4.3 Does the broker only expect messages that (a) have no
>     > > tombstone
>     > > > > and
>     > > > > >     non-null value; (b) have tombstone and null value and
> reject
>     > the
>     > > > > > messages
>     > > > > >     with an error code otherwise?
>     > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
> compacted
>     > on
>     > > > > not?
>     > > > > >
>     > > > > >     Thanks,
>     > > > > >
>     > > > > >     Jun
>     > > > > >
>     > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
>     > > > > Michael.Pearce@ig.com
>     > > > > > >
>     > > > > >     wrote:
>     > > > > >
>     > > > > >     > Not at all.  This only acts on compacted topics just
> as what
>     > > > occurs
>     > > > > > today
>     > > > > >     >
>     > > > > >     > Sent using OWA for iPhone
>     > > > > >     > ________________________________________
>     > > > > >     > From: Jun Rao <ju...@confluent.io>
>     > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
>     > > > > >     > To: dev@kafka.apache.org
>     > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> Flag
>     > > > > >     >
>     > > > > >     > Hi, Michael,
>     > > > > >     >
>     > > > > >     > 4. Hmm, does that mean the new client library can
> never send
>     > a
>     > > > null
>     > > > > > message
>     > > > > >     > even to a regular topic? This seems like a change of
> the
>     > > existing
>     > > > > > behavior.
>     > > > > >     >
>     > > > > >     > Thanks,
>     > > > > >     >
>     > > > > >     > Jun
>     > > > > >     >
>     > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
>     > > > > > Michael.Pearce@ig.com>
>     > > > > >     > wrote:
>     > > > > >     >
>     > > > > >     > > Hi Jun,
>     > > > > >     > >
>     > > > > >     > > Re 4) That's because we expect the tombstone value
> to be
>     > set
>     > > > > > correctly if
>     > > > > >     > > message bit is 2, as such if an older client sends
> in on
>     > old
>     > > > > > message the
>     > > > > >     > > message is upcast and the bit is set correctly. And
> such no
>     > > > > longer
>     > > > > > need
>     > > > > >     > to
>     > > > > >     > > check the value. Mayuresh can you confirm my
> thinking and
>     > > > > > understanding
>     > > > > >     > of
>     > > > > >     > > what we've discussed?
>     > > > > >     > >
>     > > > > >     > > The second point I understand what you're getting at
> now my
>     > > > > > apologies.
>     > > > > >     > Yes
>     > > > > >     > > this makes sense to save on touching the message, if
> we're
>     > > the
>     > > > > > only kip
>     > > > > >     > > going in, in this release.
>     > > > > >     > >
>     > > > > >     > > Cheers
>     > > > > >     > > Mike
>     > > > > >     > >
>     > > > > >     > > Sent using OWA for iPhone
>     > > > > >     > > ________________________________________
>     > > > > >     > > From: Jun Rao <ju...@confluent.io>
>     > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>     > > > > >     > > To: dev@kafka.apache.org
>     > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone Flag
>     > > > > >     > >
>     > > > > >     > > Hi, Michael,
>     > > > > >     > >
>     > > > > >     > > 4. Is this updated in the wiki? The text "If the
> magic byte
>     > > on
>     > > > > > message is
>     > > > > >     > > 2, the broker should use the tombstone bit for log
>     > > compaction."
>     > > > > > doesn't
>     > > > > >     > > seem to have changed.
>     > > > > >     > >
>     > > > > >     > > 2. My point is that if we change the message format
> just
>     > for
>     > > > this
>     > > > > > KIP, we
>     > > > > >     > > should consider whether it's worth optimizing the
> down
>     > > > conversion
>     > > > > > path
>     > > > > >     > > (i.e., decide whether a conversion is needed by just
>     > looking
>     > > at
>     > > > > the
>     > > > > >     > > tombstone bit in the wrapper message) since
> tombstone will
>     > be
>     > > > > used
>     > > > > >     > rarely.
>     > > > > >     > > However, if the message format change here is
> combined with
>     > > > other
>     > > > > > KIPs,
>     > > > > >     > > then this optimization likely won't be needed. The
> latter
>     > > > > probably
>     > > > > > makes
>     > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> think?
>     > > > > >     > >
>     > > > > >     > > Other than those, +1 from me,
>     > > > > >     > >
>     > > > > >     > > Thanks,
>     > > > > >     > >
>     > > > > >     > > Jun
>     > > > > >     > >
>     > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
>     > > > > > Michael.Pearce@ig.com>
>     > > > > >     > > wrote:
>     > > > > >     > >
>     > > > > >     > > > Hi Jun
>     > > > > >     > > >
>     > > > > >     > > > do we have your vote on this now?
>     > > > > >     > > >
>     > > > > >     > > > Any other concerns?
>     > > > > >     > > >
>     > > > > >     > > > Cheers
>     > > > > >     > > > Mike
>     > > > > >     > > >
>     > > > > >     > > > Sent using OWA for iPhone
>     > > > > >     > > > ________________________________________
>     > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
>     > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>     > > > > >     > > > To: dev@kafka.apache.org
>     > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone
>     > Flag
>     > > > > >     > > >
>     > > > > >     > > > Hi Jun,
>     > > > > >     > > >
>     > > > > >     > > > Have updated. Thanks again for the feedback.
>     > > > > >     > > >
>     > > > > >     > > > Agree yes we should align up when it gets to that,
> I
>     > assume
>     > > > > > you’ve
>     > > > > >     > > flagged
>     > > > > >     > > > the same in the other KIP?
>     > > > > >     > > >
>     > > > > >     > > > I think for now let’s get this KIP approved, then
> we can
>     > > > start
>     > > > > > the work
>     > > > > >     > > to
>     > > > > >     > > > get an initial PR, then we can discuss how to
> align the
>     > two
>     > > > if
>     > > > > > needed.
>     > > > > >     > > >
>     > > > > >     > > > Cheers,
>     > > > > >     > > > Mike
>     > > > > >     > > >
>     > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io>
>     > wrote:
>     > > > > >     > > >
>     > > > > >     > > >     Hi, Michael,
>     > > > > >     > > >
>     > > > > >     > > >     For 2), this is fine. Could you update the KIP
> wiki
>     > to
>     > > > make
>     > > > > > this
>     > > > > >     > and
>     > > > > >     > > > other
>     > > > > >     > > >     points clearer? Other than that, the KIP looks
> good
>     > to
>     > > > me.
>     > > > > >     > > >
>     > > > > >     > > >     An orthogonal thing is that there are other
> KIPs such
>     > > as
>     > > > > > KIP-98
>     > > > > >     > that
>     > > > > >     > > > also
>     > > > > >     > > >     intend to change the message format. If they
> all get
>     > > > > > approved, we
>     > > > > >     > > > should
>     > > > > >     > > >     think about whether it's better to just bump
> up the
>     > > magic
>     > > > > > byte once
>     > > > > >     > > to
>     > > > > >     > > >     incorporate multiple format changes like we
> did in
>     > > > > > KIP-31/KIP-32.
>     > > > > >     > > >
>     > > > > >     > > >     Thanks,
>     > > > > >     > > >
>     > > > > >     > > >     Jun
>     > > > > >     > > >
>     > > > > >     > > >
>     > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce
> <
>     > > > > >     > > Michael.Pearce@ig.com>
>     > > > > >     > > >     wrote:
>     > > > > >     > > >
>     > > > > >     > > >     > Hi Jao
>     > > > > >     > > >     >
>     > > > > >     > > >     > Thanks for the response. Sorry for slow
> reply, both
>     > > > with
>     > > > > > personal
>     > > > > >     > > > sickness
>     > > > > >     > > >     > and also battling some critical issues
> encountered
>     > > > since
>     > > > > >     > upgrading
>     > > > > >     > > to
>     > > > > >     > > >     > 0.10.1.0
>     > > > > >     > > >     >
>     > > > > >     > > >     > 1) Thans for spotting, Document error where
> we
>     > > branched
>     > > > > > this KIP
>     > > > > >     > > from
>     > > > > >     > > >     > KIP-82, will get that removed.
>     > > > > >     > > >     > 2) Intent is to do this just at the record
> message
>     > > > level.
>     > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
>     > > corrected.
>     > > > > >     > > >     > 4) As per discussion thread we will support
>     > > tombstone +
>     > > > > > null
>     > > > > >     > value,
>     > > > > >     > > >     > tombstone + non null value, no tombstone +
> null
>     > > value.
>     > > > > >     > > >     > 5) I believe this was in the discussion
> thread,
>     > > > @Mayuresh
>     > > > > > is this
>     > > > > >     > > >     > something we’ve overlooked? I thought we
> would down
>     > > > > > convert and
>     > > > > >     > > > remove the
>     > > > > >     > > >     > value so the old consumer had existing
> behavior, or
>     > > is
>     > > > > > there
>     > > > > >     > > > something we
>     > > > > >     > > >     > haven’t thought about?
>     > > > > >     > > >     >
>     > > > > >     > > >     > Cheers
>     > > > > >     > > >     > Mike
>     > > > > >     > > >     >
>     > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> jun@confluent.io>
>     > > > > wrote:
>     > > > > >     > > >     >
>     > > > > >     > > >     >     Hi, Michael,
>     > > > > >     > > >     >
>     > > > > >     > > >     >     Thanks for the KIP. A few comments below.
>     > > > > >     > > >     >
>     > > > > >     > > >     >     1. The message format change contains
>     > > > "HeadersLength
>     > > > > >     > Headers".
>     > > > > >     > > > Is that
>     > > > > >     > > >     >     intended?
>     > > > > >     > > >     >
>     > > > > >     > > >     >     2. For compressed messageset, is the
> tombstone
>     > > bit
>     > > > > > only set
>     > > > > >     > at
>     > > > > >     > > > the
>     > > > > >     > > >     > shallow
>     > > > > >     > > >     >     level? Do we always leave that bit in the
>     > wrapper
>     > > > > > message
>     > > > > >     > > unset?
>     > > > > >     > > > An
>     > > > > >     > > >     >     alternative is to set the tombstone bit
> in the
>     > > > > wrapper
>     > > > > > if at
>     > > > > >     > > > least one
>     > > > > >     > > >     >     inner message has the tombstone bit set.
> This
>     > > makes
>     > > > > > things a
>     > > > > >     > > bit
>     > > > > >     > > > more
>     > > > > >     > > >     >     complicated, but we could potentially
> exploit
>     > > that
>     > > > > for
>     > > > > >     > > > optimizing down
>     > > > > >     > > >     >     conversion. For example, we only need to
>     > convert
>     > > > > > messages
>     > > > > >     > with
>     > > > > >     > > > magic 2
>     > > > > >     > > >     > to
>     > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit
> is set
>     > > > > > (conversion is
>     > > > > >     > > > always
>     > > > > >     > > >     > needed
>     > > > > >     > > >     >     from magic 2 to magic 0). Not sure if the
>     > > > > optimization
>     > > > > > is
>     > > > > >     > worth
>     > > > > >     > > > the
>     > > > > >     > > >     >     complexity though.
>     > > > > >     > > >     >
>     > > > > >     > > >     >     3. The referencing of the new version of
>     > > > > >     > > > ProducerRequest/FetchRequest
>     > > > > >     > > >     > is
>     > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> convention
>     > > > starts
>     > > > > at
>     > > > > >     > version
>     > > > > >     > > > at 0, I
>     > > > > >     > > >     >     think the new version would be 3.
>     > > > > >     > > >     >
>     > > > > >     > > >     >     4. "If the magic byte on message is 2,
> the
>     > broker
>     > > > > > should use
>     > > > > >     > > the
>     > > > > >     > > >     > tombstone
>     > > > > >     > > >     >     bit for log compaction." What about null
> value?
>     > > My
>     > > > > >     > > understanding
>     > > > > >     > > > is
>     > > > > >     > > >     > that
>     > > > > >     > > >     >     null value will be treated the same as
> setting
>     > > the
>     > > > > > tombstone
>     > > > > >     > > bit.
>     > > > > >     > > >     >
>     > > > > >     > > >     >     5. For the migration path, it would be
> useful
>     > to
>     > > > > > describe the
>     > > > > >     > > > down
>     > > > > >     > > >     >     conversion path to consumers (i.e.,
> brokers on
>     > > > > message
>     > > > > > format
>     > > > > >     > > > 0.10.2
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     consumers on older version).
>     > > > > >     > > >     >
>     > > > > >     > > >     >     Thanks,
>     > > > > >     > > >     >
>     > > > > >     > > >     >     Jun
>     > > > > >     > > >     >
>     > > > > >     > > >     >
>     > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael
>     > Pearce <
>     > > > > >     > > > Michael.Pearce@ig.com
>     > > > > >     > > >     > >
>     > > > > >     > > >     >     wrote:
>     > > > > >     > > >     >
>     > > > > >     > > >     >     > Hi All,
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > We have been discussing in the below
> thread
>     > and
>     > > > > final
>     > > > > >     > changes
>     > > > > >     > > > have
>     > > > > >     > > >     > been
>     > > > > >     > > >     >     > made to the KIP wiki based on these
>     > > discussions.
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > We would now like to put to the vote
> the
>     > > > following
>     > > > > > KIP:
>     > > > > >     > > >     >     > https://cwiki.apache.org/
>     > > > > > confluence/display/KAFKA/KIP-
>     > > > > >     > 87+-+
>     > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > This kip is for having a distinct
> compaction
>     > > > > > attribute
>     > > > > >     > > > “tombstone”
>     > > > > >     > > >     > flag
>     > > > > >     > > >     >     > instead of relying on null value,
> allowing
>     > > > non-null
>     > > > > > value
>     > > > > >     > > > delete
>     > > > > >     > > >     > messages.
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > Many thanks,
>     > > > > >     > > >     >     > Michael
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce"
> <
>     > > > > >     > > Michael.Pearce@ig.com>
>     > > > > >     > > >     > wrote:
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     Hi Mayuresh,
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     LGTM. Ive just made one small
> adjustment
>     > > > > > updating the
>     > > > > >     > > wire
>     > > > > >     > > >     > protocol to
>     > > > > >     > > >     >     > show the magic byte bump.
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     Do we think we’re good to put to a
> vote?
>     > Is
>     > > > > > there any
>     > > > > >     > > > other bits
>     > > > > >     > > >     >     > needing discussion?
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     Cheers
>     > > > > >     > > >     >     >     Mike
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
> Gharat" <
>     > > > > >     > > >     > gharatmayuresh15@gmail.com>
>     > > > > >     > > >     >     > wrote:
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         Hi Michael,
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         I have updated the migration
> section
>     > of
>     > > > the
>     > > > > > KIP.
>     > > > > >     > Can
>     > > > > >     > > > you
>     > > > > >     > > >     > please
>     > > > > >     > > >     >     > take a look?
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         Thanks,
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         Mayuresh
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07
> AM,
>     > > Mayuresh
>     > > > > > Gharat <
>     > > > > >     > > >     >     > gharatmayuresh15@gmail.com
>     > > > > >     > > >     >     >         > wrote:
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         > Hi Michael,
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > That whilst sending
> tombstone and
>     > non
>     > > > > null
>     > > > > > value,
>     > > > > >     > > the
>     > > > > >     > > >     > consumer
>     > > > > >     > > >     >     > can expect
>     > > > > >     > > >     >     >         > only to receive the non-null
>     > message
>     > > > only
>     > > > > > in step
>     > > > > >     > > > (3) is
>     > > > > >     > > >     > this
>     > > > > >     > > >     >     > correct?
>     > > > > >     > > >     >     >         > ---> I do agree with you
> here.
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > Becket, Ismael : can you guys
>     > review
>     > > > the
>     > > > > >     > migration
>     > > > > >     > > > plan
>     > > > > >     > > >     > listed
>     > > > > >     > > >     >     > above using
>     > > > > >     > > >     >     >         > magic byte?
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > Thanks,
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > Mayuresh
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58
> AM,
>     > > > Michael
>     > > > > > Pearce <
>     > > > > >     > > >     >     > Michael.Pearce@ig.com>
>     > > > > >     > > >     >     >         > wrote:
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         >> Many thanks for this
> Mayuresh. I
>     > > don't
>     > > > > > have any
>     > > > > >     > > >     > objections.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> I assume we should state:
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> That whilst sending
> tombstone and
>     > > non
>     > > > > null
>     > > > > >     > value,
>     > > > > >     > > > the
>     > > > > >     > > >     > consumer
>     > > > > >     > > >     >     > can expect
>     > > > > >     > > >     >     >         >> only to receive the non-null
>     > message
>     > > > > only
>     > > > > > in
>     > > > > >     > step
>     > > > > >     > > > (3) is
>     > > > > >     > > >     > this
>     > > > > >     > > >     >     > correct?
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Cheers
>     > > > > >     > > >     >     >         >> Mike
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Sent using OWA for iPhone
>     > > > > >     > > >     >     >         >>
> ______________________________
>     > > > > __________
>     > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
>     > > > > >     > gharatmayuresh15@gmail.com
>     > > > > >     > > >
>     > > > > >     > > >     >     >         >> Sent: Thursday, November
> 17, 2016
>     > > > > 5:18:41
>     > > > > > PM
>     > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
>     > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> KIP-87 -
>     > Add
>     > > > > > Compaction
>     > > > > >     > > > Tombstone
>     > > > > >     > > >     > Flag
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Hi Ismael,
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Thanks for the explanation.
>     > > > > >     > > >     >     >         >> Specially I like this part
> where
>     > in
>     > > > you
>     > > > > >     > mentioned
>     > > > > >     > > > we can
>     > > > > >     > > >     > get
>     > > > > >     > > >     >     > rid of the
>     > > > > >     > > >     >     >         >> older null value support
> for log
>     > > > > > compaction
>     > > > > >     > later
>     > > > > >     > > > on,
>     > > > > >     > > >     > here :
>     > > > > >     > > >     >     >         >> We can't change semantics
> of the
>     > > > message
>     > > > > > format
>     > > > > >     > > > without
>     > > > > >     > > >     > having
>     > > > > >     > > >     >     > a long
>     > > > > >     > > >     >     >         >> transition period. And we
> can't
>     > rely
>     > > > > >     > > >     >     >         >> on people reading
> documentation or
>     > > > > acting
>     > > > > > on a
>     > > > > >     > > > warning for
>     > > > > >     > > >     >     > something so
>     > > > > >     > > >     >     >         >> fundamental. As such, my
> take is
>     > > that
>     > > > we
>     > > > > > need to
>     > > > > >     > > > bump the
>     > > > > >     > > >     > magic
>     > > > > >     > > >     >     > byte. The
>     > > > > >     > > >     >     >         >> good news is
>     > > > > >     > > >     >     >         >> that we don't have to
> support all
>     > > > > versions
>     > > > > >     > > forever.
>     > > > > >     > > > We
>     > > > > >     > > >     > have
>     > > > > >     > > >     >     > said that we
>     > > > > >     > > >     >     >         >> will support direct
> upgrades for 2
>     > > > > years.
>     > > > > > That
>     > > > > >     > > > means that
>     > > > > >     > > >     >     > message format
>     > > > > >     > > >     >     >         >> version n could, in theory,
> be
>     > > > removed 2
>     > > > > > years
>     > > > > >     > > > after the
>     > > > > >     > > >     > it's
>     > > > > >     > > >     >     > introduced.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Just a heads up, I would
> like to
>     > > > mention
>     > > > > > that
>     > > > > >     > even
>     > > > > >     > > > without
>     > > > > >     > > >     >     > bumping magic
>     > > > > >     > > >     >     >         >> byte, we will *NOT* loose
> zero
>     > copy
>     > > as
>     > > > > in
>     > > > > > the
>     > > > > >     > > > client(x+1)
>     > > > > >     > > >     > in my
>     > > > > >     > > >     >     >         >> explanation
>     > > > > >     > > >     >     >         >> above will convert
> internally a
>     > null
>     > > > > > value to
>     > > > > >     > > have a
>     > > > > >     > > >     > tombstone
>     > > > > >     > > >     >     > bit set and
>     > > > > >     > > >     >     >         >> a tombstone bit set to have
> a null
>     > > > value
>     > > > > >     > > > automatically
>     > > > > >     > > >     >     > internally and by
>     > > > > >     > > >     >     >         >> the time we move to version
> (x+2),
>     > > the
>     > > > > > clients
>     > > > > >     > > > would have
>     > > > > >     > > >     >     > upgraded.
>     > > > > >     > > >     >     >         >> Obviously if we support a
> request
>     > > from
>     > > > > >     > > consumer(x),
>     > > > > >     > > > we
>     > > > > >     > > >     > will
>     > > > > >     > > >     >     > loose zero
>     > > > > >     > > >     >     >         >> copy
>     > > > > >     > > >     >     >         >> but that is the same case
> with
>     > magic
>     > > > > byte.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> But if magic byte bump
> makes life
>     > > > easier
>     > > > > > for
>     > > > > >     > > > transition
>     > > > > >     > > >     > for the
>     > > > > >     > > >     >     > above
>     > > > > >     > > >     >     >         >> reasons that you explained,
> I am
>     > OK
>     > > > with
>     > > > > > it
>     > > > > >     > since
>     > > > > >     > > > we are
>     > > > > >     > > >     > going
>     > > > > >     > > >     >     > to meet the
>     > > > > >     > > >     >     >         >> end goal down the road :)
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> On a side note can we
> update the
>     > doc
>     > > > > here
>     > > > > > on
>     > > > > >     > magic
>     > > > > >     > > > byte
>     > > > > >     > > >     > to say
>     > > > > >     > > >     >     > that "*it
>     > > > > >     > > >     >     >         >> should be bumped whenever
> the
>     > > message
>     > > > > > format is
>     > > > > >     > > > changed
>     > > > > >     > > >     > or the
>     > > > > >     > > >     >     >         >> interpretation of message
> format
>     > > > (usage
>     > > > > > of the
>     > > > > >     > > > reserved
>     > > > > >     > > >     > bits as
>     > > > > >     > > >     >     > well) is
>     > > > > >     > > >     >     >         >> changed*".
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Hi Michael,
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Here is the update plan
> that we
>     > > > > discussed
>     > > > > >     > offline
>     > > > > >     > > >     > yesterday :
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Currently the magic-byte
> which
>     > > > > > corresponds to
>     > > > > >     > the
>     > > > > >     > > >     >     > "message.format.version"
>     > > > > >     > > >     >     >         >> is set to 1.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> 1) On broker it will be set
> to 1
>     > > > > > initially.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> 2) When a producer client
> sends a
>     > > > > message
>     > > > > > with
>     > > > > >     > > > magic-byte
>     > > > > >     > > >     > = 2,
>     > > > > >     > > >     >     > since the
>     > > > > >     > > >     >     >         >> broker is on magic-byte =
> 1, we
>     > will
>     > > > > down
>     > > > > >     > convert
>     > > > > >     > > > it,
>     > > > > >     > > >     > which
>     > > > > >     > > >     >     > means if the
>     > > > > >     > > >     >     >         >> tombstone bit is set, the
> value
>     > will
>     > > > be
>     > > > > > set to
>     > > > > >     > > > null. A
>     > > > > >     > > >     > consumer
>     > > > > >     > > >     >     >         >> understanding magic-byte =
> 1, will
>     > > > still
>     > > > > > work
>     > > > > >     > with
>     > > > > >     > > > this. A
>     > > > > >     > > >     >     > consumer
>     > > > > >     > > >     >     >         >> working
>     > > > > >     > > >     >     >         >> with magic-byte =2 will
> also be
>     > able
>     > > > to
>     > > > > >     > understand
>     > > > > >     > > > this,
>     > > > > >     > > >     > since
>     > > > > >     > > >     >     > it
>     > > > > >     > > >     >     >         >> understands the tombstone.
>     > > > > >     > > >     >     >         >> Now there is still the
> question of
>     > > > > > supporting a
>     > > > > >     > > >     > non-tombstone
>     > > > > >     > > >     >     > and null
>     > > > > >     > > >     >     >         >> value from producer client
> with
>     > > > > > magic-byte = 2.*
>     > > > > >     > > (I
>     > > > > >     > > > am
>     > > > > >     > > >     > not sure
>     > > > > >     > > >     >     > if we
>     > > > > >     > > >     >     >         >> should support this.
> Ismael/Becket
>     > > can
>     > > > > > comment
>     > > > > >     > > > here)*
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> 3) When almost all the
> clients
>     > have
>     > > > > > upgraded,
>     > > > > >     > the
>     > > > > >     > > >     >     > message.format.version
>     > > > > >     > > >     >     >         >> on
>     > > > > >     > > >     >     >         >> the broker can be changed
> to 2,
>     > > where
>     > > > in
>     > > > > > the
>     > > > > >     > down
>     > > > > >     > > >     > conversion in
>     > > > > >     > > >     >     > the above
>     > > > > >     > > >     >     >         >> step will not happen. If at
> this
>     > > point
>     > > > > we
>     > > > > > get a
>     > > > > >     > > > consumer
>     > > > > >     > > >     >     > request from a
>     > > > > >     > > >     >     >         >> older consumer, we might
> have to
>     > > down
>     > > > > > convert
>     > > > > >     > > where
>     > > > > >     > > > in we
>     > > > > >     > > >     > loose
>     > > > > >     > > >     >     > zero copy,
>     > > > > >     > > >     >     >         >> but these cases should be
> rare.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Becket can you review this
> plan
>     > and
>     > > > add
>     > > > > > more
>     > > > > >     > > > details if I
>     > > > > >     > > >     > have
>     > > > > >     > > >     >     >         >> missed/wronged something,
> before
>     > we
>     > > > put
>     > > > > > it on
>     > > > > >     > KIP.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Thanks,
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> Mayuresh
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> 11:07 PM,
>     > > > > Michael
>     > > > > >     > Pearce <
>     > > > > >     > > >     >     > Michael.Pearce@ig.com>
>     > > > > >     > > >     >     >         >> wrote:
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> > Thanks guys, for
> discussing this
>     > > > > > offline and
>     > > > > >     > > > getting
>     > > > > >     > > >     > some
>     > > > > >     > > >     >     > consensus.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > So its clear for myself
> and
>     > others
>     > > > > what
>     > > > > > is
>     > > > > >     > > > proposed now
>     > > > > >     > > >     > (i
>     > > > > >     > > >     >     > think i
>     > > > > >     > > >     >     >         >> > understand, but want to
> make
>     > sure)
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Could i ask either
> directly
>     > update
>     > > > the
>     > > > > > kip to
>     > > > > >     > > > detail the
>     > > > > >     > > >     >     > migration
>     > > > > >     > > >     >     >         >> > strategy, or (re-)state
> your
>     > > offline
>     > > > > > discussed
>     > > > > >     > > and
>     > > > > >     > > >     > agreed
>     > > > > >     > > >     >     > migration
>     > > > > >     > > >     >     >         >> > strategy based on a magic
> byte
>     > is
>     > > in
>     > > > > > this
>     > > > > >     > > thread.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > The main original driver
> for the
>     > > KIP
>     > > > > > was to
>     > > > > >     > > > support
>     > > > > >     > > >     >     > compaction where
>     > > > > >     > > >     >     >         >> value
>     > > > > >     > > >     >     >         >> > isn't null, based off the
>     > > > discussions
>     > > > > on
>     > > > > >     > KIP-82
>     > > > > >     > > > thread.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > We should be able to
> support
>     > > > > > non-tombstone +
>     > > > > >     > > null
>     > > > > >     > > > value
>     > > > > >     > > >     > by the
>     > > > > >     > > >     >     >         >> completion
>     > > > > >     > > >     >     >         >> > of the KIP, as we noted
> when
>     > > > > discussing
>     > > > > > this
>     > > > > >     > > kip,
>     > > > > >     > > > having
>     > > > > >     > > >     >     > logic based on
>     > > > > >     > > >     >     >         >> a
>     > > > > >     > > >     >     >         >> > null value isn't very
> clean and
>     > > also
>     > > > > > separates
>     > > > > >     > > the
>     > > > > >     > > >     > concerns.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > As discussed already
> though we
>     > can
>     > > > > > split this
>     > > > > >     > > into
>     > > > > >     > > >     > KIP-87a
>     > > > > >     > > >     >     > and KIP-87b
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Where we look to deliver
> KIP-87a
>     > > on
>     > > > a
>     > > > > >     > compacted
>     > > > > >     > > > topic
>     > > > > >     > > >     > (to
>     > > > > >     > > >     >     > address the
>     > > > > >     > > >     >     >         >> > immediate issues)
>     > > > > >     > > >     >     >         >> > * tombstone + null value
>     > > > > >     > > >     >     >         >> > * tombstone + non-null
> value
>     > > > > >     > > >     >     >         >> > * non-tombstone +
> non-null value
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Then we can discuss once
> KIP-87a
>     > > is
>     > > > > > completed
>     > > > > >     > > > options
>     > > > > >     > > >     > later
>     > > > > >     > > >     >     > and how we
>     > > > > >     > > >     >     >         >> > support the second part
> KIP-87b
>     > to
>     > > > > > deliver:
>     > > > > >     > > >     >     >         >> > * non-tombstone + null
> value
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Cheers
>     > > > > >     > > >     >     >         >> > Mike
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
> ______________________________
>     > > > > > __________
>     > > > > >     > > >     >     >         >> > From: Becket Qin <
>     > > > > becket.qin@gmail.com>
>     > > > > >     > > >     >     >         >> > Sent: Thursday, November
> 17,
>     > 2016
>     > > > 1:43
>     > > > > > AM
>     > > > > >     > > >     >     >         >> > To: dev@kafka.apache.org
>     > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> KIP-87 -
>     > > Add
>     > > > > > Compaction
>     > > > > >     > > >     > Tombstone Flag
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had
> an
>     > > offline
>     > > > > >     > discussion,
>     > > > > >     > > > and
>     > > > > >     > > >     > following
>     > > > > >     > > >     >     > is a brief
>     > > > > >     > > >     >     >         >> > summary.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > 1. We agreed that not
> bumping up
>     > > > magic
>     > > > > > value
>     > > > > >     > may
>     > > > > >     > > > result
>     > > > > >     > > >     > in
>     > > > > >     > > >     >     > losing zero
>     > > > > >     > > >     >     >         >> copy
>     > > > > >     > > >     >     >         >> > during migration.
>     > > > > >     > > >     >     >         >> > 2. Given that bumping up
> magic
>     > > value
>     > > > > is
>     > > > > > almost
>     > > > > >     > > > free and
>     > > > > >     > > >     > has
>     > > > > >     > > >     >     > benefit of
>     > > > > >     > > >     >     >         >> > avoiding potential
> performance
>     > > > issue.
>     > > > > > It is
>     > > > > >     > > > probably
>     > > > > >     > > >     > worth
>     > > > > >     > > >     >     > doing.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > One issue we still need
> to think
>     > > > about
>     > > > > > is
>     > > > > >     > > whether
>     > > > > >     > > > we
>     > > > > >     > > >     > want to
>     > > > > >     > > >     >     > support a
>     > > > > >     > > >     >     >         >> > non-tombstone message
> with null
>     > > > value.
>     > > > > >     > > >     >     >         >> > Currently it is not
> supported by
>     > > > > Kafka.
>     > > > > > If we
>     > > > > >     > > > allow a
>     > > > > >     > > >     >     > non-tombstone null
>     > > > > >     > > >     >     >         >> > value message to exist
> after
>     > > KIP-87.
>     > > > > The
>     > > > > >     > problem
>     > > > > >     > > > is
>     > > > > >     > > >     > that such
>     > > > > >     > > >     >     > message
>     > > > > >     > > >     >     >         >> will
>     > > > > >     > > >     >     >         >> > not be supported by the
>     > consumers
>     > > > > prior
>     > > > > > to
>     > > > > >     > > KIP-87.
>     > > > > >     > > >     > Because a
>     > > > > >     > > >     >     > null value
>     > > > > >     > > >     >     >         >> > will always be
> interpreted to a
>     > > > > > tombstone.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > One option is that we
> keep the
>     > > > current
>     > > > > > way,
>     > > > > >     > i.e.
>     > > > > >     > > > do not
>     > > > > >     > > >     >     > support such
>     > > > > >     > > >     >     >         >> > message. It would be good
> to
>     > know
>     > > if
>     > > > > > there is
>     > > > > >     > a
>     > > > > >     > > >     > concrete use
>     > > > > >     > > >     >     > case for
>     > > > > >     > > >     >     >         >> such
>     > > > > >     > > >     >     >         >> > message. If there is not,
> we can
>     > > > > > probably just
>     > > > > >     > > not
>     > > > > >     > > >     > support it.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > Thanks,
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at
> 1:28 PM,
>     > > > > > Mayuresh
>     > > > > >     > > Gharat <
>     > > > > >     > > >     >     >         >> >
> gharatmayuresh15@gmail.com
>     > > > > >     > > >     >     >         >> > > wrote:
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >> > > Hi Ismael,
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > This is something I can
> think
>     > of
>     > > > for
>     > > > > >     > migration
>     > > > > >     > > > plan:
>     > > > > >     > > >     >     >         >> > > So the migration plan
> can look
>     > > > > > something
>     > > > > >     > like
>     > > > > >     > > > this,
>     > > > > >     > > >     > with up
>     > > > > >     > > >     >     >         >> conversion :
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > 1) Currently lets say
> we have
>     > > > Broker
>     > > > > > at
>     > > > > >     > > version
>     > > > > >     > > > x.
>     > > > > >     > > >     >     >         >> > > 2) Currently we have
> clients
>     > at
>     > > > > > version x.
>     > > > > >     > > >     >     >         >> > > 3) a) We move the
> version to
>     > > > > > Broker(x+1) :
>     > > > > >     > > > supports
>     > > > > >     > > >     > both
>     > > > > >     > > >     >     > tombstone and
>     > > > > >     > > >     >     >         >> > null
>     > > > > >     > > >     >     >         >> > > for log compaction.
>     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> client
>     > to
>     > > > > > version
>     > > > > >     > > > client(x+1) :
>     > > > > >     > > >     > if in
>     > > > > >     > > >     >     > the
>     > > > > >     > > >     >     >         >> producer
>     > > > > >     > > >     >     >         >> > > client(x+1) the value
> is set
>     > to
>     > > > > null,
>     > > > > > we
>     > > > > >     > will
>     > > > > >     > > >     > automatically
>     > > > > >     > > >     >     > set the
>     > > > > >     > > >     >     >         >> > > Tombstone bit
> internally. If
>     > the
>     > > > > > producer
>     > > > > >     > > > client(x+1)
>     > > > > >     > > >     > sets
>     > > > > >     > > >     >     > the
>     > > > > >     > > >     >     >         >> tombstone
>     > > > > >     > > >     >     >         >> > > itself, well and good.
> For
>     > > > producer
>     > > > > >     > client(x),
>     > > > > >     > > > the
>     > > > > >     > > >     > broker
>     > > > > >     > > >     >     > will up
>     > > > > >     > > >     >     >         >> convert
>     > > > > >     > > >     >     >         >> > > to have the tombstone
> bit.
>     > > > > > Broker(x+1) is
>     > > > > >     > > > supporting
>     > > > > >     > > >     > both.
>     > > > > >     > > >     >     > Consumer
>     > > > > >     > > >     >     >         >> > > client(x+1) will be
> aware of
>     > > this
>     > > > > and
>     > > > > > should
>     > > > > >     > > be
>     > > > > >     > > > able
>     > > > > >     > > >     > to
>     > > > > >     > > >     >     > handle this.
>     > > > > >     > > >     >     >         >> For
>     > > > > >     > > >     >     >         >> > > consumer client(x) we
> will
>     > down
>     > > > > > convert the
>     > > > > >     > > > message
>     > > > > >     > > >     > on the
>     > > > > >     > > >     >     > broker
>     > > > > >     > > >     >     >         >> side.
>     > > > > >     > > >     >     >         >> > >     c) At this point we
> will
>     > > have
>     > > > to
>     > > > > >     > specify a
>     > > > > >     > > >     > warning or
>     > > > > >     > > >     >     > clearly
>     > > > > >     > > >     >     >         >> specify
>     > > > > >     > > >     >     >         >> > > in docs that this
> behavior is
>     > > > about
>     > > > > > to be
>     > > > > >     > > > changed for
>     > > > > >     > > >     > log
>     > > > > >     > > >     >     > compaction.
>     > > > > >     > > >     >     >         >> > > 4) a) In next release
> of the
>     > > > > > Broker(x+2), we
>     > > > > >     > > > say that
>     > > > > >     > > >     > only
>     > > > > >     > > >     >     > Tombstone
>     > > > > >     > > >     >     >         >> is
>     > > > > >     > > >     >     >         >> > > used for log compaction
> on the
>     > > > > Broker
>     > > > > > side.
>     > > > > >     > > >     > Clients(x+1)
>     > > > > >     > > >     >     > still is
>     > > > > >     > > >     >     >         >> > > supported.
>     > > > > >     > > >     >     >         >> > >     b) We upgrade the
> client
>     > to
>     > > > > > version
>     > > > > >     > > > client(x+2) :
>     > > > > >     > > >     > if
>     > > > > >     > > >     >     > value is set
>     > > > > >     > > >     >     >         >> to
>     > > > > >     > > >     >     >         >> > > null, tombstone will
> not be
>     > set
>     > > > > >     > automatically.
>     > > > > >     > > > The
>     > > > > >     > > >     > client
>     > > > > >     > > >     >     > will have to
>     > > > > >     > > >     >     >         >> > call
>     > > > > >     > > >     >     >         >> > > setTombstone() to
> actually set
>     > > the
>     > > > > >     > tombstone.
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > We should compare this
>     > migration
>     > > > > plan
>     > > > > > with
>     > > > > >     > the
>     > > > > >     > > >     > migration
>     > > > > >     > > >     >     > plan for
>     > > > > >     > > >     >     >         >> magic
>     > > > > >     > > >     >     >         >> > > byte bump and do
> whatever
>     > looks
>     > > > > good.
>     > > > > >     > > >     >     >         >> > > I am just worried that
> if we
>     > go
>     > > > down
>     > > > > > magic
>     > > > > >     > > byte
>     > > > > >     > > > route,
>     > > > > >     > > >     >     > unless I am
>     > > > > >     > > >     >     >         >> > missing
>     > > > > >     > > >     >     >         >> > > something, it sounds
> like
>     > kafka
>     > > > will
>     > > > > > be
>     > > > > >     > stuck
>     > > > > >     > > > with
>     > > > > >     > > >     >     > supporting both
>     > > > > >     > > >     >     >         >> null
>     > > > > >     > > >     >     >         >> > > value and tombstone bit
> for
>     > log
>     > > > > > compaction
>     > > > > >     > for
>     > > > > >     > > > life
>     > > > > >     > > >     > long,
>     > > > > >     > > >     >     > which does
>     > > > > >     > > >     >     >         >> not
>     > > > > >     > > >     >     >         >> > > look like a good end
> state.
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > Thanks,
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > Mayuresh
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at
> 9:32
>     > AM,
>     > > > > > Mayuresh
>     > > > > >     > > > Gharat <
>     > > > > >     > > >     >     >         >> > >
> gharatmayuresh15@gmail.com
>     > > > > >     > > >     >     >         >> > > > wrote:
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > > Hi Ismael,
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > That's a very good
> point
>     > > which I
>     > > > > > might
>     > > > > >     > have
>     > > > > >     > > > not
>     > > > > >     > > >     >     > considered earlier.
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > Here is a plan that I
> can
>     > > think
>     > > > > of:
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > Stage 1) The broker
> from now
>     > > on,
>     > > > > up
>     > > > > >     > converts
>     > > > > >     > > > the
>     > > > > >     > > >     > message
>     > > > > >     > > >     >     > to have the
>     > > > > >     > > >     >     >         >> > > > tombstone marker. The
> log
>     > > > > compaction
>     > > > > >     > thread
>     > > > > >     > > > does log
>     > > > > >     > > >     >     > compaction
>     > > > > >     > > >     >     >         >> based
>     > > > > >     > > >     >     >         >> > on
>     > > > > >     > > >     >     >         >> > > > both null and
> tombstone
>     > > marker.
>     > > > > > This is
>     > > > > >     > our
>     > > > > >     > > >     > transition
>     > > > > >     > > >     >     > period.
>     > > > > >     > > >     >     >         >> > > > Stage 2) The next
> release we
>     > > > only
>     > > > > > say that
>     > > > > >     > > log
>     > > > > >     > > >     > compaction
>     > > > > >     > > >     >     > is based
>     > > > > >     > > >     >     >         >> on
>     > > > > >     > > >     >     >         >> > > > tombstone marker.
> (Open
>     > source
>     > > > > > kafka makes
>     > > > > >     > > > this as a
>     > > > > >     > > >     >     > policy). By
>     > > > > >     > > >     >     >         >> this
>     > > > > >     > > >     >     >         >> > > time,
>     > > > > >     > > >     >     >         >> > > > the organization
> which is
>     > > moving
>     > > > > to
>     > > > > > this
>     > > > > >     > > > release
>     > > > > >     > > >     > will be
>     > > > > >     > > >     >     > sure that
>     > > > > >     > > >     >     >         >> they
>     > > > > >     > > >     >     >         >> > > > have gone through the
> entire
>     > > > > > transition
>     > > > > >     > > > period.
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > My only goal of doing
> this
>     > is
>     > > > that
>     > > > > > Kafka
>     > > > > >     > > > clearly
>     > > > > >     > > >     >     > specifies the end
>     > > > > >     > > >     >     >         >> > state
>     > > > > >     > > >     >     >         >> > > > about what log
> compaction
>     > > means
>     > > > > (is
>     > > > > > it
>     > > > > >     > null
>     > > > > >     > > > value
>     > > > > >     > > >     > or a
>     > > > > >     > > >     >     > tombstone
>     > > > > >     > > >     >     >         >> > marker,
>     > > > > >     > > >     >     >         >> > > > but not both).
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > What do you think?
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > Thanks,
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > Mayuresh
>     > > > > >     > > >     >     >         >> > > > .
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016
> at 9:17
>     > > AM,
>     > > > > > Ismael
>     > > > > >     > > Juma <
>     > > > > >     > > >     >     > ismael@juma.me.uk>
>     > > > > >     > > >     >     >         >> > wrote:
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > >> One comment below.
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016
> at
>     > 5:08
>     > > > PM,
>     > > > > > Mayuresh
>     > > > > >     > > > Gharat <
>     > > > > >     > > >     >     >         >> > > >>
> gharatmayuresh15@gmail.com
>     > > > > >     > > >     >     >         >> > > >> > wrote:
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >> >    - If we don't
> bump up
>     > > the
>     > > > > > magic
>     > > > > >     > byte,
>     > > > > >     > > > on the
>     > > > > >     > > >     > broker
>     > > > > >     > > >     >     > side, the
>     > > > > >     > > >     >     >         >> > > broker
>     > > > > >     > > >     >     >         >> > > >> >    will always
> have to
>     > look
>     > > > at
>     > > > > > both
>     > > > > >     > > > tombstone
>     > > > > >     > > >     > bit and
>     > > > > >     > > >     >     > the value
>     > > > > >     > > >     >     >         >> when
>     > > > > >     > > >     >     >         >> > > do
>     > > > > >     > > >     >     >         >> > > >> the
>     > > > > >     > > >     >     >         >> > > >> >    compaction.
> Assuming
>     > we
>     > > do
>     > > > > > not bump
>     > > > > >     > up
>     > > > > >     > > > the
>     > > > > >     > > >     > magic
>     > > > > >     > > >     >     > byte,
>     > > > > >     > > >     >     >         >> > > >> >    imagine the
> broker
>     > sees
>     > > a
>     > > > > > message
>     > > > > >     > > which
>     > > > > >     > > > does
>     > > > > >     > > >     > not
>     > > > > >     > > >     >     > have a
>     > > > > >     > > >     >     >         >> tombstone
>     > > > > >     > > >     >     >         >> > > bit
>     > > > > >     > > >     >     >         >> > > >> >    set. The broker
> does
>     > not
>     > > > > know
>     > > > > > when
>     > > > > >     > the
>     > > > > >     > > >     > message was
>     > > > > >     > > >     >     > produced
>     > > > > >     > > >     >     >         >> (i.e.
>     > > > > >     > > >     >     >         >> > > >> > whether
>     > > > > >     > > >     >     >         >> > > >> >    the message has
> been
>     > up
>     > > > > > converted or
>     > > > > >     > > > not), it
>     > > > > >     > > >     > has
>     > > > > >     > > >     >     > to take a
>     > > > > >     > > >     >     >         >> > further
>     > > > > >     > > >     >     >         >> > > >> > look at
>     > > > > >     > > >     >     >         >> > > >> >    the value to
> see if it
>     > > is
>     > > > > > null or
>     > > > > >     > not
>     > > > > >     > > in
>     > > > > >     > > >     > order to
>     > > > > >     > > >     >     > determine
>     > > > > >     > > >     >     >         >> if it
>     > > > > >     > > >     >     >         >> > > is
>     > > > > >     > > >     >     >         >> > > >> a
>     > > > > >     > > >     >     >         >> > > >> >    tombstone. The
> same
>     > > logic
>     > > > > has
>     > > > > > to be
>     > > > > >     > > put
>     > > > > >     > > > on the
>     > > > > >     > > >     >     > consumer as
>     > > > > >     > > >     >     >         >> well
>     > > > > >     > > >     >     >         >> > > >> because
>     > > > > >     > > >     >     >         >> > > >> > the
>     > > > > >     > > >     >     >         >> > > >> >    consumer does
> not know
>     > > if
>     > > > > the
>     > > > > >     > message
>     > > > > >     > > > has
>     > > > > >     > > >     > been up
>     > > > > >     > > >     >     > converted or
>     > > > > >     > > >     >     >         >> > not.
>     > > > > >     > > >     >     >         >> > > >> >       - If we
> upconvert
>     > > while
>     > > > > >     > appending,
>     > > > > >     > > > this is
>     > > > > >     > > >     > not
>     > > > > >     > > >     >     > the case,
>     > > > > >     > > >     >     >         >> > right?
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >> If I understand you
>     > > correctly,
>     > > > > > this is
>     > > > > >     > not
>     > > > > >     > > >     > sufficient
>     > > > > >     > > >     >     > because the
>     > > > > >     > > >     >     >         >> log
>     > > > > >     > > >     >     >         >> > > may
>     > > > > >     > > >     >     >         >> > > >> have messages
> appended
>     > before
>     > > > it
>     > > > > > was
>     > > > > >     > > > upgraded to
>     > > > > >     > > >     > include
>     > > > > >     > > >     >     > KIP-87.
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >> Ismael
>     > > > > >     > > >     >     >         >> > > >>
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > > > --
>     > > > > >     > > >     >     >         >> > > > -Regards,
>     > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
>     > > > > >     > > >     >     >         >> > > > (862) 250-7125
>     > > > > >     > > >     >     >         >> > > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > > --
>     > > > > >     > > >     >     >         >> > > -Regards,
>     > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
>     > > > > >     > > >     >     >         >> > > (862) 250-7125
>     > > > > >     > > >     >     >         >> > >
>     > > > > >     > > >     >     >         >> > The information contained
> in
>     > this
>     > > > > email
>     > > > > > is
>     > > > > >     > > > strictly
>     > > > > >     > > >     >     > confidential and for
>     > > > > >     > > >     >     >         >> > the use of the addressee
> only,
>     > > > unless
>     > > > > >     > otherwise
>     > > > > >     > > >     > indicated. If
>     > > > > >     > > >     >     > you are
>     > > > > >     > > >     >     >         >> not
>     > > > > >     > > >     >     >         >> > the intended recipient,
> please
>     > do
>     > > > not
>     > > > > > read,
>     > > > > >     > > copy,
>     > > > > >     > > > use or
>     > > > > >     > > >     >     > disclose to
>     > > > > >     > > >     >     >         >> others
>     > > > > >     > > >     >     >         >> > this message or any
> attachment.
>     > > > Please
>     > > > > > also
>     > > > > >     > > > notify the
>     > > > > >     > > >     > sender
>     > > > > >     > > >     >     > by
>     > > > > >     > > >     >     >         >> replying
>     > > > > >     > > >     >     >         >> > to this email or by
> telephone
>     > > > (+44(020
>     > > > > > 7896
>     > > > > >     > > 0011)
>     > > > > >     > > > and
>     > > > > >     > > >     > then
>     > > > > >     > > >     >     > delete the
>     > > > > >     > > >     >     >         >> email
>     > > > > >     > > >     >     >         >> > and any copies of it.
> Opinions,
>     > > > > > conclusion
>     > > > > >     > (etc)
>     > > > > >     > > > that
>     > > > > >     > > >     > do not
>     > > > > >     > > >     >     > relate to
>     > > > > >     > > >     >     >         >> the
>     > > > > >     > > >     >     >         >> > official business of this
>     > company
>     > > > > shall
>     > > > > > be
>     > > > > >     > > > understood as
>     > > > > >     > > >     >     > neither given
>     > > > > >     > > >     >     >         >> nor
>     > > > > >     > > >     >     >         >> > endorsed by it. IG is a
> trading
>     > > name
>     > > > > of
>     > > > > > IG
>     > > > > >     > > Markets
>     > > > > >     > > >     > Limited (a
>     > > > > >     > > >     >     > company
>     > > > > >     > > >     >     >         >> > registered in England and
> Wales,
>     > > > > company
>     > > > > >     > number
>     > > > > >     > > >     > 04008957) and
>     > > > > >     > > >     >     > IG Index
>     > > > > >     > > >     >     >         >> > Limited (a company
> registered in
>     > > > > > England and
>     > > > > >     > > > Wales,
>     > > > > >     > > >     > company
>     > > > > >     > > >     >     > number
>     > > > > >     > > >     >     >         >> > 01190902). Registered
> address at
>     > > > > Cannon
>     > > > > > Bridge
>     > > > > >     > > > House, 25
>     > > > > >     > > >     >     > Dowgate Hill,
>     > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
> Markets
>     > > > > Limited
>     > > > > >     > > (register
>     > > > > >     > > >     > number
>     > > > > >     > > >     >     > 195355) and IG
>     > > > > >     > > >     >     >         >> > Index Limited (register
> number
>     > > > 114059)
>     > > > > > are
>     > > > > >     > > > authorised
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     > regulated by
>     > > > > >     > > >     >     >         >> the
>     > > > > >     > > >     >     >         >> > Financial Conduct
> Authority.
>     > > > > >     > > >     >     >         >> >
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >> --
>     > > > > >     > > >     >     >         >> -Regards,
>     > > > > >     > > >     >     >         >> Mayuresh R. Gharat
>     > > > > >     > > >     >     >         >> (862) 250-7125
>     > > > > >     > > >     >     >         >> The information contained
> in this
>     > > > email
>     > > > > is
>     > > > > >     > > strictly
>     > > > > >     > > >     >     > confidential and for
>     > > > > >     > > >     >     >         >> the use of the addressee
> only,
>     > > unless
>     > > > > > otherwise
>     > > > > >     > > >     > indicated. If
>     > > > > >     > > >     >     > you are not
>     > > > > >     > > >     >     >         >> the intended recipient,
> please do
>     > > not
>     > > > > > read,
>     > > > > >     > copy,
>     > > > > >     > > > use or
>     > > > > >     > > >     >     > disclose to others
>     > > > > >     > > >     >     >         >> this message or any
> attachment.
>     > > Please
>     > > > > > also
>     > > > > >     > notify
>     > > > > >     > > > the
>     > > > > >     > > >     > sender
>     > > > > >     > > >     >     > by replying
>     > > > > >     > > >     >     >         >> to this email or by
> telephone
>     > > (+44(020
>     > > > > > 7896
>     > > > > >     > 0011)
>     > > > > >     > > > and then
>     > > > > >     > > >     >     > delete the email
>     > > > > >     > > >     >     >         >> and any copies of it.
> Opinions,
>     > > > > > conclusion (etc)
>     > > > > >     > > > that do
>     > > > > >     > > >     > not
>     > > > > >     > > >     >     > relate to the
>     > > > > >     > > >     >     >         >> official business of this
> company
>     > > > shall
>     > > > > be
>     > > > > >     > > > understood as
>     > > > > >     > > >     >     > neither given nor
>     > > > > >     > > >     >     >         >> endorsed by it. IG is a
> trading
>     > name
>     > > > of
>     > > > > IG
>     > > > > >     > Markets
>     > > > > >     > > >     > Limited (a
>     > > > > >     > > >     >     > company
>     > > > > >     > > >     >     >         >> registered in England and
> Wales,
>     > > > company
>     > > > > > number
>     > > > > >     > > > 04008957)
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     > IG Index
>     > > > > >     > > >     >     >         >> Limited (a company
> registered in
>     > > > England
>     > > > > > and
>     > > > > >     > > Wales,
>     > > > > >     > > >     > company
>     > > > > >     > > >     >     > number
>     > > > > >     > > >     >     >         >> 01190902). Registered
> address at
>     > > > Cannon
>     > > > > > Bridge
>     > > > > >     > > > House, 25
>     > > > > >     > > >     >     > Dowgate Hill,
>     > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> Markets
>     > > > Limited
>     > > > > >     > (register
>     > > > > >     > > > number
>     > > > > >     > > >     >     > 195355) and IG
>     > > > > >     > > >     >     >         >> Index Limited (register
> number
>     > > 114059)
>     > > > > are
>     > > > > >     > > > authorised and
>     > > > > >     > > >     >     > regulated by the
>     > > > > >     > > >     >     >         >> Financial Conduct Authority.
>     > > > > >     > > >     >     >         >>
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >         > --
>     > > > > >     > > >     >     >         > -Regards,
>     > > > > >     > > >     >     >         > Mayuresh R. Gharat
>     > > > > >     > > >     >     >         > (862) 250-7125
>     > > > > >     > > >     >     >         >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >         --
>     > > > > >     > > >     >     >         -Regards,
>     > > > > >     > > >     >     >         Mayuresh R. Gharat
>     > > > > >     > > >     >     >         (862) 250-7125
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >     The information contained in this
> email
>     > is
>     > > > > > strictly
>     > > > > >     > > > confidential
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     > for the use of the addressee only,
> unless
>     > > > otherwise
>     > > > > >     > > indicated.
>     > > > > >     > > > If
>     > > > > >     > > >     > you are
>     > > > > >     > > >     >     > not the intended recipient, please do
> not
>     > read,
>     > > > > > copy, use
>     > > > > >     > or
>     > > > > >     > > >     > disclose to
>     > > > > >     > > >     >     > others this message or any attachment.
> Please
>     > > > also
>     > > > > > notify
>     > > > > >     > the
>     > > > > >     > > > sender
>     > > > > >     > > >     > by
>     > > > > >     > > >     >     > replying to this email or by telephone
>     > (+44(020
>     > > > > 7896
>     > > > > > 0011)
>     > > > > >     > > and
>     > > > > >     > > > then
>     > > > > >     > > >     > delete
>     > > > > >     > > >     >     > the email and any copies of it.
> Opinions,
>     > > > > conclusion
>     > > > > > (etc)
>     > > > > >     > > > that do
>     > > > > >     > > >     > not
>     > > > > >     > > >     >     > relate to the official business of this
>     > company
>     > > > > > shall be
>     > > > > >     > > > understood
>     > > > > >     > > >     > as
>     > > > > >     > > >     >     > neither given nor endorsed by it. IG
> is a
>     > > trading
>     > > > > > name of
>     > > > > >     > IG
>     > > > > >     > > > Markets
>     > > > > >     > > >     >     > Limited (a company registered in
> England and
>     > > > Wales,
>     > > > > > company
>     > > > > >     > > > number
>     > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> company
>     > > > > registered
>     > > > > > in
>     > > > > >     > > > England and
>     > > > > >     > > >     > Wales,
>     > > > > >     > > >     >     > company number 01190902). Registered
> address
>     > at
>     > > > > > Cannon
>     > > > > >     > Bridge
>     > > > > >     > > > House,
>     > > > > >     > > >     > 25
>     > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG
>     > Markets
>     > > > > > Limited
>     > > > > >     > > > (register
>     > > > > >     > > >     > number
>     > > > > >     > > >     >     > 195355) and IG Index Limited (register
> number
>     > > > > > 114059) are
>     > > > > >     > > > authorised
>     > > > > >     > > >     > and
>     > > > > >     > > >     >     > regulated by the Financial Conduct
> Authority.
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >     >
>     > > > > >     > > >     >
>     > > > > >     > > >     >
>     > > > > >     > > >     > The information contained in this email is
> strictly
>     > > > > > confidential
>     > > > > >     > > and
>     > > > > >     > > > for
>     > > > > >     > > >     > the use of the addressee only, unless
> otherwise
>     > > > > indicated.
>     > > > > > If you
>     > > > > >     > > > are not
>     > > > > >     > > >     > the intended recipient, please do not read,
> copy,
>     > use
>     > > > or
>     > > > > > disclose
>     > > > > >     > > to
>     > > > > >     > > > others
>     > > > > >     > > >     > this message or any attachment. Please also
> notify
>     > > the
>     > > > > > sender by
>     > > > > >     > > > replying
>     > > > > >     > > >     > to this email or by telephone (+44(020 7896
> 0011
>     > <020%207896%200011>) and
>     > > > > then
>     > > > > > delete
>     > > > > >     > > > the email
>     > > > > >     > > >     > and any copies of it. Opinions, conclusion
> (etc)
>     > that
>     > > > do
>     > > > > > not
>     > > > > >     > relate
>     > > > > >     > > > to the
>     > > > > >     > > >     > official business of this company shall be
>     > understood
>     > > > as
>     > > > > > neither
>     > > > > >     > > > given nor
>     > > > > >     > > >     > endorsed by it. IG is a trading name of IG
> Markets
>     > > > > Limited
>     > > > > > (a
>     > > > > >     > > company
>     > > > > >     > > >     > registered in England and Wales, company
> number
>     > > > 04008957)
>     > > > > > and IG
>     > > > > >     > > > Index
>     > > > > >     > > >     > Limited (a company registered in England and
> Wales,
>     > > > > company
>     > > > > >     > number
>     > > > > >     > > >     > 01190902). Registered address at Cannon
> Bridge
>     > House,
>     > > > 25
>     > > > > > Dowgate
>     > > > > >     > > > Hill,
>     > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
> (register
>     > > > number
>     > > > > > 195355)
>     > > > > >     > > > and IG
>     > > > > >     > > >     > Index Limited (register number 114059) are
>     > authorised
>     > > > and
>     > > > > >     > regulated
>     > > > > >     > > > by the
>     > > > > >     > > >     > Financial Conduct Authority.
>     > > > > >     > > >     >
>     > > > > >     > > >
>     > > > > >     > > >
>     > > > > >     > > > The information contained in this email is strictly
>     > > > > confidential
>     > > > > > and
>     > > > > >     > for
>     > > > > >     > > > the use of the addressee only, unless otherwise
>     > indicated.
>     > > If
>     > > > > > you are
>     > > > > >     > not
>     > > > > >     > > > the intended recipient, please do not read, copy,
> use or
>     > > > > > disclose to
>     > > > > >     > > others
>     > > > > >     > > > this message or any attachment. Please also notify
> the
>     > > sender
>     > > > > by
>     > > > > >     > replying
>     > > > > >     > > > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>) and then
>     > > > > > delete the
>     > > > > >     > > email
>     > > > > >     > > > and any copies of it. Opinions, conclusion (etc)
> that do
>     > > not
>     > > > > > relate to
>     > > > > >     > > the
>     > > > > >     > > > official business of this company shall be
> understood as
>     > > > > neither
>     > > > > > given
>     > > > > >     > > nor
>     > > > > >     > > > endorsed by it. IG is a trading name of IG Markets
>     > Limited
>     > > (a
>     > > > > > company
>     > > > > >     > > > registered in England and Wales, company number
> 04008957)
>     > > and
>     > > > > IG
>     > > > > > Index
>     > > > > >     > > > Limited (a company registered in England and Wales,
>     > company
>     > > > > > number
>     > > > > >     > > > 01190902). Registered address at Cannon Bridge
> House, 25
>     > > > > Dowgate
>     > > > > > Hill,
>     > > > > >     > > > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > > > > 195355) and
>     > > > > >     > IG
>     > > > > >     > > > Index Limited (register number 114059) are
> authorised and
>     > > > > > regulated by
>     > > > > >     > > the
>     > > > > >     > > > Financial Conduct Authority.
>     > > > > >     > > >
>     > > > > >     > > The information contained in this email is strictly
>     > > > confidential
>     > > > > > and for
>     > > > > >     > > the use of the addressee only, unless otherwise
> indicated.
>     > If
>     > > > you
>     > > > > > are not
>     > > > > >     > > the intended recipient, please do not read, copy,
> use or
>     > > > disclose
>     > > > > > to
>     > > > > >     > others
>     > > > > >     > > this message or any attachment. Please also notify
> the
>     > sender
>     > > > by
>     > > > > > replying
>     > > > > >     > > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>) and then
>     > > > delete
>     > > > > > the
>     > > > > >     > email
>     > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> that do
>     > not
>     > > > > > relate to
>     > > > > >     > the
>     > > > > >     > > official business of this company shall be
> understood as
>     > > > neither
>     > > > > > given
>     > > > > >     > nor
>     > > > > >     > > endorsed by it. IG is a trading name of IG Markets
> Limited
>     > (a
>     > > > > > company
>     > > > > >     > > registered in England and Wales, company number
> 04008957)
>     > and
>     > > > IG
>     > > > > > Index
>     > > > > >     > > Limited (a company registered in England and Wales,
> company
>     > > > > number
>     > > > > >     > > 01190902). Registered address at Cannon Bridge
> House, 25
>     > > > Dowgate
>     > > > > > Hill,
>     > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > > 195355)
>     > > > > > and IG
>     > > > > >     > > Index Limited (register number 114059) are
> authorised and
>     > > > > > regulated by
>     > > > > >     > the
>     > > > > >     > > Financial Conduct Authority.
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > The information contained in this email is strictly
>     > > confidential
>     > > > > and
>     > > > > > for
>     > > > > >     > the use of the addressee only, unless otherwise
> indicated. If
>     > > you
>     > > > > > are not
>     > > > > >     > the intended recipient, please do not read, copy, use
> or
>     > > disclose
>     > > > > to
>     > > > > > others
>     > > > > >     > this message or any attachment. Please also notify the
> sender
>     > > by
>     > > > > > replying
>     > > > > >     > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>) and then
>     > > delete
>     > > > > > the email
>     > > > > >     > and any copies of it. Opinions, conclusion (etc) that
> do not
>     > > > relate
>     > > > > > to the
>     > > > > >     > official business of this company shall be understood
> as
>     > > neither
>     > > > > > given nor
>     > > > > >     > endorsed by it. IG is a trading name of IG Markets
> Limited (a
>     > > > > company
>     > > > > >     > registered in England and Wales, company number
> 04008957) and
>     > > IG
>     > > > > > Index
>     > > > > >     > Limited (a company registered in England and Wales,
> company
>     > > > number
>     > > > > >     > 01190902). Registered address at Cannon Bridge House,
> 25
>     > > Dowgate
>     > > > > > Hill,
>     > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > 195355)
>     > > > > > and IG
>     > > > > >     > Index Limited (register number 114059) are authorised
> and
>     > > > regulated
>     > > > > > by the
>     > > > > >     > Financial Conduct Authority.
>     > > > > >     >
>     > > > > >     >
>     > > > > >
>     > > > > >
>     > > > > > The information contained in this email is strictly
> confidential
>     > and
>     > > > for
>     > > > > > the use of the addressee only, unless otherwise indicated.
> If you
>     > are
>     > > > not
>     > > > > > the intended recipient, please do not read, copy, use or
> disclose
>     > to
>     > > > > others
>     > > > > > this message or any attachment. Please also notify the
> sender by
>     > > > replying
>     > > > > > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>) and then delete the
>     > > > > email
>     > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > > to
>     > > > > the
>     > > > > > official business of this company shall be understood as
> neither
>     > > given
>     > > > > nor
>     > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>     > company
>     > > > > > registered in England and Wales, company number 04008957)
> and IG
>     > > Index
>     > > > > > Limited (a company registered in England and Wales, company
> number
>     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > > Hill,
>     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
>     > and
>     > > > IG
>     > > > > > Index Limited (register number 114059) are authorised and
> regulated
>     > > by
>     > > > > the
>     > > > > > Financial Conduct Authority.
>     > > > > >
>     > > > > The information contained in this email is strictly
> confidential and
>     > > for
>     > > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > > not
>     > > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > > others
>     > > > > this message or any attachment. Please also notify the sender
> by
>     > > replying
>     > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > and then delete the
>     > > > email
>     > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > to
>     > > > the
>     > > > > official business of this company shall be understood as
> neither
>     > given
>     > > > nor
>     > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > > registered in England and Wales, company number 04008957) and
> IG
>     > Index
>     > > > > Limited (a company registered in England and Wales, company
> number
>     > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > Hill,
>     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > > IG
>     > > > > Index Limited (register number 114059) are authorised and
> regulated
>     > by
>     > > > the
>     > > > > Financial Conduct Authority.
>     > > > >
>     > > > The information contained in this email is strictly confidential
> and
>     > for
>     > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > not
>     > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > others
>     > > > this message or any attachment. Please also notify the sender by
>     > replying
>     > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > and then delete the
>     > > email
>     > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > > the
>     > > > official business of this company shall be understood as neither
> given
>     > > nor
>     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > > Limited (a company registered in England and Wales, company
> number
>     > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > IG
>     > > > Index Limited (register number 114059) are authorised and
> regulated by
>     > > the
>     > > > Financial Conduct Authority.
>     > > >
>     > > The information contained in this email is strictly confidential
> and for
>     > > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > > the intended recipient, please do not read, copy, use or disclose
> to
>     > others
>     > > this message or any attachment. Please also notify the sender by
> replying
>     > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > and then delete the email
>     > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > the
>     > > official business of this company shall be understood as neither
> given
>     > nor
>     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > Limited (a company registered in England and Wales, company number
>     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > > Index Limited (register number 114059) are authorised and
> regulated by
>     > the
>     > > Financial Conduct Authority.
>     > >
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Chaps,

Can we either get one more +1 binding (we have 2 already) on the existing?

Or have some response on the below possible alternative. We are keen to get working on this, so we make next feature release.

Cheers
Mike


On 13/12/2016, 16:32, "Michael Pearce" <Mi...@ig.com> wrote:

    Hi Ismael

    Did you see our email this morning, what's your thoughts on this approach to instead we simply have a brand new policy?

    Cheers
    Mike


    Sent using OWA for iPhone
    ________________________________________
    From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <is...@juma.me.uk>
    Sent: Tuesday, December 13, 2016 11:30:05 AM
    To: dev@kafka.apache.org
    Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

    Yes, this is actually tricky to do in a way where we both have the desired
    semantics and maintain compatibility. When someone creates a
    `ProducerRecord` with a `null` value today, the producer doesn't know if
    it's meant to be a tombstone or not. For V3 messages, it's easy when the
    constructor that takes a tombstone is used. However, if any other
    constructor is used, it's not clear. A couple of options I can think of,
    none of them particularly nice:

    1. Have a third state where tombstone = unknown and the broker would set
    the tombstone bit if the value was null and the topic was compacted. People
    that wanted to pass a non-null value for the tombstone would have to use
    the constructor that takes a tombstone. The drawbacks: third state for
    tombstone in message format, message conversion at the broker for a common
    case.

    2. Extend MetadataResponse to optionally include topic configs, which would
    make it possible for the producer to be smarter about setting the
    tombstone. It would only do it if a tombstone was not passed explicitly,
    the value was null and the topic was compacted. The main drawback is that
    the producer would be getting a bit more data for each topic even though it
    probably won't use it most of the time. Extending MetadataResponse to
    return topic configs would be useful for other reasons as well, so that
    part seems OK.

    In addition, for both proposals, we could consider adding warnings to the
    documentation that the behaviour of the constructors that don't take a
    tombstone would change in the next major release so that tombstone = false.
    Not sure if this would be worth it though.

    Ismael

    On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <ew...@confluent.io>
    wrote:

    > Michael,
    >
    > It kind of depends on how you want to interpret the tombstone flag. If it's
    > purely a producer-facing Kafka-level thing that we treat as internal to the
    > broker and log cleaner once the record is sent, then your approach makes
    > sense. You're just moving copying the null-indicates-delete behavior of the
    > old constructor into the tombstone flag.
    >
    > However, if you want this change to more generally decouple the idea of
    > deletion and null values, then you are sometimes converting what might be a
    > completely valid null value that doesn't indicate deletion into a
    > tombstone. Downstream applications could potentially handle these cases
    > differently given the separation of deletion from value.
    >
    > I guess the question is if we want to try to support the latter even for
    > topics where we have older produce requests. An example where this could
    > come up is in something like a CDC Connector. If we try to support the
    > semantic difference, a connector might write changes to Kafka using the
    > tombstone flag to indicate when a row was truly deleted (vs an update that
    > sets it to null but still present; this probably makes more sense for CDC
    > from document stores or extracting single columns). There are various
    > reasons we might want to maintain the full log and not turn compaction on
    > (or just use a time-based retention policy), but downstream applications
    > might care to know the difference between a delete and a null value. In
    > fact both versions of the same log (compacted and time-retention) could be
    > useful and I don't think it'll be uncommon to maintain both or use KIP-71
    > to maintain a hybrid compacted/retention topic.
    >
    > -Ewen
    >
    > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <Mi...@ig.com>
    > wrote:
    >
    > > Hi Jay,
    > >
    > > Why wouldn't that work, the tombstone value is only looked at by the
    > > broker, on a topic configured for compaction as such is benign on non
    > > compacted topics. This is as much as sending a null value currently
    > >
    > >
    > > Regards
    > > Mike
    > >
    > >
    > >
    > > Sent using OWA for iPhone
    > > ________________________________________
    > > From: Jay Kreps <ja...@confluent.io>
    > > Sent: Sunday, December 11, 2016 8:58:53 PM
    > > To: dev@kafka.apache.org
    > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > >
    > > Hey Michael,
    > >
    > > I'm not quite sure that works as that would translate ALL null values to
    > > tombstones, even for non-compacted topics that use null as an acceptable
    > > value sent by the producer and expected by the consumer.
    > >
    > > -Jay
    > >
    > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Mi...@ig.com>
    > > wrote:
    > >
    > > > Hi Ewen,
    > > >
    > > > I think the easiest way to show this is with code.
    > > >
    > > > As you can see we keep the existing behaviour for code/binaries calling
    > > > the pre-existing constructors, whereby if the value is null the
    > tombstone
    > > > is set to true.
    > > >
    > > > Regards
    > > > Mike
    > > >
    > > >
    > > >
    > > >     /**
    > > >      * Creates a record with a specified timestamp to be sent to a
    > > > specified topic and partition
    > > >      *
    > > >      * @param topic The topic the record will be appended to
    > > >      * @param partition The partition to which the record should be
    > sent
    > > >      * @param timestamp The timestamp of the record
    > > >      * @param tombstone if the record should be treated as a tombstone
    > if
    > > > the topic is compacted
    > > >      * @param key The key that will be included in the record
    > > >      * @param value The record contents
    > > >      */
    > > >     public ProducerRecord(String topic, Integer partition, Boolean
    > > > tombstone, Long timestamp, K key, V value) {
    > > >         if (topic == null)
    > > >             throw new IllegalArgumentException("Topic cannot be
    > null.");
    > > >         if (timestamp != null && timestamp < 0)
    > > >             throw new IllegalArgumentException(
    > > >                     String.format("Invalid timestamp: %d. Timestamp
    > > should
    > > > always be non-negative or null.", timestamp));
    > > >         if (partition != null && partition < 0)
    > > >             throw new IllegalArgumentException(
    > > >                     String.format("Invalid partition: %d. Partition
    > > number
    > > > should always be non-negative or null.", partition));
    > > >         if (!tombstone && value == null){
    > > >             throw new IllegalArgumentException(
    > > >                     String.format("Tombstone must be true if null
    > > value"));
    > > >         }
    > > >         this.topic = topic;
    > > >         this.partition = partition;
    > > >         this.tombstone = tombstone;
    > > >         this.key = key;
    > > >         this.value = value;
    > > >         this.timestamp = timestamp;
    > > >     }
    > > >
    > > >     public ProducerRecord(String topic, Integer partition, Long
    > > timestamp,
    > > > K key, V value) {
    > > >         this(topic, partition, value == null, timestamp, key, value);
    > > >     }
    > > > ________________________________________
    > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
    > > > Sent: Sunday, December 11, 2016 5:45 AM
    > > > To: dev@kafka.apache.org
    > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > >
    > > > It seemed like this was addressed in the migration section, wasn't it?
    > > The
    > > > V2 vs V3 requests and the fact that the broker will downgrade the
    > message
    > > > format (losing zero copy) if you issues a V2 request to a broker using
    > V3
    > > > format handles compatibility, does it not? The existing consumers won't
    > > see
    > > > the extra metadata in the value, but they will get a null instead and
    > > treat
    > > > it as a tombstone. Certainly there is a performance impact, but it
    > seemed
    > > > compatible.
    > > >
    > > > I'm worried about this though:
    > > >
    > > >
    > > >    - *NOTE *: With the new version of producer client using
    > > ProduceRequest
    > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and
    > null
    > > > value
    > > >    should not be allowed as the older version of consumer using
    > > > FetchRequest
    > > >    V2 will think of this as a tombstone when its actually not.
    > > >
    > > >
    > > > Unless I'm misunderstanding, this ends up breaking binary compatibility
    > > for
    > > > the Java API. It sounds like this suggests that passing a null value to
    > > the
    > > > existing ProducerRecord constructors would generate an exception since
    > > you
    > > > didn't explicitly enable the tombstone (via whatever new constructor is
    > > > provided). But that means you can't swap in a newer client jar without
    > > > recompiling and get the same behavior if your app deletes keys using
    > the
    > > > current approach because your app will start throwing exceptions. Maybe
    > > > this is a tradeoff we're ok with, but we've tried pretty hard to avoid
    > > > breaking compatibility like this so far -- it makes picking up bug
    > fixes
    > > in
    > > > the clients harder for users of frameworks that have to pin to earlier
    > > > default versions for compatibility.
    > > >
    > > > But then later the KIP says:
    > > >
    > > >
    > > >    - The old constructors for ProducerRecord without this parameter
    > will
    > > be
    > > >    kept but updated so that their default behaviour if setting null
    > value
    > > > will
    > > >    be the tombstone will be set to true to keep existing behaviour.
    > > >
    > > >
    > > > So maybe I am misinterpreting something.
    > > >
    > > > And just a nit re: motivation section, item 6 would be clearer for a
    > > union
    > > > schema with null (or optional schemas in other formats), e.g. [null,
    > > > string], in which case losing the schema truly is losing information
    > > > (whereas null is already the only valid value for a pure null schema).
    > > >
    > > > -Ewen
    > > >
    > > >
    > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Michael.Pearce@ig.com
    > >
    > > > wrote:
    > > >
    > > > > Hi Jay,
    > > > >
    > > > > Good point this detail is missing in the KIP write up. Ive added this
    > > > now.
    > > > >
    > > > > Essentially simply just upgrading the clients we do not expect any
    > > client
    > > > > app code change needed.
    > > > >
    > > > > Cheers
    > > > > Mike
    > > > > ________________________________________
    > > > > From: Jay Kreps <ja...@confluent.io>
    > > > > Sent: Saturday, December 10, 2016 10:51 PM
    > > > > To: dev@kafka.apache.org
    > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > >
    > > > > Michael,
    > > > >
    > > > > The compatibility section goes through the migration path, but isn't
    > > the
    > > > > bigger compatibility issue with existing apps? There are many
    > (probably
    > > > > thousands) of apps in production that use this feature and send null
    > to
    > > > > mean delete. It seems like this would break compatibility with them,
    > > and
    > > > > they would have to be rewritten to right?
    > > > >
    > > > > -Jay
    > > > >
    > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
    > Michael.Pearce@ig.com
    > > >
    > > > > wrote:
    > > > >
    > > > > > Hi Jun,
    > > > > >
    > > > > > 4) On v3 we honour the tombstone. As such we expect it to be set
    > > > > correctly
    > > > > > as per the KIP.
    > > > > >
    > > > > > 4.1) why would we want to produce an error when its v3? This is the
    > > > exact
    > > > > > purpose to support non-null tombstone’s
    > > > > > 4.2) again here im unclear on the question on the v3, produce
    > request
    > > > we
    > > > > > expect the tombstone flag to be set correctly.
    > > > > >
    > > > > > 4.4) compaction only occurs on compacted topics, the bit makes no
    > > > > > difference and not looked at on un-compacted (time/size based
    > > > eviction).
    > > > > >
    > > > > >
    > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
    > > > > >
    > > > > >     Hi, Michael,
    > > > > >
    > > > > >     4. Then, I think I misunderstood this point. Could you document
    > > the
    > > > > >     following points in the wiki?
    > > > > >     4.1 If producer V3 sets tombstone, but provides a non-null
    > value,
    > > > > does
    > > > > > the
    > > > > >     send() get an error or does the producer automatically set the
    > > > value
    > > > > to
    > > > > >     null?
    > > > > >     4.2 If producer V3 doesn't set tombstone, but provides a null
    > > > value,
    > > > > > does
    > > > > >     the send() get an error or does the producer automatically sets
    > > the
    > > > > >     tombstone?
    > > > > >     4.3 Does the broker only expect messages that (a) have no
    > > tombstone
    > > > > and
    > > > > >     non-null value; (b) have tombstone and null value and reject
    > the
    > > > > > messages
    > > > > >     with an error code otherwise?
    > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted
    > on
    > > > > not?
    > > > > >
    > > > > >     Thanks,
    > > > > >
    > > > > >     Jun
    > > > > >
    > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
    > > > > Michael.Pearce@ig.com
    > > > > > >
    > > > > >     wrote:
    > > > > >
    > > > > >     > Not at all.  This only acts on compacted topics just as what
    > > > occurs
    > > > > > today
    > > > > >     >
    > > > > >     > Sent using OWA for iPhone
    > > > > >     > ________________________________________
    > > > > >     > From: Jun Rao <ju...@confluent.io>
    > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
    > > > > >     > To: dev@kafka.apache.org
    > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > > >     >
    > > > > >     > Hi, Michael,
    > > > > >     >
    > > > > >     > 4. Hmm, does that mean the new client library can never send
    > a
    > > > null
    > > > > > message
    > > > > >     > even to a regular topic? This seems like a change of the
    > > existing
    > > > > > behavior.
    > > > > >     >
    > > > > >     > Thanks,
    > > > > >     >
    > > > > >     > Jun
    > > > > >     >
    > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
    > > > > > Michael.Pearce@ig.com>
    > > > > >     > wrote:
    > > > > >     >
    > > > > >     > > Hi Jun,
    > > > > >     > >
    > > > > >     > > Re 4) That's because we expect the tombstone value to be
    > set
    > > > > > correctly if
    > > > > >     > > message bit is 2, as such if an older client sends in on
    > old
    > > > > > message the
    > > > > >     > > message is upcast and the bit is set correctly. And such no
    > > > > longer
    > > > > > need
    > > > > >     > to
    > > > > >     > > check the value. Mayuresh can you confirm my thinking and
    > > > > > understanding
    > > > > >     > of
    > > > > >     > > what we've discussed?
    > > > > >     > >
    > > > > >     > > The second point I understand what you're getting at now my
    > > > > > apologies.
    > > > > >     > Yes
    > > > > >     > > this makes sense to save on touching the message, if we're
    > > the
    > > > > > only kip
    > > > > >     > > going in, in this release.
    > > > > >     > >
    > > > > >     > > Cheers
    > > > > >     > > Mike
    > > > > >     > >
    > > > > >     > > Sent using OWA for iPhone
    > > > > >     > > ________________________________________
    > > > > >     > > From: Jun Rao <ju...@confluent.io>
    > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
    > > > > >     > > To: dev@kafka.apache.org
    > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > > >     > >
    > > > > >     > > Hi, Michael,
    > > > > >     > >
    > > > > >     > > 4. Is this updated in the wiki? The text "If the magic byte
    > > on
    > > > > > message is
    > > > > >     > > 2, the broker should use the tombstone bit for log
    > > compaction."
    > > > > > doesn't
    > > > > >     > > seem to have changed.
    > > > > >     > >
    > > > > >     > > 2. My point is that if we change the message format just
    > for
    > > > this
    > > > > > KIP, we
    > > > > >     > > should consider whether it's worth optimizing the down
    > > > conversion
    > > > > > path
    > > > > >     > > (i.e., decide whether a conversion is needed by just
    > looking
    > > at
    > > > > the
    > > > > >     > > tombstone bit in the wrapper message) since tombstone will
    > be
    > > > > used
    > > > > >     > rarely.
    > > > > >     > > However, if the message format change here is combined with
    > > > other
    > > > > > KIPs,
    > > > > >     > > then this optimization likely won't be needed. The latter
    > > > > probably
    > > > > > makes
    > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
    > > > > >     > >
    > > > > >     > > Other than those, +1 from me,
    > > > > >     > >
    > > > > >     > > Thanks,
    > > > > >     > >
    > > > > >     > > Jun
    > > > > >     > >
    > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
    > > > > > Michael.Pearce@ig.com>
    > > > > >     > > wrote:
    > > > > >     > >
    > > > > >     > > > Hi Jun
    > > > > >     > > >
    > > > > >     > > > do we have your vote on this now?
    > > > > >     > > >
    > > > > >     > > > Any other concerns?
    > > > > >     > > >
    > > > > >     > > > Cheers
    > > > > >     > > > Mike
    > > > > >     > > >
    > > > > >     > > > Sent using OWA for iPhone
    > > > > >     > > > ________________________________________
    > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
    > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
    > > > > >     > > > To: dev@kafka.apache.org
    > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
    > Flag
    > > > > >     > > >
    > > > > >     > > > Hi Jun,
    > > > > >     > > >
    > > > > >     > > > Have updated. Thanks again for the feedback.
    > > > > >     > > >
    > > > > >     > > > Agree yes we should align up when it gets to that, I
    > assume
    > > > > > you’ve
    > > > > >     > > flagged
    > > > > >     > > > the same in the other KIP?
    > > > > >     > > >
    > > > > >     > > > I think for now let’s get this KIP approved, then we can
    > > > start
    > > > > > the work
    > > > > >     > > to
    > > > > >     > > > get an initial PR, then we can discuss how to align the
    > two
    > > > if
    > > > > > needed.
    > > > > >     > > >
    > > > > >     > > > Cheers,
    > > > > >     > > > Mike
    > > > > >     > > >
    > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io>
    > wrote:
    > > > > >     > > >
    > > > > >     > > >     Hi, Michael,
    > > > > >     > > >
    > > > > >     > > >     For 2), this is fine. Could you update the KIP wiki
    > to
    > > > make
    > > > > > this
    > > > > >     > and
    > > > > >     > > > other
    > > > > >     > > >     points clearer? Other than that, the KIP looks good
    > to
    > > > me.
    > > > > >     > > >
    > > > > >     > > >     An orthogonal thing is that there are other KIPs such
    > > as
    > > > > > KIP-98
    > > > > >     > that
    > > > > >     > > > also
    > > > > >     > > >     intend to change the message format. If they all get
    > > > > > approved, we
    > > > > >     > > > should
    > > > > >     > > >     think about whether it's better to just bump up the
    > > magic
    > > > > > byte once
    > > > > >     > > to
    > > > > >     > > >     incorporate multiple format changes like we did in
    > > > > > KIP-31/KIP-32.
    > > > > >     > > >
    > > > > >     > > >     Thanks,
    > > > > >     > > >
    > > > > >     > > >     Jun
    > > > > >     > > >
    > > > > >     > > >
    > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
    > > > > >     > > Michael.Pearce@ig.com>
    > > > > >     > > >     wrote:
    > > > > >     > > >
    > > > > >     > > >     > Hi Jao
    > > > > >     > > >     >
    > > > > >     > > >     > Thanks for the response. Sorry for slow reply, both
    > > > with
    > > > > > personal
    > > > > >     > > > sickness
    > > > > >     > > >     > and also battling some critical issues encountered
    > > > since
    > > > > >     > upgrading
    > > > > >     > > to
    > > > > >     > > >     > 0.10.1.0
    > > > > >     > > >     >
    > > > > >     > > >     > 1) Thans for spotting, Document error where we
    > > branched
    > > > > > this KIP
    > > > > >     > > from
    > > > > >     > > >     > KIP-82, will get that removed.
    > > > > >     > > >     > 2) Intent is to do this just at the record message
    > > > level.
    > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
    > > corrected.
    > > > > >     > > >     > 4) As per discussion thread we will support
    > > tombstone +
    > > > > > null
    > > > > >     > value,
    > > > > >     > > >     > tombstone + non null value, no tombstone + null
    > > value.
    > > > > >     > > >     > 5) I believe this was in the discussion thread,
    > > > @Mayuresh
    > > > > > is this
    > > > > >     > > >     > something we’ve overlooked? I thought we would down
    > > > > > convert and
    > > > > >     > > > remove the
    > > > > >     > > >     > value so the old consumer had existing behavior, or
    > > is
    > > > > > there
    > > > > >     > > > something we
    > > > > >     > > >     > haven’t thought about?
    > > > > >     > > >     >
    > > > > >     > > >     > Cheers
    > > > > >     > > >     > Mike
    > > > > >     > > >     >
    > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
    > > > > wrote:
    > > > > >     > > >     >
    > > > > >     > > >     >     Hi, Michael,
    > > > > >     > > >     >
    > > > > >     > > >     >     Thanks for the KIP. A few comments below.
    > > > > >     > > >     >
    > > > > >     > > >     >     1. The message format change contains
    > > > "HeadersLength
    > > > > >     > Headers".
    > > > > >     > > > Is that
    > > > > >     > > >     >     intended?
    > > > > >     > > >     >
    > > > > >     > > >     >     2. For compressed messageset, is the tombstone
    > > bit
    > > > > > only set
    > > > > >     > at
    > > > > >     > > > the
    > > > > >     > > >     > shallow
    > > > > >     > > >     >     level? Do we always leave that bit in the
    > wrapper
    > > > > > message
    > > > > >     > > unset?
    > > > > >     > > > An
    > > > > >     > > >     >     alternative is to set the tombstone bit in the
    > > > > wrapper
    > > > > > if at
    > > > > >     > > > least one
    > > > > >     > > >     >     inner message has the tombstone bit set. This
    > > makes
    > > > > > things a
    > > > > >     > > bit
    > > > > >     > > > more
    > > > > >     > > >     >     complicated, but we could potentially exploit
    > > that
    > > > > for
    > > > > >     > > > optimizing down
    > > > > >     > > >     >     conversion. For example, we only need to
    > convert
    > > > > > messages
    > > > > >     > with
    > > > > >     > > > magic 2
    > > > > >     > > >     > to
    > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
    > > > > > (conversion is
    > > > > >     > > > always
    > > > > >     > > >     > needed
    > > > > >     > > >     >     from magic 2 to magic 0). Not sure if the
    > > > > optimization
    > > > > > is
    > > > > >     > worth
    > > > > >     > > > the
    > > > > >     > > >     >     complexity though.
    > > > > >     > > >     >
    > > > > >     > > >     >     3. The referencing of the new version of
    > > > > >     > > > ProducerRequest/FetchRequest
    > > > > >     > > >     > is
    > > > > >     > > >     >     inconsistent (v4 vs v3). Since our convention
    > > > starts
    > > > > at
    > > > > >     > version
    > > > > >     > > > at 0, I
    > > > > >     > > >     >     think the new version would be 3.
    > > > > >     > > >     >
    > > > > >     > > >     >     4. "If the magic byte on message is 2, the
    > broker
    > > > > > should use
    > > > > >     > > the
    > > > > >     > > >     > tombstone
    > > > > >     > > >     >     bit for log compaction." What about null value?
    > > My
    > > > > >     > > understanding
    > > > > >     > > > is
    > > > > >     > > >     > that
    > > > > >     > > >     >     null value will be treated the same as setting
    > > the
    > > > > > tombstone
    > > > > >     > > bit.
    > > > > >     > > >     >
    > > > > >     > > >     >     5. For the migration path, it would be useful
    > to
    > > > > > describe the
    > > > > >     > > > down
    > > > > >     > > >     >     conversion path to consumers (i.e., brokers on
    > > > > message
    > > > > > format
    > > > > >     > > > 0.10.2
    > > > > >     > > >     > and
    > > > > >     > > >     >     consumers on older version).
    > > > > >     > > >     >
    > > > > >     > > >     >     Thanks,
    > > > > >     > > >     >
    > > > > >     > > >     >     Jun
    > > > > >     > > >     >
    > > > > >     > > >     >
    > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael
    > Pearce <
    > > > > >     > > > Michael.Pearce@ig.com
    > > > > >     > > >     > >
    > > > > >     > > >     >     wrote:
    > > > > >     > > >     >
    > > > > >     > > >     >     > Hi All,
    > > > > >     > > >     >     >
    > > > > >     > > >     >     > We have been discussing in the below thread
    > and
    > > > > final
    > > > > >     > changes
    > > > > >     > > > have
    > > > > >     > > >     > been
    > > > > >     > > >     >     > made to the KIP wiki based on these
    > > discussions.
    > > > > >     > > >     >     >
    > > > > >     > > >     >     > We would now like to put to the vote the
    > > > following
    > > > > > KIP:
    > > > > >     > > >     >     > https://cwiki.apache.org/
    > > > > > confluence/display/KAFKA/KIP-
    > > > > >     > 87+-+
    > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
    > > > > >     > > >     >     >
    > > > > >     > > >     >     > This kip is for having a distinct compaction
    > > > > > attribute
    > > > > >     > > > “tombstone”
    > > > > >     > > >     > flag
    > > > > >     > > >     >     > instead of relying on null value, allowing
    > > > non-null
    > > > > > value
    > > > > >     > > > delete
    > > > > >     > > >     > messages.
    > > > > >     > > >     >     >
    > > > > >     > > >     >     > Many thanks,
    > > > > >     > > >     >     > Michael
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >
    > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
    > > > > >     > > Michael.Pearce@ig.com>
    > > > > >     > > >     > wrote:
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >     Hi Mayuresh,
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >     LGTM. Ive just made one small adjustment
    > > > > > updating the
    > > > > >     > > wire
    > > > > >     > > >     > protocol to
    > > > > >     > > >     >     > show the magic byte bump.
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >     Do we think we’re good to put to a vote?
    > Is
    > > > > > there any
    > > > > >     > > > other bits
    > > > > >     > > >     >     > needing discussion?
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >     Cheers
    > > > > >     > > >     >     >     Mike
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
    > > > > >     > > >     > gharatmayuresh15@gmail.com>
    > > > > >     > > >     >     > wrote:
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >         Hi Michael,
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >         I have updated the migration section
    > of
    > > > the
    > > > > > KIP.
    > > > > >     > Can
    > > > > >     > > > you
    > > > > >     > > >     > please
    > > > > >     > > >     >     > take a look?
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >         Thanks,
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >         Mayuresh
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM,
    > > Mayuresh
    > > > > > Gharat <
    > > > > >     > > >     >     > gharatmayuresh15@gmail.com
    > > > > >     > > >     >     >         > wrote:
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >         > Hi Michael,
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         > That whilst sending tombstone and
    > non
    > > > > null
    > > > > > value,
    > > > > >     > > the
    > > > > >     > > >     > consumer
    > > > > >     > > >     >     > can expect
    > > > > >     > > >     >     >         > only to receive the non-null
    > message
    > > > only
    > > > > > in step
    > > > > >     > > > (3) is
    > > > > >     > > >     > this
    > > > > >     > > >     >     > correct?
    > > > > >     > > >     >     >         > ---> I do agree with you here.
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         > Becket, Ismael : can you guys
    > review
    > > > the
    > > > > >     > migration
    > > > > >     > > > plan
    > > > > >     > > >     > listed
    > > > > >     > > >     >     > above using
    > > > > >     > > >     >     >         > magic byte?
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         > Thanks,
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         > Mayuresh
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
    > > > Michael
    > > > > > Pearce <
    > > > > >     > > >     >     > Michael.Pearce@ig.com>
    > > > > >     > > >     >     >         > wrote:
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         >> Many thanks for this Mayuresh. I
    > > don't
    > > > > > have any
    > > > > >     > > >     > objections.
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> I assume we should state:
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> That whilst sending tombstone and
    > > non
    > > > > null
    > > > > >     > value,
    > > > > >     > > > the
    > > > > >     > > >     > consumer
    > > > > >     > > >     >     > can expect
    > > > > >     > > >     >     >         >> only to receive the non-null
    > message
    > > > > only
    > > > > > in
    > > > > >     > step
    > > > > >     > > > (3) is
    > > > > >     > > >     > this
    > > > > >     > > >     >     > correct?
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Cheers
    > > > > >     > > >     >     >         >> Mike
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Sent using OWA for iPhone
    > > > > >     > > >     >     >         >> ______________________________
    > > > > __________
    > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
    > > > > >     > gharatmayuresh15@gmail.com
    > > > > >     > > >
    > > > > >     > > >     >     >         >> Sent: Thursday, November 17, 2016
    > > > > 5:18:41
    > > > > > PM
    > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
    > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 -
    > Add
    > > > > > Compaction
    > > > > >     > > > Tombstone
    > > > > >     > > >     > Flag
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Hi Ismael,
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Thanks for the explanation.
    > > > > >     > > >     >     >         >> Specially I like this part where
    > in
    > > > you
    > > > > >     > mentioned
    > > > > >     > > > we can
    > > > > >     > > >     > get
    > > > > >     > > >     >     > rid of the
    > > > > >     > > >     >     >         >> older null value support for log
    > > > > > compaction
    > > > > >     > later
    > > > > >     > > > on,
    > > > > >     > > >     > here :
    > > > > >     > > >     >     >         >> We can't change semantics of the
    > > > message
    > > > > > format
    > > > > >     > > > without
    > > > > >     > > >     > having
    > > > > >     > > >     >     > a long
    > > > > >     > > >     >     >         >> transition period. And we can't
    > rely
    > > > > >     > > >     >     >         >> on people reading documentation or
    > > > > acting
    > > > > > on a
    > > > > >     > > > warning for
    > > > > >     > > >     >     > something so
    > > > > >     > > >     >     >         >> fundamental. As such, my take is
    > > that
    > > > we
    > > > > > need to
    > > > > >     > > > bump the
    > > > > >     > > >     > magic
    > > > > >     > > >     >     > byte. The
    > > > > >     > > >     >     >         >> good news is
    > > > > >     > > >     >     >         >> that we don't have to support all
    > > > > versions
    > > > > >     > > forever.
    > > > > >     > > > We
    > > > > >     > > >     > have
    > > > > >     > > >     >     > said that we
    > > > > >     > > >     >     >         >> will support direct upgrades for 2
    > > > > years.
    > > > > > That
    > > > > >     > > > means that
    > > > > >     > > >     >     > message format
    > > > > >     > > >     >     >         >> version n could, in theory, be
    > > > removed 2
    > > > > > years
    > > > > >     > > > after the
    > > > > >     > > >     > it's
    > > > > >     > > >     >     > introduced.
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Just a heads up, I would like to
    > > > mention
    > > > > > that
    > > > > >     > even
    > > > > >     > > > without
    > > > > >     > > >     >     > bumping magic
    > > > > >     > > >     >     >         >> byte, we will *NOT* loose zero
    > copy
    > > as
    > > > > in
    > > > > > the
    > > > > >     > > > client(x+1)
    > > > > >     > > >     > in my
    > > > > >     > > >     >     >         >> explanation
    > > > > >     > > >     >     >         >> above will convert internally a
    > null
    > > > > > value to
    > > > > >     > > have a
    > > > > >     > > >     > tombstone
    > > > > >     > > >     >     > bit set and
    > > > > >     > > >     >     >         >> a tombstone bit set to have a null
    > > > value
    > > > > >     > > > automatically
    > > > > >     > > >     >     > internally and by
    > > > > >     > > >     >     >         >> the time we move to version (x+2),
    > > the
    > > > > > clients
    > > > > >     > > > would have
    > > > > >     > > >     >     > upgraded.
    > > > > >     > > >     >     >         >> Obviously if we support a request
    > > from
    > > > > >     > > consumer(x),
    > > > > >     > > > we
    > > > > >     > > >     > will
    > > > > >     > > >     >     > loose zero
    > > > > >     > > >     >     >         >> copy
    > > > > >     > > >     >     >         >> but that is the same case with
    > magic
    > > > > byte.
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> But if magic byte bump makes life
    > > > easier
    > > > > > for
    > > > > >     > > > transition
    > > > > >     > > >     > for the
    > > > > >     > > >     >     > above
    > > > > >     > > >     >     >         >> reasons that you explained, I am
    > OK
    > > > with
    > > > > > it
    > > > > >     > since
    > > > > >     > > > we are
    > > > > >     > > >     > going
    > > > > >     > > >     >     > to meet the
    > > > > >     > > >     >     >         >> end goal down the road :)
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> On a side note can we update the
    > doc
    > > > > here
    > > > > > on
    > > > > >     > magic
    > > > > >     > > > byte
    > > > > >     > > >     > to say
    > > > > >     > > >     >     > that "*it
    > > > > >     > > >     >     >         >> should be bumped whenever the
    > > message
    > > > > > format is
    > > > > >     > > > changed
    > > > > >     > > >     > or the
    > > > > >     > > >     >     >         >> interpretation of message format
    > > > (usage
    > > > > > of the
    > > > > >     > > > reserved
    > > > > >     > > >     > bits as
    > > > > >     > > >     >     > well) is
    > > > > >     > > >     >     >         >> changed*".
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Hi Michael,
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Here is the update plan that we
    > > > > discussed
    > > > > >     > offline
    > > > > >     > > >     > yesterday :
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Currently the magic-byte which
    > > > > > corresponds to
    > > > > >     > the
    > > > > >     > > >     >     > "message.format.version"
    > > > > >     > > >     >     >         >> is set to 1.
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> 1) On broker it will be set to 1
    > > > > > initially.
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> 2) When a producer client sends a
    > > > > message
    > > > > > with
    > > > > >     > > > magic-byte
    > > > > >     > > >     > = 2,
    > > > > >     > > >     >     > since the
    > > > > >     > > >     >     >         >> broker is on magic-byte = 1, we
    > will
    > > > > down
    > > > > >     > convert
    > > > > >     > > > it,
    > > > > >     > > >     > which
    > > > > >     > > >     >     > means if the
    > > > > >     > > >     >     >         >> tombstone bit is set, the value
    > will
    > > > be
    > > > > > set to
    > > > > >     > > > null. A
    > > > > >     > > >     > consumer
    > > > > >     > > >     >     >         >> understanding magic-byte = 1, will
    > > > still
    > > > > > work
    > > > > >     > with
    > > > > >     > > > this. A
    > > > > >     > > >     >     > consumer
    > > > > >     > > >     >     >         >> working
    > > > > >     > > >     >     >         >> with magic-byte =2 will also be
    > able
    > > > to
    > > > > >     > understand
    > > > > >     > > > this,
    > > > > >     > > >     > since
    > > > > >     > > >     >     > it
    > > > > >     > > >     >     >         >> understands the tombstone.
    > > > > >     > > >     >     >         >> Now there is still the question of
    > > > > > supporting a
    > > > > >     > > >     > non-tombstone
    > > > > >     > > >     >     > and null
    > > > > >     > > >     >     >         >> value from producer client with
    > > > > > magic-byte = 2.*
    > > > > >     > > (I
    > > > > >     > > > am
    > > > > >     > > >     > not sure
    > > > > >     > > >     >     > if we
    > > > > >     > > >     >     >         >> should support this. Ismael/Becket
    > > can
    > > > > > comment
    > > > > >     > > > here)*
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> 3) When almost all the clients
    > have
    > > > > > upgraded,
    > > > > >     > the
    > > > > >     > > >     >     > message.format.version
    > > > > >     > > >     >     >         >> on
    > > > > >     > > >     >     >         >> the broker can be changed to 2,
    > > where
    > > > in
    > > > > > the
    > > > > >     > down
    > > > > >     > > >     > conversion in
    > > > > >     > > >     >     > the above
    > > > > >     > > >     >     >         >> step will not happen. If at this
    > > point
    > > > > we
    > > > > > get a
    > > > > >     > > > consumer
    > > > > >     > > >     >     > request from a
    > > > > >     > > >     >     >         >> older consumer, we might have to
    > > down
    > > > > > convert
    > > > > >     > > where
    > > > > >     > > > in we
    > > > > >     > > >     > loose
    > > > > >     > > >     >     > zero copy,
    > > > > >     > > >     >     >         >> but these cases should be rare.
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Becket can you review this plan
    > and
    > > > add
    > > > > > more
    > > > > >     > > > details if I
    > > > > >     > > >     > have
    > > > > >     > > >     >     >         >> missed/wronged something, before
    > we
    > > > put
    > > > > > it on
    > > > > >     > KIP.
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Thanks,
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> Mayuresh
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
    > > > > Michael
    > > > > >     > Pearce <
    > > > > >     > > >     >     > Michael.Pearce@ig.com>
    > > > > >     > > >     >     >         >> wrote:
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> > Thanks guys, for discussing this
    > > > > > offline and
    > > > > >     > > > getting
    > > > > >     > > >     > some
    > > > > >     > > >     >     > consensus.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > So its clear for myself and
    > others
    > > > > what
    > > > > > is
    > > > > >     > > > proposed now
    > > > > >     > > >     > (i
    > > > > >     > > >     >     > think i
    > > > > >     > > >     >     >         >> > understand, but want to make
    > sure)
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > Could i ask either directly
    > update
    > > > the
    > > > > > kip to
    > > > > >     > > > detail the
    > > > > >     > > >     >     > migration
    > > > > >     > > >     >     >         >> > strategy, or (re-)state your
    > > offline
    > > > > > discussed
    > > > > >     > > and
    > > > > >     > > >     > agreed
    > > > > >     > > >     >     > migration
    > > > > >     > > >     >     >         >> > strategy based on a magic byte
    > is
    > > in
    > > > > > this
    > > > > >     > > thread.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > The main original driver for the
    > > KIP
    > > > > > was to
    > > > > >     > > > support
    > > > > >     > > >     >     > compaction where
    > > > > >     > > >     >     >         >> value
    > > > > >     > > >     >     >         >> > isn't null, based off the
    > > > discussions
    > > > > on
    > > > > >     > KIP-82
    > > > > >     > > > thread.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > We should be able to support
    > > > > > non-tombstone +
    > > > > >     > > null
    > > > > >     > > > value
    > > > > >     > > >     > by the
    > > > > >     > > >     >     >         >> completion
    > > > > >     > > >     >     >         >> > of the KIP, as we noted when
    > > > > discussing
    > > > > > this
    > > > > >     > > kip,
    > > > > >     > > > having
    > > > > >     > > >     >     > logic based on
    > > > > >     > > >     >     >         >> a
    > > > > >     > > >     >     >         >> > null value isn't very clean and
    > > also
    > > > > > separates
    > > > > >     > > the
    > > > > >     > > >     > concerns.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > As discussed already though we
    > can
    > > > > > split this
    > > > > >     > > into
    > > > > >     > > >     > KIP-87a
    > > > > >     > > >     >     > and KIP-87b
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > Where we look to deliver KIP-87a
    > > on
    > > > a
    > > > > >     > compacted
    > > > > >     > > > topic
    > > > > >     > > >     > (to
    > > > > >     > > >     >     > address the
    > > > > >     > > >     >     >         >> > immediate issues)
    > > > > >     > > >     >     >         >> > * tombstone + null value
    > > > > >     > > >     >     >         >> > * tombstone + non-null value
    > > > > >     > > >     >     >         >> > * non-tombstone + non-null value
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > Then we can discuss once KIP-87a
    > > is
    > > > > > completed
    > > > > >     > > > options
    > > > > >     > > >     > later
    > > > > >     > > >     >     > and how we
    > > > > >     > > >     >     >         >> > support the second part KIP-87b
    > to
    > > > > > deliver:
    > > > > >     > > >     >     >         >> > * non-tombstone + null value
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > Cheers
    > > > > >     > > >     >     >         >> > Mike
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > ______________________________
    > > > > > __________
    > > > > >     > > >     >     >         >> > From: Becket Qin <
    > > > > becket.qin@gmail.com>
    > > > > >     > > >     >     >         >> > Sent: Thursday, November 17,
    > 2016
    > > > 1:43
    > > > > > AM
    > > > > >     > > >     >     >         >> > To: dev@kafka.apache.org
    > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 -
    > > Add
    > > > > > Compaction
    > > > > >     > > >     > Tombstone Flag
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had an
    > > offline
    > > > > >     > discussion,
    > > > > >     > > > and
    > > > > >     > > >     > following
    > > > > >     > > >     >     > is a brief
    > > > > >     > > >     >     >         >> > summary.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > 1. We agreed that not bumping up
    > > > magic
    > > > > > value
    > > > > >     > may
    > > > > >     > > > result
    > > > > >     > > >     > in
    > > > > >     > > >     >     > losing zero
    > > > > >     > > >     >     >         >> copy
    > > > > >     > > >     >     >         >> > during migration.
    > > > > >     > > >     >     >         >> > 2. Given that bumping up magic
    > > value
    > > > > is
    > > > > > almost
    > > > > >     > > > free and
    > > > > >     > > >     > has
    > > > > >     > > >     >     > benefit of
    > > > > >     > > >     >     >         >> > avoiding potential performance
    > > > issue.
    > > > > > It is
    > > > > >     > > > probably
    > > > > >     > > >     > worth
    > > > > >     > > >     >     > doing.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > One issue we still need to think
    > > > about
    > > > > > is
    > > > > >     > > whether
    > > > > >     > > > we
    > > > > >     > > >     > want to
    > > > > >     > > >     >     > support a
    > > > > >     > > >     >     >         >> > non-tombstone message with null
    > > > value.
    > > > > >     > > >     >     >         >> > Currently it is not supported by
    > > > > Kafka.
    > > > > > If we
    > > > > >     > > > allow a
    > > > > >     > > >     >     > non-tombstone null
    > > > > >     > > >     >     >         >> > value message to exist after
    > > KIP-87.
    > > > > The
    > > > > >     > problem
    > > > > >     > > > is
    > > > > >     > > >     > that such
    > > > > >     > > >     >     > message
    > > > > >     > > >     >     >         >> will
    > > > > >     > > >     >     >         >> > not be supported by the
    > consumers
    > > > > prior
    > > > > > to
    > > > > >     > > KIP-87.
    > > > > >     > > >     > Because a
    > > > > >     > > >     >     > null value
    > > > > >     > > >     >     >         >> > will always be interpreted to a
    > > > > > tombstone.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > One option is that we keep the
    > > > current
    > > > > > way,
    > > > > >     > i.e.
    > > > > >     > > > do not
    > > > > >     > > >     >     > support such
    > > > > >     > > >     >     >         >> > message. It would be good to
    > know
    > > if
    > > > > > there is
    > > > > >     > a
    > > > > >     > > >     > concrete use
    > > > > >     > > >     >     > case for
    > > > > >     > > >     >     >         >> such
    > > > > >     > > >     >     >         >> > message. If there is not, we can
    > > > > > probably just
    > > > > >     > > not
    > > > > >     > > >     > support it.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > Thanks,
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
    > > > > > Mayuresh
    > > > > >     > > Gharat <
    > > > > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
    > > > > >     > > >     >     >         >> > > wrote:
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >> > > Hi Ismael,
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > > This is something I can think
    > of
    > > > for
    > > > > >     > migration
    > > > > >     > > > plan:
    > > > > >     > > >     >     >         >> > > So the migration plan can look
    > > > > > something
    > > > > >     > like
    > > > > >     > > > this,
    > > > > >     > > >     > with up
    > > > > >     > > >     >     >         >> conversion :
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > > 1) Currently lets say we have
    > > > Broker
    > > > > > at
    > > > > >     > > version
    > > > > >     > > > x.
    > > > > >     > > >     >     >         >> > > 2) Currently we have clients
    > at
    > > > > > version x.
    > > > > >     > > >     >     >         >> > > 3) a) We move the version to
    > > > > > Broker(x+1) :
    > > > > >     > > > supports
    > > > > >     > > >     > both
    > > > > >     > > >     >     > tombstone and
    > > > > >     > > >     >     >         >> > null
    > > > > >     > > >     >     >         >> > > for log compaction.
    > > > > >     > > >     >     >         >> > >     b) We upgrade the client
    > to
    > > > > > version
    > > > > >     > > > client(x+1) :
    > > > > >     > > >     > if in
    > > > > >     > > >     >     > the
    > > > > >     > > >     >     >         >> producer
    > > > > >     > > >     >     >         >> > > client(x+1) the value is set
    > to
    > > > > null,
    > > > > > we
    > > > > >     > will
    > > > > >     > > >     > automatically
    > > > > >     > > >     >     > set the
    > > > > >     > > >     >     >         >> > > Tombstone bit internally. If
    > the
    > > > > > producer
    > > > > >     > > > client(x+1)
    > > > > >     > > >     > sets
    > > > > >     > > >     >     > the
    > > > > >     > > >     >     >         >> tombstone
    > > > > >     > > >     >     >         >> > > itself, well and good. For
    > > > producer
    > > > > >     > client(x),
    > > > > >     > > > the
    > > > > >     > > >     > broker
    > > > > >     > > >     >     > will up
    > > > > >     > > >     >     >         >> convert
    > > > > >     > > >     >     >         >> > > to have the tombstone bit.
    > > > > > Broker(x+1) is
    > > > > >     > > > supporting
    > > > > >     > > >     > both.
    > > > > >     > > >     >     > Consumer
    > > > > >     > > >     >     >         >> > > client(x+1) will be aware of
    > > this
    > > > > and
    > > > > > should
    > > > > >     > > be
    > > > > >     > > > able
    > > > > >     > > >     > to
    > > > > >     > > >     >     > handle this.
    > > > > >     > > >     >     >         >> For
    > > > > >     > > >     >     >         >> > > consumer client(x) we will
    > down
    > > > > > convert the
    > > > > >     > > > message
    > > > > >     > > >     > on the
    > > > > >     > > >     >     > broker
    > > > > >     > > >     >     >         >> side.
    > > > > >     > > >     >     >         >> > >     c) At this point we will
    > > have
    > > > to
    > > > > >     > specify a
    > > > > >     > > >     > warning or
    > > > > >     > > >     >     > clearly
    > > > > >     > > >     >     >         >> specify
    > > > > >     > > >     >     >         >> > > in docs that this behavior is
    > > > about
    > > > > > to be
    > > > > >     > > > changed for
    > > > > >     > > >     > log
    > > > > >     > > >     >     > compaction.
    > > > > >     > > >     >     >         >> > > 4) a) In next release of the
    > > > > > Broker(x+2), we
    > > > > >     > > > say that
    > > > > >     > > >     > only
    > > > > >     > > >     >     > Tombstone
    > > > > >     > > >     >     >         >> is
    > > > > >     > > >     >     >         >> > > used for log compaction on the
    > > > > Broker
    > > > > > side.
    > > > > >     > > >     > Clients(x+1)
    > > > > >     > > >     >     > still is
    > > > > >     > > >     >     >         >> > > supported.
    > > > > >     > > >     >     >         >> > >     b) We upgrade the client
    > to
    > > > > > version
    > > > > >     > > > client(x+2) :
    > > > > >     > > >     > if
    > > > > >     > > >     >     > value is set
    > > > > >     > > >     >     >         >> to
    > > > > >     > > >     >     >         >> > > null, tombstone will not be
    > set
    > > > > >     > automatically.
    > > > > >     > > > The
    > > > > >     > > >     > client
    > > > > >     > > >     >     > will have to
    > > > > >     > > >     >     >         >> > call
    > > > > >     > > >     >     >         >> > > setTombstone() to actually set
    > > the
    > > > > >     > tombstone.
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > > We should compare this
    > migration
    > > > > plan
    > > > > > with
    > > > > >     > the
    > > > > >     > > >     > migration
    > > > > >     > > >     >     > plan for
    > > > > >     > > >     >     >         >> magic
    > > > > >     > > >     >     >         >> > > byte bump and do whatever
    > looks
    > > > > good.
    > > > > >     > > >     >     >         >> > > I am just worried that if we
    > go
    > > > down
    > > > > > magic
    > > > > >     > > byte
    > > > > >     > > > route,
    > > > > >     > > >     >     > unless I am
    > > > > >     > > >     >     >         >> > missing
    > > > > >     > > >     >     >         >> > > something, it sounds like
    > kafka
    > > > will
    > > > > > be
    > > > > >     > stuck
    > > > > >     > > > with
    > > > > >     > > >     >     > supporting both
    > > > > >     > > >     >     >         >> null
    > > > > >     > > >     >     >         >> > > value and tombstone bit for
    > log
    > > > > > compaction
    > > > > >     > for
    > > > > >     > > > life
    > > > > >     > > >     > long,
    > > > > >     > > >     >     > which does
    > > > > >     > > >     >     >         >> not
    > > > > >     > > >     >     >         >> > > look like a good end state.
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > > Thanks,
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > > Mayuresh
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32
    > AM,
    > > > > > Mayuresh
    > > > > >     > > > Gharat <
    > > > > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
    > > > > >     > > >     >     >         >> > > > wrote:
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > > > Hi Ismael,
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > That's a very good point
    > > which I
    > > > > > might
    > > > > >     > have
    > > > > >     > > > not
    > > > > >     > > >     >     > considered earlier.
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > Here is a plan that I can
    > > think
    > > > > of:
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > Stage 1) The broker from now
    > > on,
    > > > > up
    > > > > >     > converts
    > > > > >     > > > the
    > > > > >     > > >     > message
    > > > > >     > > >     >     > to have the
    > > > > >     > > >     >     >         >> > > > tombstone marker. The log
    > > > > compaction
    > > > > >     > thread
    > > > > >     > > > does log
    > > > > >     > > >     >     > compaction
    > > > > >     > > >     >     >         >> based
    > > > > >     > > >     >     >         >> > on
    > > > > >     > > >     >     >         >> > > > both null and tombstone
    > > marker.
    > > > > > This is
    > > > > >     > our
    > > > > >     > > >     > transition
    > > > > >     > > >     >     > period.
    > > > > >     > > >     >     >         >> > > > Stage 2) The next release we
    > > > only
    > > > > > say that
    > > > > >     > > log
    > > > > >     > > >     > compaction
    > > > > >     > > >     >     > is based
    > > > > >     > > >     >     >         >> on
    > > > > >     > > >     >     >         >> > > > tombstone marker. (Open
    > source
    > > > > > kafka makes
    > > > > >     > > > this as a
    > > > > >     > > >     >     > policy). By
    > > > > >     > > >     >     >         >> this
    > > > > >     > > >     >     >         >> > > time,
    > > > > >     > > >     >     >         >> > > > the organization which is
    > > moving
    > > > > to
    > > > > > this
    > > > > >     > > > release
    > > > > >     > > >     > will be
    > > > > >     > > >     >     > sure that
    > > > > >     > > >     >     >         >> they
    > > > > >     > > >     >     >         >> > > > have gone through the entire
    > > > > > transition
    > > > > >     > > > period.
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > My only goal of doing this
    > is
    > > > that
    > > > > > Kafka
    > > > > >     > > > clearly
    > > > > >     > > >     >     > specifies the end
    > > > > >     > > >     >     >         >> > state
    > > > > >     > > >     >     >         >> > > > about what log compaction
    > > means
    > > > > (is
    > > > > > it
    > > > > >     > null
    > > > > >     > > > value
    > > > > >     > > >     > or a
    > > > > >     > > >     >     > tombstone
    > > > > >     > > >     >     >         >> > marker,
    > > > > >     > > >     >     >         >> > > > but not both).
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > What do you think?
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > Thanks,
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > Mayuresh
    > > > > >     > > >     >     >         >> > > > .
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17
    > > AM,
    > > > > > Ismael
    > > > > >     > > Juma <
    > > > > >     > > >     >     > ismael@juma.me.uk>
    > > > > >     > > >     >     >         >> > wrote:
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > >> One comment below.
    > > > > >     > > >     >     >         >> > > >>
    > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at
    > 5:08
    > > > PM,
    > > > > > Mayuresh
    > > > > >     > > > Gharat <
    > > > > >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
    > > > > >     > > >     >     >         >> > > >> > wrote:
    > > > > >     > > >     >     >         >> > > >>
    > > > > >     > > >     >     >         >> > > >> >    - If we don't bump up
    > > the
    > > > > > magic
    > > > > >     > byte,
    > > > > >     > > > on the
    > > > > >     > > >     > broker
    > > > > >     > > >     >     > side, the
    > > > > >     > > >     >     >         >> > > broker
    > > > > >     > > >     >     >         >> > > >> >    will always have to
    > look
    > > > at
    > > > > > both
    > > > > >     > > > tombstone
    > > > > >     > > >     > bit and
    > > > > >     > > >     >     > the value
    > > > > >     > > >     >     >         >> when
    > > > > >     > > >     >     >         >> > > do
    > > > > >     > > >     >     >         >> > > >> the
    > > > > >     > > >     >     >         >> > > >> >    compaction. Assuming
    > we
    > > do
    > > > > > not bump
    > > > > >     > up
    > > > > >     > > > the
    > > > > >     > > >     > magic
    > > > > >     > > >     >     > byte,
    > > > > >     > > >     >     >         >> > > >> >    imagine the broker
    > sees
    > > a
    > > > > > message
    > > > > >     > > which
    > > > > >     > > > does
    > > > > >     > > >     > not
    > > > > >     > > >     >     > have a
    > > > > >     > > >     >     >         >> tombstone
    > > > > >     > > >     >     >         >> > > bit
    > > > > >     > > >     >     >         >> > > >> >    set. The broker does
    > not
    > > > > know
    > > > > > when
    > > > > >     > the
    > > > > >     > > >     > message was
    > > > > >     > > >     >     > produced
    > > > > >     > > >     >     >         >> (i.e.
    > > > > >     > > >     >     >         >> > > >> > whether
    > > > > >     > > >     >     >         >> > > >> >    the message has been
    > up
    > > > > > converted or
    > > > > >     > > > not), it
    > > > > >     > > >     > has
    > > > > >     > > >     >     > to take a
    > > > > >     > > >     >     >         >> > further
    > > > > >     > > >     >     >         >> > > >> > look at
    > > > > >     > > >     >     >         >> > > >> >    the value to see if it
    > > is
    > > > > > null or
    > > > > >     > not
    > > > > >     > > in
    > > > > >     > > >     > order to
    > > > > >     > > >     >     > determine
    > > > > >     > > >     >     >         >> if it
    > > > > >     > > >     >     >         >> > > is
    > > > > >     > > >     >     >         >> > > >> a
    > > > > >     > > >     >     >         >> > > >> >    tombstone. The same
    > > logic
    > > > > has
    > > > > > to be
    > > > > >     > > put
    > > > > >     > > > on the
    > > > > >     > > >     >     > consumer as
    > > > > >     > > >     >     >         >> well
    > > > > >     > > >     >     >         >> > > >> because
    > > > > >     > > >     >     >         >> > > >> > the
    > > > > >     > > >     >     >         >> > > >> >    consumer does not know
    > > if
    > > > > the
    > > > > >     > message
    > > > > >     > > > has
    > > > > >     > > >     > been up
    > > > > >     > > >     >     > converted or
    > > > > >     > > >     >     >         >> > not.
    > > > > >     > > >     >     >         >> > > >> >       - If we upconvert
    > > while
    > > > > >     > appending,
    > > > > >     > > > this is
    > > > > >     > > >     > not
    > > > > >     > > >     >     > the case,
    > > > > >     > > >     >     >         >> > right?
    > > > > >     > > >     >     >         >> > > >>
    > > > > >     > > >     >     >         >> > > >>
    > > > > >     > > >     >     >         >> > > >> If I understand you
    > > correctly,
    > > > > > this is
    > > > > >     > not
    > > > > >     > > >     > sufficient
    > > > > >     > > >     >     > because the
    > > > > >     > > >     >     >         >> log
    > > > > >     > > >     >     >         >> > > may
    > > > > >     > > >     >     >         >> > > >> have messages appended
    > before
    > > > it
    > > > > > was
    > > > > >     > > > upgraded to
    > > > > >     > > >     > include
    > > > > >     > > >     >     > KIP-87.
    > > > > >     > > >     >     >         >> > > >>
    > > > > >     > > >     >     >         >> > > >> Ismael
    > > > > >     > > >     >     >         >> > > >>
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > > > --
    > > > > >     > > >     >     >         >> > > > -Regards,
    > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
    > > > > >     > > >     >     >         >> > > > (862) 250-7125
    > > > > >     > > >     >     >         >> > > >
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > > --
    > > > > >     > > >     >     >         >> > > -Regards,
    > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
    > > > > >     > > >     >     >         >> > > (862) 250-7125
    > > > > >     > > >     >     >         >> > >
    > > > > >     > > >     >     >         >> > The information contained in
    > this
    > > > > email
    > > > > > is
    > > > > >     > > > strictly
    > > > > >     > > >     >     > confidential and for
    > > > > >     > > >     >     >         >> > the use of the addressee only,
    > > > unless
    > > > > >     > otherwise
    > > > > >     > > >     > indicated. If
    > > > > >     > > >     >     > you are
    > > > > >     > > >     >     >         >> not
    > > > > >     > > >     >     >         >> > the intended recipient, please
    > do
    > > > not
    > > > > > read,
    > > > > >     > > copy,
    > > > > >     > > > use or
    > > > > >     > > >     >     > disclose to
    > > > > >     > > >     >     >         >> others
    > > > > >     > > >     >     >         >> > this message or any attachment.
    > > > Please
    > > > > > also
    > > > > >     > > > notify the
    > > > > >     > > >     > sender
    > > > > >     > > >     >     > by
    > > > > >     > > >     >     >         >> replying
    > > > > >     > > >     >     >         >> > to this email or by telephone
    > > > (+44(020
    > > > > > 7896
    > > > > >     > > 0011)
    > > > > >     > > > and
    > > > > >     > > >     > then
    > > > > >     > > >     >     > delete the
    > > > > >     > > >     >     >         >> email
    > > > > >     > > >     >     >         >> > and any copies of it. Opinions,
    > > > > > conclusion
    > > > > >     > (etc)
    > > > > >     > > > that
    > > > > >     > > >     > do not
    > > > > >     > > >     >     > relate to
    > > > > >     > > >     >     >         >> the
    > > > > >     > > >     >     >         >> > official business of this
    > company
    > > > > shall
    > > > > > be
    > > > > >     > > > understood as
    > > > > >     > > >     >     > neither given
    > > > > >     > > >     >     >         >> nor
    > > > > >     > > >     >     >         >> > endorsed by it. IG is a trading
    > > name
    > > > > of
    > > > > > IG
    > > > > >     > > Markets
    > > > > >     > > >     > Limited (a
    > > > > >     > > >     >     > company
    > > > > >     > > >     >     >         >> > registered in England and Wales,
    > > > > company
    > > > > >     > number
    > > > > >     > > >     > 04008957) and
    > > > > >     > > >     >     > IG Index
    > > > > >     > > >     >     >         >> > Limited (a company registered in
    > > > > > England and
    > > > > >     > > > Wales,
    > > > > >     > > >     > company
    > > > > >     > > >     >     > number
    > > > > >     > > >     >     >         >> > 01190902). Registered address at
    > > > > Cannon
    > > > > > Bridge
    > > > > >     > > > House, 25
    > > > > >     > > >     >     > Dowgate Hill,
    > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
    > > > > Limited
    > > > > >     > > (register
    > > > > >     > > >     > number
    > > > > >     > > >     >     > 195355) and IG
    > > > > >     > > >     >     >         >> > Index Limited (register number
    > > > 114059)
    > > > > > are
    > > > > >     > > > authorised
    > > > > >     > > >     > and
    > > > > >     > > >     >     > regulated by
    > > > > >     > > >     >     >         >> the
    > > > > >     > > >     >     >         >> > Financial Conduct Authority.
    > > > > >     > > >     >     >         >> >
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >> --
    > > > > >     > > >     >     >         >> -Regards,
    > > > > >     > > >     >     >         >> Mayuresh R. Gharat
    > > > > >     > > >     >     >         >> (862) 250-7125
    > > > > >     > > >     >     >         >> The information contained in this
    > > > email
    > > > > is
    > > > > >     > > strictly
    > > > > >     > > >     >     > confidential and for
    > > > > >     > > >     >     >         >> the use of the addressee only,
    > > unless
    > > > > > otherwise
    > > > > >     > > >     > indicated. If
    > > > > >     > > >     >     > you are not
    > > > > >     > > >     >     >         >> the intended recipient, please do
    > > not
    > > > > > read,
    > > > > >     > copy,
    > > > > >     > > > use or
    > > > > >     > > >     >     > disclose to others
    > > > > >     > > >     >     >         >> this message or any attachment.
    > > Please
    > > > > > also
    > > > > >     > notify
    > > > > >     > > > the
    > > > > >     > > >     > sender
    > > > > >     > > >     >     > by replying
    > > > > >     > > >     >     >         >> to this email or by telephone
    > > (+44(020
    > > > > > 7896
    > > > > >     > 0011)
    > > > > >     > > > and then
    > > > > >     > > >     >     > delete the email
    > > > > >     > > >     >     >         >> and any copies of it. Opinions,
    > > > > > conclusion (etc)
    > > > > >     > > > that do
    > > > > >     > > >     > not
    > > > > >     > > >     >     > relate to the
    > > > > >     > > >     >     >         >> official business of this company
    > > > shall
    > > > > be
    > > > > >     > > > understood as
    > > > > >     > > >     >     > neither given nor
    > > > > >     > > >     >     >         >> endorsed by it. IG is a trading
    > name
    > > > of
    > > > > IG
    > > > > >     > Markets
    > > > > >     > > >     > Limited (a
    > > > > >     > > >     >     > company
    > > > > >     > > >     >     >         >> registered in England and Wales,
    > > > company
    > > > > > number
    > > > > >     > > > 04008957)
    > > > > >     > > >     > and
    > > > > >     > > >     >     > IG Index
    > > > > >     > > >     >     >         >> Limited (a company registered in
    > > > England
    > > > > > and
    > > > > >     > > Wales,
    > > > > >     > > >     > company
    > > > > >     > > >     >     > number
    > > > > >     > > >     >     >         >> 01190902). Registered address at
    > > > Cannon
    > > > > > Bridge
    > > > > >     > > > House, 25
    > > > > >     > > >     >     > Dowgate Hill,
    > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
    > > > Limited
    > > > > >     > (register
    > > > > >     > > > number
    > > > > >     > > >     >     > 195355) and IG
    > > > > >     > > >     >     >         >> Index Limited (register number
    > > 114059)
    > > > > are
    > > > > >     > > > authorised and
    > > > > >     > > >     >     > regulated by the
    > > > > >     > > >     >     >         >> Financial Conduct Authority.
    > > > > >     > > >     >     >         >>
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >         > --
    > > > > >     > > >     >     >         > -Regards,
    > > > > >     > > >     >     >         > Mayuresh R. Gharat
    > > > > >     > > >     >     >         > (862) 250-7125
    > > > > >     > > >     >     >         >
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >         --
    > > > > >     > > >     >     >         -Regards,
    > > > > >     > > >     >     >         Mayuresh R. Gharat
    > > > > >     > > >     >     >         (862) 250-7125
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >     The information contained in this email
    > is
    > > > > > strictly
    > > > > >     > > > confidential
    > > > > >     > > >     > and
    > > > > >     > > >     >     > for the use of the addressee only, unless
    > > > otherwise
    > > > > >     > > indicated.
    > > > > >     > > > If
    > > > > >     > > >     > you are
    > > > > >     > > >     >     > not the intended recipient, please do not
    > read,
    > > > > > copy, use
    > > > > >     > or
    > > > > >     > > >     > disclose to
    > > > > >     > > >     >     > others this message or any attachment. Please
    > > > also
    > > > > > notify
    > > > > >     > the
    > > > > >     > > > sender
    > > > > >     > > >     > by
    > > > > >     > > >     >     > replying to this email or by telephone
    > (+44(020
    > > > > 7896
    > > > > > 0011)
    > > > > >     > > and
    > > > > >     > > > then
    > > > > >     > > >     > delete
    > > > > >     > > >     >     > the email and any copies of it. Opinions,
    > > > > conclusion
    > > > > > (etc)
    > > > > >     > > > that do
    > > > > >     > > >     > not
    > > > > >     > > >     >     > relate to the official business of this
    > company
    > > > > > shall be
    > > > > >     > > > understood
    > > > > >     > > >     > as
    > > > > >     > > >     >     > neither given nor endorsed by it. IG is a
    > > trading
    > > > > > name of
    > > > > >     > IG
    > > > > >     > > > Markets
    > > > > >     > > >     >     > Limited (a company registered in England and
    > > > Wales,
    > > > > > company
    > > > > >     > > > number
    > > > > >     > > >     >     > 04008957) and IG Index Limited (a company
    > > > > registered
    > > > > > in
    > > > > >     > > > England and
    > > > > >     > > >     > Wales,
    > > > > >     > > >     >     > company number 01190902). Registered address
    > at
    > > > > > Cannon
    > > > > >     > Bridge
    > > > > >     > > > House,
    > > > > >     > > >     > 25
    > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG
    > Markets
    > > > > > Limited
    > > > > >     > > > (register
    > > > > >     > > >     > number
    > > > > >     > > >     >     > 195355) and IG Index Limited (register number
    > > > > > 114059) are
    > > > > >     > > > authorised
    > > > > >     > > >     > and
    > > > > >     > > >     >     > regulated by the Financial Conduct Authority.
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >
    > > > > >     > > >     >     >
    > > > > >     > > >     >
    > > > > >     > > >     >
    > > > > >     > > >     > The information contained in this email is strictly
    > > > > > confidential
    > > > > >     > > and
    > > > > >     > > > for
    > > > > >     > > >     > the use of the addressee only, unless otherwise
    > > > > indicated.
    > > > > > If you
    > > > > >     > > > are not
    > > > > >     > > >     > the intended recipient, please do not read, copy,
    > use
    > > > or
    > > > > > disclose
    > > > > >     > > to
    > > > > >     > > > others
    > > > > >     > > >     > this message or any attachment. Please also notify
    > > the
    > > > > > sender by
    > > > > >     > > > replying
    > > > > >     > > >     > to this email or by telephone (+44(020 7896 0011
    > <020%207896%200011>) and
    > > > > then
    > > > > > delete
    > > > > >     > > > the email
    > > > > >     > > >     > and any copies of it. Opinions, conclusion (etc)
    > that
    > > > do
    > > > > > not
    > > > > >     > relate
    > > > > >     > > > to the
    > > > > >     > > >     > official business of this company shall be
    > understood
    > > > as
    > > > > > neither
    > > > > >     > > > given nor
    > > > > >     > > >     > endorsed by it. IG is a trading name of IG Markets
    > > > > Limited
    > > > > > (a
    > > > > >     > > company
    > > > > >     > > >     > registered in England and Wales, company number
    > > > 04008957)
    > > > > > and IG
    > > > > >     > > > Index
    > > > > >     > > >     > Limited (a company registered in England and Wales,
    > > > > company
    > > > > >     > number
    > > > > >     > > >     > 01190902). Registered address at Cannon Bridge
    > House,
    > > > 25
    > > > > > Dowgate
    > > > > >     > > > Hill,
    > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited (register
    > > > number
    > > > > > 195355)
    > > > > >     > > > and IG
    > > > > >     > > >     > Index Limited (register number 114059) are
    > authorised
    > > > and
    > > > > >     > regulated
    > > > > >     > > > by the
    > > > > >     > > >     > Financial Conduct Authority.
    > > > > >     > > >     >
    > > > > >     > > >
    > > > > >     > > >
    > > > > >     > > > The information contained in this email is strictly
    > > > > confidential
    > > > > > and
    > > > > >     > for
    > > > > >     > > > the use of the addressee only, unless otherwise
    > indicated.
    > > If
    > > > > > you are
    > > > > >     > not
    > > > > >     > > > the intended recipient, please do not read, copy, use or
    > > > > > disclose to
    > > > > >     > > others
    > > > > >     > > > this message or any attachment. Please also notify the
    > > sender
    > > > > by
    > > > > >     > replying
    > > > > >     > > > to this email or by telephone (+44(020 7896 0011
    > <020%207896%200011>) and then
    > > > > > delete the
    > > > > >     > > email
    > > > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
    > > not
    > > > > > relate to
    > > > > >     > > the
    > > > > >     > > > official business of this company shall be understood as
    > > > > neither
    > > > > > given
    > > > > >     > > nor
    > > > > >     > > > endorsed by it. IG is a trading name of IG Markets
    > Limited
    > > (a
    > > > > > company
    > > > > >     > > > registered in England and Wales, company number 04008957)
    > > and
    > > > > IG
    > > > > > Index
    > > > > >     > > > Limited (a company registered in England and Wales,
    > company
    > > > > > number
    > > > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
    > > > > Dowgate
    > > > > > Hill,
    > > > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
    > > > > > 195355) and
    > > > > >     > IG
    > > > > >     > > > Index Limited (register number 114059) are authorised and
    > > > > > regulated by
    > > > > >     > > the
    > > > > >     > > > Financial Conduct Authority.
    > > > > >     > > >
    > > > > >     > > The information contained in this email is strictly
    > > > confidential
    > > > > > and for
    > > > > >     > > the use of the addressee only, unless otherwise indicated.
    > If
    > > > you
    > > > > > are not
    > > > > >     > > the intended recipient, please do not read, copy, use or
    > > > disclose
    > > > > > to
    > > > > >     > others
    > > > > >     > > this message or any attachment. Please also notify the
    > sender
    > > > by
    > > > > > replying
    > > > > >     > > to this email or by telephone (+44(020 7896 0011
    > <020%207896%200011>) and then
    > > > delete
    > > > > > the
    > > > > >     > email
    > > > > >     > > and any copies of it. Opinions, conclusion (etc) that do
    > not
    > > > > > relate to
    > > > > >     > the
    > > > > >     > > official business of this company shall be understood as
    > > > neither
    > > > > > given
    > > > > >     > nor
    > > > > >     > > endorsed by it. IG is a trading name of IG Markets Limited
    > (a
    > > > > > company
    > > > > >     > > registered in England and Wales, company number 04008957)
    > and
    > > > IG
    > > > > > Index
    > > > > >     > > Limited (a company registered in England and Wales, company
    > > > > number
    > > > > >     > > 01190902). Registered address at Cannon Bridge House, 25
    > > > Dowgate
    > > > > > Hill,
    > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
    > > > 195355)
    > > > > > and IG
    > > > > >     > > Index Limited (register number 114059) are authorised and
    > > > > > regulated by
    > > > > >     > the
    > > > > >     > > Financial Conduct Authority.
    > > > > >     > >
    > > > > >     > >
    > > > > >     > The information contained in this email is strictly
    > > confidential
    > > > > and
    > > > > > for
    > > > > >     > the use of the addressee only, unless otherwise indicated. If
    > > you
    > > > > > are not
    > > > > >     > the intended recipient, please do not read, copy, use or
    > > disclose
    > > > > to
    > > > > > others
    > > > > >     > this message or any attachment. Please also notify the sender
    > > by
    > > > > > replying
    > > > > >     > to this email or by telephone (+44(020 7896 0011
    > <020%207896%200011>) and then
    > > delete
    > > > > > the email
    > > > > >     > and any copies of it. Opinions, conclusion (etc) that do not
    > > > relate
    > > > > > to the
    > > > > >     > official business of this company shall be understood as
    > > neither
    > > > > > given nor
    > > > > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
    > > > > company
    > > > > >     > registered in England and Wales, company number 04008957) and
    > > IG
    > > > > > Index
    > > > > >     > Limited (a company registered in England and Wales, company
    > > > number
    > > > > >     > 01190902). Registered address at Cannon Bridge House, 25
    > > Dowgate
    > > > > > Hill,
    > > > > >     > London EC4R 2YA. Both IG Markets Limited (register number
    > > 195355)
    > > > > > and IG
    > > > > >     > Index Limited (register number 114059) are authorised and
    > > > regulated
    > > > > > by the
    > > > > >     > Financial Conduct Authority.
    > > > > >     >
    > > > > >     >
    > > > > >
    > > > > >
    > > > > > The information contained in this email is strictly confidential
    > and
    > > > for
    > > > > > the use of the addressee only, unless otherwise indicated. If you
    > are
    > > > not
    > > > > > the intended recipient, please do not read, copy, use or disclose
    > to
    > > > > others
    > > > > > this message or any attachment. Please also notify the sender by
    > > > replying
    > > > > > to this email or by telephone (+44(020 7896 0011
    > <020%207896%200011>) and then delete the
    > > > > email
    > > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
    > > to
    > > > > the
    > > > > > official business of this company shall be understood as neither
    > > given
    > > > > nor
    > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
    > company
    > > > > > registered in England and Wales, company number 04008957) and IG
    > > Index
    > > > > > Limited (a company registered in England and Wales, company number
    > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > > Hill,
    > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
    > and
    > > > IG
    > > > > > Index Limited (register number 114059) are authorised and regulated
    > > by
    > > > > the
    > > > > > Financial Conduct Authority.
    > > > > >
    > > > > The information contained in this email is strictly confidential and
    > > for
    > > > > the use of the addressee only, unless otherwise indicated. If you are
    > > not
    > > > > the intended recipient, please do not read, copy, use or disclose to
    > > > others
    > > > > this message or any attachment. Please also notify the sender by
    > > replying
    > > > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
    > and then delete the
    > > > email
    > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
    > to
    > > > the
    > > > > official business of this company shall be understood as neither
    > given
    > > > nor
    > > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > > > registered in England and Wales, company number 04008957) and IG
    > Index
    > > > > Limited (a company registered in England and Wales, company number
    > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > Hill,
    > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    > > IG
    > > > > Index Limited (register number 114059) are authorised and regulated
    > by
    > > > the
    > > > > Financial Conduct Authority.
    > > > >
    > > > The information contained in this email is strictly confidential and
    > for
    > > > the use of the addressee only, unless otherwise indicated. If you are
    > not
    > > > the intended recipient, please do not read, copy, use or disclose to
    > > others
    > > > this message or any attachment. Please also notify the sender by
    > replying
    > > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
    > and then delete the
    > > email
    > > > and any copies of it. Opinions, conclusion (etc) that do not relate to
    > > the
    > > > official business of this company shall be understood as neither given
    > > nor
    > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > > registered in England and Wales, company number 04008957) and IG Index
    > > > Limited (a company registered in England and Wales, company number
    > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    > IG
    > > > Index Limited (register number 114059) are authorised and regulated by
    > > the
    > > > Financial Conduct Authority.
    > > >
    > > The information contained in this email is strictly confidential and for
    > > the use of the addressee only, unless otherwise indicated. If you are not
    > > the intended recipient, please do not read, copy, use or disclose to
    > others
    > > this message or any attachment. Please also notify the sender by replying
    > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
    > and then delete the email
    > > and any copies of it. Opinions, conclusion (etc) that do not relate to
    > the
    > > official business of this company shall be understood as neither given
    > nor
    > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > registered in England and Wales, company number 04008957) and IG Index
    > > Limited (a company registered in England and Wales, company number
    > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    > > Index Limited (register number 114059) are authorised and regulated by
    > the
    > > Financial Conduct Authority.
    > >
    >


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Ismael

Agreed this would make sense, as long as all those kips are approved before hand.

I would have reservations holding back a feature if the others are not approved before the Jan release else we run the risk of those kips staying in their current position of spinning wheels.

eg I'd like to see this kip and kip82 and others mentioned to get approved in principle before Jan. (implementation details can be refined between Jan and May.)

If that hasn't occurred then I think we should look to push for Jan for this one.

Does this sound fair?

Cheers
Mike

Sent using OWA for iPhone
________________________________________
From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <is...@juma.me.uk>
Sent: Saturday, December 17, 2016 7:19:36 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Michael,

Yes, something along those lines. Although I'd probably have the public
constructor take a `boolean` and have a private constructor for the
`Boolean` (this way, we can retire the config at some point without
affecting the API exposed by the class). Also, I would just hardcode it in
`send` instead of using an interceptor.

It would be interesting to get some thoughts on whether people think this
is OK given the constraints.

A note: message format version bumps are relatively disruptive for users
(and they impose a non-trivial maintenance cost). So, as Jun said, it would
be good to avoid bumping it many times in rapid succession. The last bump
happened in 0.10.0.0, which was released in May 2016. My personal
preference would be to wait until the May 2017 release for the next message
format bump.

We have a number of KIPs currently being discussed that require a message
format bump: KIP-82, KIP-87, KIP-98 and KIP-101. Given that a lot of people
are going to be on holiday soon and taking into account the complexity of
the mentioned KIPs, it seems that we could, at best, get one of them in for
the January release. And it seems suboptimal to bump the message format
version for just one KIP when we know we have three others lined up. I hope
that explains my reasoning. Does it make sense at all?

Ismael



On Fri, Dec 16, 2016 at 1:53 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Ismael
>
>
> So I understand what is being suggested, is we allow on the ProducerRecord
> at time of construction to be nullable aka big B, Boolean, where old
> constructors set it to null.
>
>
>     /**
>      * Creates a record with a specified timestamp to be sent to a
> specified topic and partition
>      *
>      * @param topic The topic the record will be appended to
>      * @param partition The partition to which the record should be sent
>      * @param tombstone if the record is a tombstone marker.
>      * @param timestamp The timestamp of the record
>      * @param key The key that will be included in the record
>      * @param value The record contents
>      */
>     public ProducerRecord(String topic, Integer partition, Boolean
> tombstone, Long timestamp, K key, V value) {
>         if (topic == null)
>             throw new IllegalArgumentException("Topic cannot be null");
>         if (timestamp != null && timestamp < 0)
>             throw new IllegalArgumentException("Invalid timestamp " +
> timestamp);
>         this.topic = topic;
>         this.partition = partition;
>         this.tombstone = tombstone;
>         this.key = key;
>         this.value = value;
>         this.timestamp = timestamp;
>     }
>
>     public ProducerRecord(String topic, Integer partition, Long timestamp,
> K key, V value)
>     {
>         this(topic, partition, null, timestamp, key, value);
>     }
>
>
> Then within the KafkaProducer an enforced/defaulter interceptor or simply
> hardcoded in the on send if it detects the tombstone is null, it sets the
> tombstone to true or false based on a configuration property of the
> KafkaProducer?
>
> Aka in this method, or by adding a default interceptor ->
>
>     @Override
>     public Future<RecordMetadata> send(ProducerRecord<K, V> record,
> Callback callback) {
>         // intercept the record, which can be potentially modified; this
> method does not throw exceptions
>         ProducerRecord<K, V> interceptedRecord = this.interceptors == null
> ? record : this.interceptors.onSend(record);
>         return doSend(interceptedRecord, callback);
>     }
>
> e.g sample default interceptor method we can either just add this to the
> interceptors by default or code this same logic in the send/doSend logic.
>
> public class TombstoneInterceptor implements ProducerInterceptor
> {
>    boolean defaultNullValueTombstoneMarker;
>
>    @Override
>    public void configure(Map<String, ?> configs)
>    {
>       defaultNullValueTombstoneMarker = (Boolean)configs.get("
> tombstone.default.null.value");
>    }
>
>    @Override
>    public ProducerRecord onSend(ProducerRecord record)
>    {
>       if (record.tombstone() == null){
>          boolean tombstone = false;
>          if (record.value() == null){
>             tombstone = defaultNullValueTombstoneMarker;
>          }
>          return new ProducerRecord(record.topic(), record.partition(),
> tombstone, record.timestamp(), record.key(), record.value());
>       }
>       return record;
>    }
>
>    @Override
>    public void onAcknowledgement(RecordMetadata metadata, Exception
> exception)
>    {
>
>    }
>
>    @Override
>    public void close()
>    {
>
>    }
> }
>
> Is this a correct understanding?
>
> Cheers
> Mike
>
> On 16/12/2016, 21:16, "ismaelj@gmail.com on behalf of Ismael Juma" <
> ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
>
>     Hi Michael,
>
>     I am not sure about using a new policy. One other option (suggested by
>     Gwen) is that the behaviour of the producer with regards to tombstones
> when
>     you pass a `null` value and don't use the constructor that takes a
>     tombstone can be influenced by a producer config. Not ideal, but would
>     limit the complexity of maintaining compatibility while providing a
> path
>     for the (what I think) are the right semantics.
>
>     Ismael
>
>     On Tue, Dec 13, 2016 at 8:32 AM, Michael Pearce <Michael.Pearce@ig.com
> >
>     wrote:
>
>     > Hi Ismael
>     >
>     > Did you see our email this morning, what's your thoughts on this
> approach
>     > to instead we simply have a brand new policy?
>     >
>     > Cheers
>     > Mike
>     >
>     >
>     > Sent using OWA for iPhone
>     > ________________________________________
>     > From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
> Juma <
>     > ismael@juma.me.uk>
>     > Sent: Tuesday, December 13, 2016 11:30:05 AM
>     > To: dev@kafka.apache.org
>     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     >
>     > Yes, this is actually tricky to do in a way where we both have the
> desired
>     > semantics and maintain compatibility. When someone creates a
>     > `ProducerRecord` with a `null` value today, the producer doesn't
> know if
>     > it's meant to be a tombstone or not. For V3 messages, it's easy when
> the
>     > constructor that takes a tombstone is used. However, if any other
>     > constructor is used, it's not clear. A couple of options I can think
> of,
>     > none of them particularly nice:
>     >
>     > 1. Have a third state where tombstone = unknown and the broker would
> set
>     > the tombstone bit if the value was null and the topic was compacted.
> People
>     > that wanted to pass a non-null value for the tombstone would have to
> use
>     > the constructor that takes a tombstone. The drawbacks: third state
> for
>     > tombstone in message format, message conversion at the broker for a
> common
>     > case.
>     >
>     > 2. Extend MetadataResponse to optionally include topic configs,
> which would
>     > make it possible for the producer to be smarter about setting the
>     > tombstone. It would only do it if a tombstone was not passed
> explicitly,
>     > the value was null and the topic was compacted. The main drawback is
> that
>     > the producer would be getting a bit more data for each topic even
> though it
>     > probably won't use it most of the time. Extending MetadataResponse to
>     > return topic configs would be useful for other reasons as well, so
> that
>     > part seems OK.
>     >
>     > In addition, for both proposals, we could consider adding warnings
> to the
>     > documentation that the behaviour of the constructors that don't take
> a
>     > tombstone would change in the next major release so that tombstone =
> false.
>     > Not sure if this would be worth it though.
>     >
>     > Ismael
>     >
>     > On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> ewen@confluent.io
>     > >
>     > wrote:
>     >
>     > > Michael,
>     > >
>     > > It kind of depends on how you want to interpret the tombstone
> flag. If
>     > it's
>     > > purely a producer-facing Kafka-level thing that we treat as
> internal to
>     > the
>     > > broker and log cleaner once the record is sent, then your approach
> makes
>     > > sense. You're just moving copying the null-indicates-delete
> behavior of
>     > the
>     > > old constructor into the tombstone flag.
>     > >
>     > > However, if you want this change to more generally decouple the
> idea of
>     > > deletion and null values, then you are sometimes converting what
> might
>     > be a
>     > > completely valid null value that doesn't indicate deletion into a
>     > > tombstone. Downstream applications could potentially handle these
> cases
>     > > differently given the separation of deletion from value.
>     > >
>     > > I guess the question is if we want to try to support the latter
> even for
>     > > topics where we have older produce requests. An example where this
> could
>     > > come up is in something like a CDC Connector. If we try to support
> the
>     > > semantic difference, a connector might write changes to Kafka
> using the
>     > > tombstone flag to indicate when a row was truly deleted (vs an
> update
>     > that
>     > > sets it to null but still present; this probably makes more sense
> for CDC
>     > > from document stores or extracting single columns). There are
> various
>     > > reasons we might want to maintain the full log and not turn
> compaction on
>     > > (or just use a time-based retention policy), but downstream
> applications
>     > > might care to know the difference between a delete and a null
> value. In
>     > > fact both versions of the same log (compacted and time-retention)
> could
>     > be
>     > > useful and I don't think it'll be uncommon to maintain both or use
> KIP-71
>     > > to maintain a hybrid compacted/retention topic.
>     > >
>     > > -Ewen
>     > >
>     > > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > > wrote:
>     > >
>     > > > Hi Jay,
>     > > >
>     > > > Why wouldn't that work, the tombstone value is only looked at by
> the
>     > > > broker, on a topic configured for compaction as such is benign
> on non
>     > > > compacted topics. This is as much as sending a null value
> currently
>     > > >
>     > > >
>     > > > Regards
>     > > > Mike
>     > > >
>     > > >
>     > > >
>     > > > Sent using OWA for iPhone
>     > > > ________________________________________
>     > > > From: Jay Kreps <ja...@confluent.io>
>     > > > Sent: Sunday, December 11, 2016 8:58:53 PM
>     > > > To: dev@kafka.apache.org
>     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > >
>     > > > Hey Michael,
>     > > >
>     > > > I'm not quite sure that works as that would translate ALL null
> values
>     > to
>     > > > tombstones, even for non-compacted topics that use null as an
>     > acceptable
>     > > > value sent by the producer and expected by the consumer.
>     > > >
>     > > > -Jay
>     > > >
>     > > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> Michael.Pearce@ig.com
>     > >
>     > > > wrote:
>     > > >
>     > > > > Hi Ewen,
>     > > > >
>     > > > > I think the easiest way to show this is with code.
>     > > > >
>     > > > > As you can see we keep the existing behaviour for code/binaries
>     > calling
>     > > > > the pre-existing constructors, whereby if the value is null the
>     > > tombstone
>     > > > > is set to true.
>     > > > >
>     > > > > Regards
>     > > > > Mike
>     > > > >
>     > > > >
>     > > > >
>     > > > >     /**
>     > > > >      * Creates a record with a specified timestamp to be sent
> to a
>     > > > > specified topic and partition
>     > > > >      *
>     > > > >      * @param topic The topic the record will be appended to
>     > > > >      * @param partition The partition to which the record
> should be
>     > > sent
>     > > > >      * @param timestamp The timestamp of the record
>     > > > >      * @param tombstone if the record should be treated as a
>     > tombstone
>     > > if
>     > > > > the topic is compacted
>     > > > >      * @param key The key that will be included in the record
>     > > > >      * @param value The record contents
>     > > > >      */
>     > > > >     public ProducerRecord(String topic, Integer partition,
> Boolean
>     > > > > tombstone, Long timestamp, K key, V value) {
>     > > > >         if (topic == null)
>     > > > >             throw new IllegalArgumentException("Topic cannot
> be
>     > > null.");
>     > > > >         if (timestamp != null && timestamp < 0)
>     > > > >             throw new IllegalArgumentException(
>     > > > >                     String.format("Invalid timestamp: %d.
> Timestamp
>     > > > should
>     > > > > always be non-negative or null.", timestamp));
>     > > > >         if (partition != null && partition < 0)
>     > > > >             throw new IllegalArgumentException(
>     > > > >                     String.format("Invalid partition: %d.
> Partition
>     > > > number
>     > > > > should always be non-negative or null.", partition));
>     > > > >         if (!tombstone && value == null){
>     > > > >             throw new IllegalArgumentException(
>     > > > >                     String.format("Tombstone must be true if
> null
>     > > > value"));
>     > > > >         }
>     > > > >         this.topic = topic;
>     > > > >         this.partition = partition;
>     > > > >         this.tombstone = tombstone;
>     > > > >         this.key = key;
>     > > > >         this.value = value;
>     > > > >         this.timestamp = timestamp;
>     > > > >     }
>     > > > >
>     > > > >     public ProducerRecord(String topic, Integer partition, Long
>     > > > timestamp,
>     > > > > K key, V value) {
>     > > > >         this(topic, partition, value == null, timestamp, key,
> value);
>     > > > >     }
>     > > > > ________________________________________
>     > > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
>     > > > > Sent: Sunday, December 11, 2016 5:45 AM
>     > > > > To: dev@kafka.apache.org
>     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > > >
>     > > > > It seemed like this was addressed in the migration section,
> wasn't
>     > it?
>     > > > The
>     > > > > V2 vs V3 requests and the fact that the broker will downgrade
> the
>     > > message
>     > > > > format (losing zero copy) if you issues a V2 request to a
> broker
>     > using
>     > > V3
>     > > > > format handles compatibility, does it not? The existing
> consumers
>     > won't
>     > > > see
>     > > > > the extra metadata in the value, but they will get a null
> instead and
>     > > > treat
>     > > > > it as a tombstone. Certainly there is a performance impact,
> but it
>     > > seemed
>     > > > > compatible.
>     > > > >
>     > > > > I'm worried about this though:
>     > > > >
>     > > > >
>     > > > >    - *NOTE *: With the new version of producer client using
>     > > > ProduceRequest
>     > > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not
> set) and
>     > > null
>     > > > > value
>     > > > >    should not be allowed as the older version of consumer using
>     > > > > FetchRequest
>     > > > >    V2 will think of this as a tombstone when its actually not.
>     > > > >
>     > > > >
>     > > > > Unless I'm misunderstanding, this ends up breaking binary
>     > compatibility
>     > > > for
>     > > > > the Java API. It sounds like this suggests that passing a null
> value
>     > to
>     > > > the
>     > > > > existing ProducerRecord constructors would generate an
> exception
>     > since
>     > > > you
>     > > > > didn't explicitly enable the tombstone (via whatever new
> constructor
>     > is
>     > > > > provided). But that means you can't swap in a newer client jar
>     > without
>     > > > > recompiling and get the same behavior if your app deletes keys
> using
>     > > the
>     > > > > current approach because your app will start throwing
> exceptions.
>     > Maybe
>     > > > > this is a tradeoff we're ok with, but we've tried pretty hard
> to
>     > avoid
>     > > > > breaking compatibility like this so far -- it makes picking up
> bug
>     > > fixes
>     > > > in
>     > > > > the clients harder for users of frameworks that have to pin to
>     > earlier
>     > > > > default versions for compatibility.
>     > > > >
>     > > > > But then later the KIP says:
>     > > > >
>     > > > >
>     > > > >    - The old constructors for ProducerRecord without this
> parameter
>     > > will
>     > > > be
>     > > > >    kept but updated so that their default behaviour if setting
> null
>     > > value
>     > > > > will
>     > > > >    be the tombstone will be set to true to keep existing
> behaviour.
>     > > > >
>     > > > >
>     > > > > So maybe I am misinterpreting something.
>     > > > >
>     > > > > And just a nit re: motivation section, item 6 would be clearer
> for a
>     > > > union
>     > > > > schema with null (or optional schemas in other formats), e.g.
> [null,
>     > > > > string], in which case losing the schema truly is losing
> information
>     > > > > (whereas null is already the only valid value for a pure null
>     > schema).
>     > > > >
>     > > > > -Ewen
>     > > > >
>     > > > >
>     > > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
>     > Michael.Pearce@ig.com
>     > > >
>     > > > > wrote:
>     > > > >
>     > > > > > Hi Jay,
>     > > > > >
>     > > > > > Good point this detail is missing in the KIP write up. Ive
> added
>     > this
>     > > > > now.
>     > > > > >
>     > > > > > Essentially simply just upgrading the clients we do not
> expect any
>     > > > client
>     > > > > > app code change needed.
>     > > > > >
>     > > > > > Cheers
>     > > > > > Mike
>     > > > > > ________________________________________
>     > > > > > From: Jay Kreps <ja...@confluent.io>
>     > > > > > Sent: Saturday, December 10, 2016 10:51 PM
>     > > > > > To: dev@kafka.apache.org
>     > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > > > >
>     > > > > > Michael,
>     > > > > >
>     > > > > > The compatibility section goes through the migration path,
> but
>     > isn't
>     > > > the
>     > > > > > bigger compatibility issue with existing apps? There are many
>     > > (probably
>     > > > > > thousands) of apps in production that use this feature and
> send
>     > null
>     > > to
>     > > > > > mean delete. It seems like this would break compatibility
> with
>     > them,
>     > > > and
>     > > > > > they would have to be rewritten to right?
>     > > > > >
>     > > > > > -Jay
>     > > > > >
>     > > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
>     > > Michael.Pearce@ig.com
>     > > > >
>     > > > > > wrote:
>     > > > > >
>     > > > > > > Hi Jun,
>     > > > > > >
>     > > > > > > 4) On v3 we honour the tombstone. As such we expect it to
> be set
>     > > > > > correctly
>     > > > > > > as per the KIP.
>     > > > > > >
>     > > > > > > 4.1) why would we want to produce an error when its v3?
> This is
>     > the
>     > > > > exact
>     > > > > > > purpose to support non-null tombstone's
>     > > > > > > 4.2) again here im unclear on the question on the v3,
> produce
>     > > request
>     > > > > we
>     > > > > > > expect the tombstone flag to be set correctly.
>     > > > > > >
>     > > > > > > 4.4) compaction only occurs on compacted topics, the bit
> makes no
>     > > > > > > difference and not looked at on un-compacted (time/size
> based
>     > > > > eviction).
>     > > > > > >
>     > > > > > >
>     > > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
>     > > > > > >
>     > > > > > >     Hi, Michael,
>     > > > > > >
>     > > > > > >     4. Then, I think I misunderstood this point. Could you
>     > document
>     > > > the
>     > > > > > >     following points in the wiki?
>     > > > > > >     4.1 If producer V3 sets tombstone, but provides a
> non-null
>     > > value,
>     > > > > > does
>     > > > > > > the
>     > > > > > >     send() get an error or does the producer automatically
> set
>     > the
>     > > > > value
>     > > > > > to
>     > > > > > >     null?
>     > > > > > >     4.2 If producer V3 doesn't set tombstone, but provides
> a null
>     > > > > value,
>     > > > > > > does
>     > > > > > >     the send() get an error or does the producer
> automatically
>     > sets
>     > > > the
>     > > > > > >     tombstone?
>     > > > > > >     4.3 Does the broker only expect messages that (a) have
> no
>     > > > tombstone
>     > > > > > and
>     > > > > > >     non-null value; (b) have tombstone and null value and
> reject
>     > > the
>     > > > > > > messages
>     > > > > > >     with an error code otherwise?
>     > > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
>     > compacted
>     > > on
>     > > > > > not?
>     > > > > > >
>     > > > > > >     Thanks,
>     > > > > > >
>     > > > > > >     Jun
>     > > > > > >
>     > > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
>     > > > > > Michael.Pearce@ig.com
>     > > > > > > >
>     > > > > > >     wrote:
>     > > > > > >
>     > > > > > >     > Not at all.  This only acts on compacted topics just
> as
>     > what
>     > > > > occurs
>     > > > > > > today
>     > > > > > >     >
>     > > > > > >     > Sent using OWA for iPhone
>     > > > > > >     > ________________________________________
>     > > > > > >     > From: Jun Rao <ju...@confluent.io>
>     > > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
>     > > > > > >     > To: dev@kafka.apache.org
>     > > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone Flag
>     > > > > > >     >
>     > > > > > >     > Hi, Michael,
>     > > > > > >     >
>     > > > > > >     > 4. Hmm, does that mean the new client library can
> never
>     > send
>     > > a
>     > > > > null
>     > > > > > > message
>     > > > > > >     > even to a regular topic? This seems like a change of
> the
>     > > > existing
>     > > > > > > behavior.
>     > > > > > >     >
>     > > > > > >     > Thanks,
>     > > > > > >     >
>     > > > > > >     > Jun
>     > > > > > >     >
>     > > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
>     > > > > > > Michael.Pearce@ig.com>
>     > > > > > >     > wrote:
>     > > > > > >     >
>     > > > > > >     > > Hi Jun,
>     > > > > > >     > >
>     > > > > > >     > > Re 4) That's because we expect the tombstone value
> to be
>     > > set
>     > > > > > > correctly if
>     > > > > > >     > > message bit is 2, as such if an older client sends
> in on
>     > > old
>     > > > > > > message the
>     > > > > > >     > > message is upcast and the bit is set correctly.
> And such
>     > no
>     > > > > > longer
>     > > > > > > need
>     > > > > > >     > to
>     > > > > > >     > > check the value. Mayuresh can you confirm my
> thinking and
>     > > > > > > understanding
>     > > > > > >     > of
>     > > > > > >     > > what we've discussed?
>     > > > > > >     > >
>     > > > > > >     > > The second point I understand what you're getting
> at now
>     > my
>     > > > > > > apologies.
>     > > > > > >     > Yes
>     > > > > > >     > > this makes sense to save on touching the message,
> if
>     > we're
>     > > > the
>     > > > > > > only kip
>     > > > > > >     > > going in, in this release.
>     > > > > > >     > >
>     > > > > > >     > > Cheers
>     > > > > > >     > > Mike
>     > > > > > >     > >
>     > > > > > >     > > Sent using OWA for iPhone
>     > > > > > >     > > ________________________________________
>     > > > > > >     > > From: Jun Rao <ju...@confluent.io>
>     > > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>     > > > > > >     > > To: dev@kafka.apache.org
>     > > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone
>     > Flag
>     > > > > > >     > >
>     > > > > > >     > > Hi, Michael,
>     > > > > > >     > >
>     > > > > > >     > > 4. Is this updated in the wiki? The text "If the
> magic
>     > byte
>     > > > on
>     > > > > > > message is
>     > > > > > >     > > 2, the broker should use the tombstone bit for log
>     > > > compaction."
>     > > > > > > doesn't
>     > > > > > >     > > seem to have changed.
>     > > > > > >     > >
>     > > > > > >     > > 2. My point is that if we change the message
> format just
>     > > for
>     > > > > this
>     > > > > > > KIP, we
>     > > > > > >     > > should consider whether it's worth optimizing the
> down
>     > > > > conversion
>     > > > > > > path
>     > > > > > >     > > (i.e., decide whether a conversion is needed by
> just
>     > > looking
>     > > > at
>     > > > > > the
>     > > > > > >     > > tombstone bit in the wrapper message) since
> tombstone
>     > will
>     > > be
>     > > > > > used
>     > > > > > >     > rarely.
>     > > > > > >     > > However, if the message format change here is
> combined
>     > with
>     > > > > other
>     > > > > > > KIPs,
>     > > > > > >     > > then this optimization likely won't be needed. The
> latter
>     > > > > > probably
>     > > > > > > makes
>     > > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> think?
>     > > > > > >     > >
>     > > > > > >     > > Other than those, +1 from me,
>     > > > > > >     > >
>     > > > > > >     > > Thanks,
>     > > > > > >     > >
>     > > > > > >     > > Jun
>     > > > > > >     > >
>     > > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
>     > > > > > > Michael.Pearce@ig.com>
>     > > > > > >     > > wrote:
>     > > > > > >     > >
>     > > > > > >     > > > Hi Jun
>     > > > > > >     > > >
>     > > > > > >     > > > do we have your vote on this now?
>     > > > > > >     > > >
>     > > > > > >     > > > Any other concerns?
>     > > > > > >     > > >
>     > > > > > >     > > > Cheers
>     > > > > > >     > > > Mike
>     > > > > > >     > > >
>     > > > > > >     > > > Sent using OWA for iPhone
>     > > > > > >     > > > ________________________________________
>     > > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
>     > > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>     > > > > > >     > > > To: dev@kafka.apache.org
>     > > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone
>     > > Flag
>     > > > > > >     > > >
>     > > > > > >     > > > Hi Jun,
>     > > > > > >     > > >
>     > > > > > >     > > > Have updated. Thanks again for the feedback.
>     > > > > > >     > > >
>     > > > > > >     > > > Agree yes we should align up when it gets to
> that, I
>     > > assume
>     > > > > > > you've
>     > > > > > >     > > flagged
>     > > > > > >     > > > the same in the other KIP?
>     > > > > > >     > > >
>     > > > > > >     > > > I think for now let's get this KIP approved,
> then we
>     > can
>     > > > > start
>     > > > > > > the work
>     > > > > > >     > > to
>     > > > > > >     > > > get an initial PR, then we can discuss how to
> align the
>     > > two
>     > > > > if
>     > > > > > > needed.
>     > > > > > >     > > >
>     > > > > > >     > > > Cheers,
>     > > > > > >     > > > Mike
>     > > > > > >     > > >
>     > > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
> jun@confluent.io>
>     > > wrote:
>     > > > > > >     > > >
>     > > > > > >     > > >     Hi, Michael,
>     > > > > > >     > > >
>     > > > > > >     > > >     For 2), this is fine. Could you update the
> KIP wiki
>     > > to
>     > > > > make
>     > > > > > > this
>     > > > > > >     > and
>     > > > > > >     > > > other
>     > > > > > >     > > >     points clearer? Other than that, the KIP
> looks good
>     > > to
>     > > > > me.
>     > > > > > >     > > >
>     > > > > > >     > > >     An orthogonal thing is that there are other
> KIPs
>     > such
>     > > > as
>     > > > > > > KIP-98
>     > > > > > >     > that
>     > > > > > >     > > > also
>     > > > > > >     > > >     intend to change the message format. If they
> all
>     > get
>     > > > > > > approved, we
>     > > > > > >     > > > should
>     > > > > > >     > > >     think about whether it's better to just bump
> up the
>     > > > magic
>     > > > > > > byte once
>     > > > > > >     > > to
>     > > > > > >     > > >     incorporate multiple format changes like we
> did in
>     > > > > > > KIP-31/KIP-32.
>     > > > > > >     > > >
>     > > > > > >     > > >     Thanks,
>     > > > > > >     > > >
>     > > > > > >     > > >     Jun
>     > > > > > >     > > >
>     > > > > > >     > > >
>     > > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
> Pearce <
>     > > > > > >     > > Michael.Pearce@ig.com>
>     > > > > > >     > > >     wrote:
>     > > > > > >     > > >
>     > > > > > >     > > >     > Hi Jao
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > Thanks for the response. Sorry for slow
> reply,
>     > both
>     > > > > with
>     > > > > > > personal
>     > > > > > >     > > > sickness
>     > > > > > >     > > >     > and also battling some critical issues
>     > encountered
>     > > > > since
>     > > > > > >     > upgrading
>     > > > > > >     > > to
>     > > > > > >     > > >     > 0.10.1.0
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > 1) Thans for spotting, Document error
> where we
>     > > > branched
>     > > > > > > this KIP
>     > > > > > >     > > from
>     > > > > > >     > > >     > KIP-82, will get that removed.
>     > > > > > >     > > >     > 2) Intent is to do this just at the record
>     > message
>     > > > > level.
>     > > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
>     > > > corrected.
>     > > > > > >     > > >     > 4) As per discussion thread we will support
>     > > > tombstone +
>     > > > > > > null
>     > > > > > >     > value,
>     > > > > > >     > > >     > tombstone + non null value, no tombstone +
> null
>     > > > value.
>     > > > > > >     > > >     > 5) I believe this was in the discussion
> thread,
>     > > > > @Mayuresh
>     > > > > > > is this
>     > > > > > >     > > >     > something we've overlooked? I thought we
> would
>     > down
>     > > > > > > convert and
>     > > > > > >     > > > remove the
>     > > > > > >     > > >     > value so the old consumer had existing
> behavior,
>     > or
>     > > > is
>     > > > > > > there
>     > > > > > >     > > > something we
>     > > > > > >     > > >     > haven't thought about?
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > Cheers
>     > > > > > >     > > >     > Mike
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
>     > jun@confluent.io>
>     > > > > > wrote:
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     Hi, Michael,
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     Thanks for the KIP. A few comments
> below.
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     1. The message format change contains
>     > > > > "HeadersLength
>     > > > > > >     > Headers".
>     > > > > > >     > > > Is that
>     > > > > > >     > > >     >     intended?
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     2. For compressed messageset, is the
>     > tombstone
>     > > > bit
>     > > > > > > only set
>     > > > > > >     > at
>     > > > > > >     > > > the
>     > > > > > >     > > >     > shallow
>     > > > > > >     > > >     >     level? Do we always leave that bit in
> the
>     > > wrapper
>     > > > > > > message
>     > > > > > >     > > unset?
>     > > > > > >     > > > An
>     > > > > > >     > > >     >     alternative is to set the tombstone
> bit in
>     > the
>     > > > > > wrapper
>     > > > > > > if at
>     > > > > > >     > > > least one
>     > > > > > >     > > >     >     inner message has the tombstone bit
> set. This
>     > > > makes
>     > > > > > > things a
>     > > > > > >     > > bit
>     > > > > > >     > > > more
>     > > > > > >     > > >     >     complicated, but we could potentially
> exploit
>     > > > that
>     > > > > > for
>     > > > > > >     > > > optimizing down
>     > > > > > >     > > >     >     conversion. For example, we only need
> to
>     > > convert
>     > > > > > > messages
>     > > > > > >     > with
>     > > > > > >     > > > magic 2
>     > > > > > >     > > >     > to
>     > > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit
> is set
>     > > > > > > (conversion is
>     > > > > > >     > > > always
>     > > > > > >     > > >     > needed
>     > > > > > >     > > >     >     from magic 2 to magic 0). Not sure if
> the
>     > > > > > optimization
>     > > > > > > is
>     > > > > > >     > worth
>     > > > > > >     > > > the
>     > > > > > >     > > >     >     complexity though.
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     3. The referencing of the new version
> of
>     > > > > > >     > > > ProducerRequest/FetchRequest
>     > > > > > >     > > >     > is
>     > > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> convention
>     > > > > starts
>     > > > > > at
>     > > > > > >     > version
>     > > > > > >     > > > at 0, I
>     > > > > > >     > > >     >     think the new version would be 3.
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     4. "If the magic byte on message is 2,
> the
>     > > broker
>     > > > > > > should use
>     > > > > > >     > > the
>     > > > > > >     > > >     > tombstone
>     > > > > > >     > > >     >     bit for log compaction." What about
> null
>     > value?
>     > > > My
>     > > > > > >     > > understanding
>     > > > > > >     > > > is
>     > > > > > >     > > >     > that
>     > > > > > >     > > >     >     null value will be treated the same as
>     > setting
>     > > > the
>     > > > > > > tombstone
>     > > > > > >     > > bit.
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     5. For the migration path, it would be
> useful
>     > > to
>     > > > > > > describe the
>     > > > > > >     > > > down
>     > > > > > >     > > >     >     conversion path to consumers (i.e.,
> brokers
>     > on
>     > > > > > message
>     > > > > > > format
>     > > > > > >     > > > 0.10.2
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     consumers on older version).
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     Thanks,
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     Jun
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
> Michael
>     > > Pearce <
>     > > > > > >     > > > Michael.Pearce@ig.com
>     > > > > > >     > > >     > >
>     > > > > > >     > > >     >     wrote:
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     > Hi All,
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > We have been discussing in the below
> thread
>     > > and
>     > > > > > final
>     > > > > > >     > changes
>     > > > > > >     > > > have
>     > > > > > >     > > >     > been
>     > > > > > >     > > >     >     > made to the KIP wiki based on these
>     > > > discussions.
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > We would now like to put to the vote
> the
>     > > > > following
>     > > > > > > KIP:
>     > > > > > >     > > >     >     > https://cwiki.apache.org/
>     > > > > > > confluence/display/KAFKA/KIP-
>     > > > > > >     > 87+-+
>     > > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > This kip is for having a distinct
>     > compaction
>     > > > > > > attribute
>     > > > > > >     > > > "tombstone"
>     > > > > > >     > > >     > flag
>     > > > > > >     > > >     >     > instead of relying on null value,
> allowing
>     > > > > non-null
>     > > > > > > value
>     > > > > > >     > > > delete
>     > > > > > >     > > >     > messages.
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > Many thanks,
>     > > > > > >     > > >     >     > Michael
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
> Pearce" <
>     > > > > > >     > > Michael.Pearce@ig.com>
>     > > > > > >     > > >     > wrote:
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     Hi Mayuresh,
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     LGTM. Ive just made one small
>     > adjustment
>     > > > > > > updating the
>     > > > > > >     > > wire
>     > > > > > >     > > >     > protocol to
>     > > > > > >     > > >     >     > show the magic byte bump.
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     Do we think we're good to put to
> a
>     > vote?
>     > > Is
>     > > > > > > there any
>     > > > > > >     > > > other bits
>     > > > > > >     > > >     >     > needing discussion?
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     Cheers
>     > > > > > >     > > >     >     >     Mike
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
>     > Gharat" <
>     > > > > > >     > > >     > gharatmayuresh15@gmail.com>
>     > > > > > >     > > >     >     > wrote:
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         Hi Michael,
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         I have updated the migration
>     > section
>     > > of
>     > > > > the
>     > > > > > > KIP.
>     > > > > > >     > Can
>     > > > > > >     > > > you
>     > > > > > >     > > >     > please
>     > > > > > >     > > >     >     > take a look?
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         Thanks,
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         Mayuresh
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07
> AM,
>     > > > Mayuresh
>     > > > > > > Gharat <
>     > > > > > >     > > >     >     > gharatmayuresh15@gmail.com
>     > > > > > >     > > >     >     >         > wrote:
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         > Hi Michael,
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > That whilst sending
> tombstone and
>     > > non
>     > > > > > null
>     > > > > > > value,
>     > > > > > >     > > the
>     > > > > > >     > > >     > consumer
>     > > > > > >     > > >     >     > can expect
>     > > > > > >     > > >     >     >         > only to receive the
> non-null
>     > > message
>     > > > > only
>     > > > > > > in step
>     > > > > > >     > > > (3) is
>     > > > > > >     > > >     > this
>     > > > > > >     > > >     >     > correct?
>     > > > > > >     > > >     >     >         > ---> I do agree with you
> here.
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > Becket, Ismael : can you
> guys
>     > > review
>     > > > > the
>     > > > > > >     > migration
>     > > > > > >     > > > plan
>     > > > > > >     > > >     > listed
>     > > > > > >     > > >     >     > above using
>     > > > > > >     > > >     >     >         > magic byte?
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > Thanks,
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > Mayuresh
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at
> 8:58 AM,
>     > > > > Michael
>     > > > > > > Pearce <
>     > > > > > >     > > >     >     > Michael.Pearce@ig.com>
>     > > > > > >     > > >     >     >         > wrote:
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         >> Many thanks for this
> Mayuresh. I
>     > > > don't
>     > > > > > > have any
>     > > > > > >     > > >     > objections.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> I assume we should state:
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> That whilst sending
> tombstone
>     > and
>     > > > non
>     > > > > > null
>     > > > > > >     > value,
>     > > > > > >     > > > the
>     > > > > > >     > > >     > consumer
>     > > > > > >     > > >     >     > can expect
>     > > > > > >     > > >     >     >         >> only to receive the
> non-null
>     > > message
>     > > > > > only
>     > > > > > > in
>     > > > > > >     > step
>     > > > > > >     > > > (3) is
>     > > > > > >     > > >     > this
>     > > > > > >     > > >     >     > correct?
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Cheers
>     > > > > > >     > > >     >     >         >> Mike
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Sent using OWA for iPhone
>     > > > > > >     > > >     >     >         >>
> ______________________________
>     > > > > > __________
>     > > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
>     > > > > > >     > gharatmayuresh15@gmail.com
>     > > > > > >     > > >
>     > > > > > >     > > >     >     >         >> Sent: Thursday, November
> 17,
>     > 2016
>     > > > > > 5:18:41
>     > > > > > > PM
>     > > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
>     > > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> KIP-87 -
>     > > Add
>     > > > > > > Compaction
>     > > > > > >     > > > Tombstone
>     > > > > > >     > > >     > Flag
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Hi Ismael,
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Thanks for the
> explanation.
>     > > > > > >     > > >     >     >         >> Specially I like this
> part where
>     > > in
>     > > > > you
>     > > > > > >     > mentioned
>     > > > > > >     > > > we can
>     > > > > > >     > > >     > get
>     > > > > > >     > > >     >     > rid of the
>     > > > > > >     > > >     >     >         >> older null value support
> for log
>     > > > > > > compaction
>     > > > > > >     > later
>     > > > > > >     > > > on,
>     > > > > > >     > > >     > here :
>     > > > > > >     > > >     >     >         >> We can't change semantics
> of the
>     > > > > message
>     > > > > > > format
>     > > > > > >     > > > without
>     > > > > > >     > > >     > having
>     > > > > > >     > > >     >     > a long
>     > > > > > >     > > >     >     >         >> transition period. And we
> can't
>     > > rely
>     > > > > > >     > > >     >     >         >> on people reading
> documentation
>     > or
>     > > > > > acting
>     > > > > > > on a
>     > > > > > >     > > > warning for
>     > > > > > >     > > >     >     > something so
>     > > > > > >     > > >     >     >         >> fundamental. As such, my
> take is
>     > > > that
>     > > > > we
>     > > > > > > need to
>     > > > > > >     > > > bump the
>     > > > > > >     > > >     > magic
>     > > > > > >     > > >     >     > byte. The
>     > > > > > >     > > >     >     >         >> good news is
>     > > > > > >     > > >     >     >         >> that we don't have to
> support
>     > all
>     > > > > > versions
>     > > > > > >     > > forever.
>     > > > > > >     > > > We
>     > > > > > >     > > >     > have
>     > > > > > >     > > >     >     > said that we
>     > > > > > >     > > >     >     >         >> will support direct
> upgrades
>     > for 2
>     > > > > > years.
>     > > > > > > That
>     > > > > > >     > > > means that
>     > > > > > >     > > >     >     > message format
>     > > > > > >     > > >     >     >         >> version n could, in
> theory, be
>     > > > > removed 2
>     > > > > > > years
>     > > > > > >     > > > after the
>     > > > > > >     > > >     > it's
>     > > > > > >     > > >     >     > introduced.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Just a heads up, I would
> like to
>     > > > > mention
>     > > > > > > that
>     > > > > > >     > even
>     > > > > > >     > > > without
>     > > > > > >     > > >     >     > bumping magic
>     > > > > > >     > > >     >     >         >> byte, we will *NOT* loose
> zero
>     > > copy
>     > > > as
>     > > > > > in
>     > > > > > > the
>     > > > > > >     > > > client(x+1)
>     > > > > > >     > > >     > in my
>     > > > > > >     > > >     >     >         >> explanation
>     > > > > > >     > > >     >     >         >> above will convert
> internally a
>     > > null
>     > > > > > > value to
>     > > > > > >     > > have a
>     > > > > > >     > > >     > tombstone
>     > > > > > >     > > >     >     > bit set and
>     > > > > > >     > > >     >     >         >> a tombstone bit set to
> have a
>     > null
>     > > > > value
>     > > > > > >     > > > automatically
>     > > > > > >     > > >     >     > internally and by
>     > > > > > >     > > >     >     >         >> the time we move to
> version
>     > (x+2),
>     > > > the
>     > > > > > > clients
>     > > > > > >     > > > would have
>     > > > > > >     > > >     >     > upgraded.
>     > > > > > >     > > >     >     >         >> Obviously if we support a
>     > request
>     > > > from
>     > > > > > >     > > consumer(x),
>     > > > > > >     > > > we
>     > > > > > >     > > >     > will
>     > > > > > >     > > >     >     > loose zero
>     > > > > > >     > > >     >     >         >> copy
>     > > > > > >     > > >     >     >         >> but that is the same case
> with
>     > > magic
>     > > > > > byte.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> But if magic byte bump
> makes
>     > life
>     > > > > easier
>     > > > > > > for
>     > > > > > >     > > > transition
>     > > > > > >     > > >     > for the
>     > > > > > >     > > >     >     > above
>     > > > > > >     > > >     >     >         >> reasons that you
> explained, I am
>     > > OK
>     > > > > with
>     > > > > > > it
>     > > > > > >     > since
>     > > > > > >     > > > we are
>     > > > > > >     > > >     > going
>     > > > > > >     > > >     >     > to meet the
>     > > > > > >     > > >     >     >         >> end goal down the road :)
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> On a side note can we
> update the
>     > > doc
>     > > > > > here
>     > > > > > > on
>     > > > > > >     > magic
>     > > > > > >     > > > byte
>     > > > > > >     > > >     > to say
>     > > > > > >     > > >     >     > that "*it
>     > > > > > >     > > >     >     >         >> should be bumped whenever
> the
>     > > > message
>     > > > > > > format is
>     > > > > > >     > > > changed
>     > > > > > >     > > >     > or the
>     > > > > > >     > > >     >     >         >> interpretation of message
> format
>     > > > > (usage
>     > > > > > > of the
>     > > > > > >     > > > reserved
>     > > > > > >     > > >     > bits as
>     > > > > > >     > > >     >     > well) is
>     > > > > > >     > > >     >     >         >> changed*".
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Hi Michael,
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Here is the update plan
> that we
>     > > > > > discussed
>     > > > > > >     > offline
>     > > > > > >     > > >     > yesterday :
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Currently the magic-byte
> which
>     > > > > > > corresponds to
>     > > > > > >     > the
>     > > > > > >     > > >     >     > "message.format.version"
>     > > > > > >     > > >     >     >         >> is set to 1.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> 1) On broker it will be
> set to 1
>     > > > > > > initially.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> 2) When a producer client
> sends
>     > a
>     > > > > > message
>     > > > > > > with
>     > > > > > >     > > > magic-byte
>     > > > > > >     > > >     > = 2,
>     > > > > > >     > > >     >     > since the
>     > > > > > >     > > >     >     >         >> broker is on magic-byte =
> 1, we
>     > > will
>     > > > > > down
>     > > > > > >     > convert
>     > > > > > >     > > > it,
>     > > > > > >     > > >     > which
>     > > > > > >     > > >     >     > means if the
>     > > > > > >     > > >     >     >         >> tombstone bit is set, the
> value
>     > > will
>     > > > > be
>     > > > > > > set to
>     > > > > > >     > > > null. A
>     > > > > > >     > > >     > consumer
>     > > > > > >     > > >     >     >         >> understanding magic-byte
> = 1,
>     > will
>     > > > > still
>     > > > > > > work
>     > > > > > >     > with
>     > > > > > >     > > > this. A
>     > > > > > >     > > >     >     > consumer
>     > > > > > >     > > >     >     >         >> working
>     > > > > > >     > > >     >     >         >> with magic-byte =2 will
> also be
>     > > able
>     > > > > to
>     > > > > > >     > understand
>     > > > > > >     > > > this,
>     > > > > > >     > > >     > since
>     > > > > > >     > > >     >     > it
>     > > > > > >     > > >     >     >         >> understands the tombstone.
>     > > > > > >     > > >     >     >         >> Now there is still the
> question
>     > of
>     > > > > > > supporting a
>     > > > > > >     > > >     > non-tombstone
>     > > > > > >     > > >     >     > and null
>     > > > > > >     > > >     >     >         >> value from producer
> client with
>     > > > > > > magic-byte = 2.*
>     > > > > > >     > > (I
>     > > > > > >     > > > am
>     > > > > > >     > > >     > not sure
>     > > > > > >     > > >     >     > if we
>     > > > > > >     > > >     >     >         >> should support this.
>     > Ismael/Becket
>     > > > can
>     > > > > > > comment
>     > > > > > >     > > > here)*
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> 3) When almost all the
> clients
>     > > have
>     > > > > > > upgraded,
>     > > > > > >     > the
>     > > > > > >     > > >     >     > message.format.version
>     > > > > > >     > > >     >     >         >> on
>     > > > > > >     > > >     >     >         >> the broker can be changed
> to 2,
>     > > > where
>     > > > > in
>     > > > > > > the
>     > > > > > >     > down
>     > > > > > >     > > >     > conversion in
>     > > > > > >     > > >     >     > the above
>     > > > > > >     > > >     >     >         >> step will not happen. If
> at this
>     > > > point
>     > > > > > we
>     > > > > > > get a
>     > > > > > >     > > > consumer
>     > > > > > >     > > >     >     > request from a
>     > > > > > >     > > >     >     >         >> older consumer, we might
> have to
>     > > > down
>     > > > > > > convert
>     > > > > > >     > > where
>     > > > > > >     > > > in we
>     > > > > > >     > > >     > loose
>     > > > > > >     > > >     >     > zero copy,
>     > > > > > >     > > >     >     >         >> but these cases should be
> rare.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Becket can you review
> this plan
>     > > and
>     > > > > add
>     > > > > > > more
>     > > > > > >     > > > details if I
>     > > > > > >     > > >     > have
>     > > > > > >     > > >     >     >         >> missed/wronged something,
> before
>     > > we
>     > > > > put
>     > > > > > > it on
>     > > > > > >     > KIP.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Thanks,
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Mayuresh
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> 11:07
>     > PM,
>     > > > > > Michael
>     > > > > > >     > Pearce <
>     > > > > > >     > > >     >     > Michael.Pearce@ig.com>
>     > > > > > >     > > >     >     >         >> wrote:
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> > Thanks guys, for
> discussing
>     > this
>     > > > > > > offline and
>     > > > > > >     > > > getting
>     > > > > > >     > > >     > some
>     > > > > > >     > > >     >     > consensus.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > So its clear for myself
> and
>     > > others
>     > > > > > what
>     > > > > > > is
>     > > > > > >     > > > proposed now
>     > > > > > >     > > >     > (i
>     > > > > > >     > > >     >     > think i
>     > > > > > >     > > >     >     >         >> > understand, but want to
> make
>     > > sure)
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Could i ask either
> directly
>     > > update
>     > > > > the
>     > > > > > > kip to
>     > > > > > >     > > > detail the
>     > > > > > >     > > >     >     > migration
>     > > > > > >     > > >     >     >         >> > strategy, or (re-)state
> your
>     > > > offline
>     > > > > > > discussed
>     > > > > > >     > > and
>     > > > > > >     > > >     > agreed
>     > > > > > >     > > >     >     > migration
>     > > > > > >     > > >     >     >         >> > strategy based on a
> magic byte
>     > > is
>     > > > in
>     > > > > > > this
>     > > > > > >     > > thread.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > The main original
> driver for
>     > the
>     > > > KIP
>     > > > > > > was to
>     > > > > > >     > > > support
>     > > > > > >     > > >     >     > compaction where
>     > > > > > >     > > >     >     >         >> value
>     > > > > > >     > > >     >     >         >> > isn't null, based off
> the
>     > > > > discussions
>     > > > > > on
>     > > > > > >     > KIP-82
>     > > > > > >     > > > thread.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > We should be able to
> support
>     > > > > > > non-tombstone +
>     > > > > > >     > > null
>     > > > > > >     > > > value
>     > > > > > >     > > >     > by the
>     > > > > > >     > > >     >     >         >> completion
>     > > > > > >     > > >     >     >         >> > of the KIP, as we noted
> when
>     > > > > > discussing
>     > > > > > > this
>     > > > > > >     > > kip,
>     > > > > > >     > > > having
>     > > > > > >     > > >     >     > logic based on
>     > > > > > >     > > >     >     >         >> a
>     > > > > > >     > > >     >     >         >> > null value isn't very
> clean
>     > and
>     > > > also
>     > > > > > > separates
>     > > > > > >     > > the
>     > > > > > >     > > >     > concerns.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > As discussed already
> though we
>     > > can
>     > > > > > > split this
>     > > > > > >     > > into
>     > > > > > >     > > >     > KIP-87a
>     > > > > > >     > > >     >     > and KIP-87b
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Where we look to deliver
>     > KIP-87a
>     > > > on
>     > > > > a
>     > > > > > >     > compacted
>     > > > > > >     > > > topic
>     > > > > > >     > > >     > (to
>     > > > > > >     > > >     >     > address the
>     > > > > > >     > > >     >     >         >> > immediate issues)
>     > > > > > >     > > >     >     >         >> > * tombstone + null value
>     > > > > > >     > > >     >     >         >> > * tombstone + non-null
> value
>     > > > > > >     > > >     >     >         >> > * non-tombstone +
> non-null
>     > value
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Then we can discuss once
>     > KIP-87a
>     > > > is
>     > > > > > > completed
>     > > > > > >     > > > options
>     > > > > > >     > > >     > later
>     > > > > > >     > > >     >     > and how we
>     > > > > > >     > > >     >     >         >> > support the second part
>     > KIP-87b
>     > > to
>     > > > > > > deliver:
>     > > > > > >     > > >     >     >         >> > * non-tombstone + null
> value
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Cheers
>     > > > > > >     > > >     >     >         >> > Mike
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
> ______________________________
>     > > > > > > __________
>     > > > > > >     > > >     >     >         >> > From: Becket Qin <
>     > > > > > becket.qin@gmail.com>
>     > > > > > >     > > >     >     >         >> > Sent: Thursday,
> November 17,
>     > > 2016
>     > > > > 1:43
>     > > > > > > AM
>     > > > > > >     > > >     >     >         >> > To:
> dev@kafka.apache.org
>     > > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> KIP-87
>     > -
>     > > > Add
>     > > > > > > Compaction
>     > > > > > >     > > >     > Tombstone Flag
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Renu, Mayuresh and I
> had an
>     > > > offline
>     > > > > > >     > discussion,
>     > > > > > >     > > > and
>     > > > > > >     > > >     > following
>     > > > > > >     > > >     >     > is a brief
>     > > > > > >     > > >     >     >         >> > summary.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > 1. We agreed that not
> bumping
>     > up
>     > > > > magic
>     > > > > > > value
>     > > > > > >     > may
>     > > > > > >     > > > result
>     > > > > > >     > > >     > in
>     > > > > > >     > > >     >     > losing zero
>     > > > > > >     > > >     >     >         >> copy
>     > > > > > >     > > >     >     >         >> > during migration.
>     > > > > > >     > > >     >     >         >> > 2. Given that bumping
> up magic
>     > > > value
>     > > > > > is
>     > > > > > > almost
>     > > > > > >     > > > free and
>     > > > > > >     > > >     > has
>     > > > > > >     > > >     >     > benefit of
>     > > > > > >     > > >     >     >         >> > avoiding potential
> performance
>     > > > > issue.
>     > > > > > > It is
>     > > > > > >     > > > probably
>     > > > > > >     > > >     > worth
>     > > > > > >     > > >     >     > doing.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > One issue we still need
> to
>     > think
>     > > > > about
>     > > > > > > is
>     > > > > > >     > > whether
>     > > > > > >     > > > we
>     > > > > > >     > > >     > want to
>     > > > > > >     > > >     >     > support a
>     > > > > > >     > > >     >     >         >> > non-tombstone message
> with
>     > null
>     > > > > value.
>     > > > > > >     > > >     >     >         >> > Currently it is not
> supported
>     > by
>     > > > > > Kafka.
>     > > > > > > If we
>     > > > > > >     > > > allow a
>     > > > > > >     > > >     >     > non-tombstone null
>     > > > > > >     > > >     >     >         >> > value message to exist
> after
>     > > > KIP-87.
>     > > > > > The
>     > > > > > >     > problem
>     > > > > > >     > > > is
>     > > > > > >     > > >     > that such
>     > > > > > >     > > >     >     > message
>     > > > > > >     > > >     >     >         >> will
>     > > > > > >     > > >     >     >         >> > not be supported by the
>     > > consumers
>     > > > > > prior
>     > > > > > > to
>     > > > > > >     > > KIP-87.
>     > > > > > >     > > >     > Because a
>     > > > > > >     > > >     >     > null value
>     > > > > > >     > > >     >     >         >> > will always be
> interpreted to
>     > a
>     > > > > > > tombstone.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > One option is that we
> keep the
>     > > > > current
>     > > > > > > way,
>     > > > > > >     > i.e.
>     > > > > > >     > > > do not
>     > > > > > >     > > >     >     > support such
>     > > > > > >     > > >     >     >         >> > message. It would be
> good to
>     > > know
>     > > > if
>     > > > > > > there is
>     > > > > > >     > a
>     > > > > > >     > > >     > concrete use
>     > > > > > >     > > >     >     > case for
>     > > > > > >     > > >     >     >         >> such
>     > > > > > >     > > >     >     >         >> > message. If there is
> not, we
>     > can
>     > > > > > > probably just
>     > > > > > >     > > not
>     > > > > > >     > > >     > support it.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Thanks,
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at
> 1:28
>     > PM,
>     > > > > > > Mayuresh
>     > > > > > >     > > Gharat <
>     > > > > > >     > > >     >     >         >> >
> gharatmayuresh15@gmail.com
>     > > > > > >     > > >     >     >         >> > > wrote:
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > > Hi Ismael,
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > This is something I
> can
>     > think
>     > > of
>     > > > > for
>     > > > > > >     > migration
>     > > > > > >     > > > plan:
>     > > > > > >     > > >     >     >         >> > > So the migration plan
> can
>     > look
>     > > > > > > something
>     > > > > > >     > like
>     > > > > > >     > > > this,
>     > > > > > >     > > >     > with up
>     > > > > > >     > > >     >     >         >> conversion :
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > 1) Currently lets say
> we
>     > have
>     > > > > Broker
>     > > > > > > at
>     > > > > > >     > > version
>     > > > > > >     > > > x.
>     > > > > > >     > > >     >     >         >> > > 2) Currently we have
> clients
>     > > at
>     > > > > > > version x.
>     > > > > > >     > > >     >     >         >> > > 3) a) We move the
> version to
>     > > > > > > Broker(x+1) :
>     > > > > > >     > > > supports
>     > > > > > >     > > >     > both
>     > > > > > >     > > >     >     > tombstone and
>     > > > > > >     > > >     >     >         >> > null
>     > > > > > >     > > >     >     >         >> > > for log compaction.
>     > > > > > >     > > >     >     >         >> > >     b) We upgrade the
> client
>     > > to
>     > > > > > > version
>     > > > > > >     > > > client(x+1) :
>     > > > > > >     > > >     > if in
>     > > > > > >     > > >     >     > the
>     > > > > > >     > > >     >     >         >> producer
>     > > > > > >     > > >     >     >         >> > > client(x+1) the value
> is set
>     > > to
>     > > > > > null,
>     > > > > > > we
>     > > > > > >     > will
>     > > > > > >     > > >     > automatically
>     > > > > > >     > > >     >     > set the
>     > > > > > >     > > >     >     >         >> > > Tombstone bit
> internally. If
>     > > the
>     > > > > > > producer
>     > > > > > >     > > > client(x+1)
>     > > > > > >     > > >     > sets
>     > > > > > >     > > >     >     > the
>     > > > > > >     > > >     >     >         >> tombstone
>     > > > > > >     > > >     >     >         >> > > itself, well and
> good. For
>     > > > > producer
>     > > > > > >     > client(x),
>     > > > > > >     > > > the
>     > > > > > >     > > >     > broker
>     > > > > > >     > > >     >     > will up
>     > > > > > >     > > >     >     >         >> convert
>     > > > > > >     > > >     >     >         >> > > to have the tombstone
> bit.
>     > > > > > > Broker(x+1) is
>     > > > > > >     > > > supporting
>     > > > > > >     > > >     > both.
>     > > > > > >     > > >     >     > Consumer
>     > > > > > >     > > >     >     >         >> > > client(x+1) will be
> aware of
>     > > > this
>     > > > > > and
>     > > > > > > should
>     > > > > > >     > > be
>     > > > > > >     > > > able
>     > > > > > >     > > >     > to
>     > > > > > >     > > >     >     > handle this.
>     > > > > > >     > > >     >     >         >> For
>     > > > > > >     > > >     >     >         >> > > consumer client(x) we
> will
>     > > down
>     > > > > > > convert the
>     > > > > > >     > > > message
>     > > > > > >     > > >     > on the
>     > > > > > >     > > >     >     > broker
>     > > > > > >     > > >     >     >         >> side.
>     > > > > > >     > > >     >     >         >> > >     c) At this point
> we will
>     > > > have
>     > > > > to
>     > > > > > >     > specify a
>     > > > > > >     > > >     > warning or
>     > > > > > >     > > >     >     > clearly
>     > > > > > >     > > >     >     >         >> specify
>     > > > > > >     > > >     >     >         >> > > in docs that this
> behavior
>     > is
>     > > > > about
>     > > > > > > to be
>     > > > > > >     > > > changed for
>     > > > > > >     > > >     > log
>     > > > > > >     > > >     >     > compaction.
>     > > > > > >     > > >     >     >         >> > > 4) a) In next release
> of the
>     > > > > > > Broker(x+2), we
>     > > > > > >     > > > say that
>     > > > > > >     > > >     > only
>     > > > > > >     > > >     >     > Tombstone
>     > > > > > >     > > >     >     >         >> is
>     > > > > > >     > > >     >     >         >> > > used for log
> compaction on
>     > the
>     > > > > > Broker
>     > > > > > > side.
>     > > > > > >     > > >     > Clients(x+1)
>     > > > > > >     > > >     >     > still is
>     > > > > > >     > > >     >     >         >> > > supported.
>     > > > > > >     > > >     >     >         >> > >     b) We upgrade the
> client
>     > > to
>     > > > > > > version
>     > > > > > >     > > > client(x+2) :
>     > > > > > >     > > >     > if
>     > > > > > >     > > >     >     > value is set
>     > > > > > >     > > >     >     >         >> to
>     > > > > > >     > > >     >     >         >> > > null, tombstone will
> not be
>     > > set
>     > > > > > >     > automatically.
>     > > > > > >     > > > The
>     > > > > > >     > > >     > client
>     > > > > > >     > > >     >     > will have to
>     > > > > > >     > > >     >     >         >> > call
>     > > > > > >     > > >     >     >         >> > > setTombstone() to
> actually
>     > set
>     > > > the
>     > > > > > >     > tombstone.
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > We should compare this
>     > > migration
>     > > > > > plan
>     > > > > > > with
>     > > > > > >     > the
>     > > > > > >     > > >     > migration
>     > > > > > >     > > >     >     > plan for
>     > > > > > >     > > >     >     >         >> magic
>     > > > > > >     > > >     >     >         >> > > byte bump and do
> whatever
>     > > looks
>     > > > > > good.
>     > > > > > >     > > >     >     >         >> > > I am just worried
> that if we
>     > > go
>     > > > > down
>     > > > > > > magic
>     > > > > > >     > > byte
>     > > > > > >     > > > route,
>     > > > > > >     > > >     >     > unless I am
>     > > > > > >     > > >     >     >         >> > missing
>     > > > > > >     > > >     >     >         >> > > something, it sounds
> like
>     > > kafka
>     > > > > will
>     > > > > > > be
>     > > > > > >     > stuck
>     > > > > > >     > > > with
>     > > > > > >     > > >     >     > supporting both
>     > > > > > >     > > >     >     >         >> null
>     > > > > > >     > > >     >     >         >> > > value and tombstone
> bit for
>     > > log
>     > > > > > > compaction
>     > > > > > >     > for
>     > > > > > >     > > > life
>     > > > > > >     > > >     > long,
>     > > > > > >     > > >     >     > which does
>     > > > > > >     > > >     >     >         >> not
>     > > > > > >     > > >     >     >         >> > > look like a good end
> state.
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > Thanks,
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > Mayuresh
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016
> at 9:32
>     > > AM,
>     > > > > > > Mayuresh
>     > > > > > >     > > > Gharat <
>     > > > > > >     > > >     >     >         >> > >
> gharatmayuresh15@gmail.com
>     > > > > > >     > > >     >     >         >> > > > wrote:
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > > Hi Ismael,
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > That's a very good
> point
>     > > > which I
>     > > > > > > might
>     > > > > > >     > have
>     > > > > > >     > > > not
>     > > > > > >     > > >     >     > considered earlier.
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > Here is a plan that
> I can
>     > > > think
>     > > > > > of:
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > Stage 1) The broker
> from
>     > now
>     > > > on,
>     > > > > > up
>     > > > > > >     > converts
>     > > > > > >     > > > the
>     > > > > > >     > > >     > message
>     > > > > > >     > > >     >     > to have the
>     > > > > > >     > > >     >     >         >> > > > tombstone marker.
> The log
>     > > > > > compaction
>     > > > > > >     > thread
>     > > > > > >     > > > does log
>     > > > > > >     > > >     >     > compaction
>     > > > > > >     > > >     >     >         >> based
>     > > > > > >     > > >     >     >         >> > on
>     > > > > > >     > > >     >     >         >> > > > both null and
> tombstone
>     > > > marker.
>     > > > > > > This is
>     > > > > > >     > our
>     > > > > > >     > > >     > transition
>     > > > > > >     > > >     >     > period.
>     > > > > > >     > > >     >     >         >> > > > Stage 2) The next
> release
>     > we
>     > > > > only
>     > > > > > > say that
>     > > > > > >     > > log
>     > > > > > >     > > >     > compaction
>     > > > > > >     > > >     >     > is based
>     > > > > > >     > > >     >     >         >> on
>     > > > > > >     > > >     >     >         >> > > > tombstone marker.
> (Open
>     > > source
>     > > > > > > kafka makes
>     > > > > > >     > > > this as a
>     > > > > > >     > > >     >     > policy). By
>     > > > > > >     > > >     >     >         >> this
>     > > > > > >     > > >     >     >         >> > > time,
>     > > > > > >     > > >     >     >         >> > > > the organization
> which is
>     > > > moving
>     > > > > > to
>     > > > > > > this
>     > > > > > >     > > > release
>     > > > > > >     > > >     > will be
>     > > > > > >     > > >     >     > sure that
>     > > > > > >     > > >     >     >         >> they
>     > > > > > >     > > >     >     >         >> > > > have gone through
> the
>     > entire
>     > > > > > > transition
>     > > > > > >     > > > period.
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > My only goal of
> doing this
>     > > is
>     > > > > that
>     > > > > > > Kafka
>     > > > > > >     > > > clearly
>     > > > > > >     > > >     >     > specifies the end
>     > > > > > >     > > >     >     >         >> > state
>     > > > > > >     > > >     >     >         >> > > > about what log
> compaction
>     > > > means
>     > > > > > (is
>     > > > > > > it
>     > > > > > >     > null
>     > > > > > >     > > > value
>     > > > > > >     > > >     > or a
>     > > > > > >     > > >     >     > tombstone
>     > > > > > >     > > >     >     >         >> > marker,
>     > > > > > >     > > >     >     >         >> > > > but not both).
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > What do you think?
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > Thanks,
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > Mayuresh
>     > > > > > >     > > >     >     >         >> > > > .
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > On Wed, Nov 16,
> 2016 at
>     > 9:17
>     > > > AM,
>     > > > > > > Ismael
>     > > > > > >     > > Juma <
>     > > > > > >     > > >     >     > ismael@juma.me.uk>
>     > > > > > >     > > >     >     >         >> > wrote:
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > >> One comment below.
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16,
> 2016 at
>     > > 5:08
>     > > > > PM,
>     > > > > > > Mayuresh
>     > > > > > >     > > > Gharat <
>     > > > > > >     > > >     >     >         >> > > >>
>     > gharatmayuresh15@gmail.com
>     > > > > > >     > > >     >     >         >> > > >> > wrote:
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >> >    - If we don't
> bump
>     > up
>     > > > the
>     > > > > > > magic
>     > > > > > >     > byte,
>     > > > > > >     > > > on the
>     > > > > > >     > > >     > broker
>     > > > > > >     > > >     >     > side, the
>     > > > > > >     > > >     >     >         >> > > broker
>     > > > > > >     > > >     >     >         >> > > >> >    will always
> have to
>     > > look
>     > > > > at
>     > > > > > > both
>     > > > > > >     > > > tombstone
>     > > > > > >     > > >     > bit and
>     > > > > > >     > > >     >     > the value
>     > > > > > >     > > >     >     >         >> when
>     > > > > > >     > > >     >     >         >> > > do
>     > > > > > >     > > >     >     >         >> > > >> the
>     > > > > > >     > > >     >     >         >> > > >> >    compaction.
> Assuming
>     > > we
>     > > > do
>     > > > > > > not bump
>     > > > > > >     > up
>     > > > > > >     > > > the
>     > > > > > >     > > >     > magic
>     > > > > > >     > > >     >     > byte,
>     > > > > > >     > > >     >     >         >> > > >> >    imagine the
> broker
>     > > sees
>     > > > a
>     > > > > > > message
>     > > > > > >     > > which
>     > > > > > >     > > > does
>     > > > > > >     > > >     > not
>     > > > > > >     > > >     >     > have a
>     > > > > > >     > > >     >     >         >> tombstone
>     > > > > > >     > > >     >     >         >> > > bit
>     > > > > > >     > > >     >     >         >> > > >> >    set. The
> broker does
>     > > not
>     > > > > > know
>     > > > > > > when
>     > > > > > >     > the
>     > > > > > >     > > >     > message was
>     > > > > > >     > > >     >     > produced
>     > > > > > >     > > >     >     >         >> (i.e.
>     > > > > > >     > > >     >     >         >> > > >> > whether
>     > > > > > >     > > >     >     >         >> > > >> >    the message
> has been
>     > > up
>     > > > > > > converted or
>     > > > > > >     > > > not), it
>     > > > > > >     > > >     > has
>     > > > > > >     > > >     >     > to take a
>     > > > > > >     > > >     >     >         >> > further
>     > > > > > >     > > >     >     >         >> > > >> > look at
>     > > > > > >     > > >     >     >         >> > > >> >    the value to
> see if
>     > it
>     > > > is
>     > > > > > > null or
>     > > > > > >     > not
>     > > > > > >     > > in
>     > > > > > >     > > >     > order to
>     > > > > > >     > > >     >     > determine
>     > > > > > >     > > >     >     >         >> if it
>     > > > > > >     > > >     >     >         >> > > is
>     > > > > > >     > > >     >     >         >> > > >> a
>     > > > > > >     > > >     >     >         >> > > >> >    tombstone.
> The same
>     > > > logic
>     > > > > > has
>     > > > > > > to be
>     > > > > > >     > > put
>     > > > > > >     > > > on the
>     > > > > > >     > > >     >     > consumer as
>     > > > > > >     > > >     >     >         >> well
>     > > > > > >     > > >     >     >         >> > > >> because
>     > > > > > >     > > >     >     >         >> > > >> > the
>     > > > > > >     > > >     >     >         >> > > >> >    consumer does
> not
>     > know
>     > > > if
>     > > > > > the
>     > > > > > >     > message
>     > > > > > >     > > > has
>     > > > > > >     > > >     > been up
>     > > > > > >     > > >     >     > converted or
>     > > > > > >     > > >     >     >         >> > not.
>     > > > > > >     > > >     >     >         >> > > >> >       - If we
> upconvert
>     > > > while
>     > > > > > >     > appending,
>     > > > > > >     > > > this is
>     > > > > > >     > > >     > not
>     > > > > > >     > > >     >     > the case,
>     > > > > > >     > > >     >     >         >> > right?
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >> If I understand you
>     > > > correctly,
>     > > > > > > this is
>     > > > > > >     > not
>     > > > > > >     > > >     > sufficient
>     > > > > > >     > > >     >     > because the
>     > > > > > >     > > >     >     >         >> log
>     > > > > > >     > > >     >     >         >> > > may
>     > > > > > >     > > >     >     >         >> > > >> have messages
> appended
>     > > before
>     > > > > it
>     > > > > > > was
>     > > > > > >     > > > upgraded to
>     > > > > > >     > > >     > include
>     > > > > > >     > > >     >     > KIP-87.
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >> Ismael
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > --
>     > > > > > >     > > >     >     >         >> > > > -Regards,
>     > > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         >> > > > (862) 250-7125
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > --
>     > > > > > >     > > >     >     >         >> > > -Regards,
>     > > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         >> > > (862) 250-7125
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > The information
> contained in
>     > > this
>     > > > > > email
>     > > > > > > is
>     > > > > > >     > > > strictly
>     > > > > > >     > > >     >     > confidential and for
>     > > > > > >     > > >     >     >         >> > the use of the
> addressee only,
>     > > > > unless
>     > > > > > >     > otherwise
>     > > > > > >     > > >     > indicated. If
>     > > > > > >     > > >     >     > you are
>     > > > > > >     > > >     >     >         >> not
>     > > > > > >     > > >     >     >         >> > the intended recipient,
> please
>     > > do
>     > > > > not
>     > > > > > > read,
>     > > > > > >     > > copy,
>     > > > > > >     > > > use or
>     > > > > > >     > > >     >     > disclose to
>     > > > > > >     > > >     >     >         >> others
>     > > > > > >     > > >     >     >         >> > this message or any
>     > attachment.
>     > > > > Please
>     > > > > > > also
>     > > > > > >     > > > notify the
>     > > > > > >     > > >     > sender
>     > > > > > >     > > >     >     > by
>     > > > > > >     > > >     >     >         >> replying
>     > > > > > >     > > >     >     >         >> > to this email or by
> telephone
>     > > > > (+44(020
>     > > > > > > 7896
>     > > > > > >     > > 0011)
>     > > > > > >     > > > and
>     > > > > > >     > > >     > then
>     > > > > > >     > > >     >     > delete the
>     > > > > > >     > > >     >     >         >> email
>     > > > > > >     > > >     >     >         >> > and any copies of it.
>     > Opinions,
>     > > > > > > conclusion
>     > > > > > >     > (etc)
>     > > > > > >     > > > that
>     > > > > > >     > > >     > do not
>     > > > > > >     > > >     >     > relate to
>     > > > > > >     > > >     >     >         >> the
>     > > > > > >     > > >     >     >         >> > official business of
> this
>     > > company
>     > > > > > shall
>     > > > > > > be
>     > > > > > >     > > > understood as
>     > > > > > >     > > >     >     > neither given
>     > > > > > >     > > >     >     >         >> nor
>     > > > > > >     > > >     >     >         >> > endorsed by it. IG is a
>     > trading
>     > > > name
>     > > > > > of
>     > > > > > > IG
>     > > > > > >     > > Markets
>     > > > > > >     > > >     > Limited (a
>     > > > > > >     > > >     >     > company
>     > > > > > >     > > >     >     >         >> > registered in England
> and
>     > Wales,
>     > > > > > company
>     > > > > > >     > number
>     > > > > > >     > > >     > 04008957) and
>     > > > > > >     > > >     >     > IG Index
>     > > > > > >     > > >     >     >         >> > Limited (a company
> registered
>     > in
>     > > > > > > England and
>     > > > > > >     > > > Wales,
>     > > > > > >     > > >     > company
>     > > > > > >     > > >     >     > number
>     > > > > > >     > > >     >     >         >> > 01190902). Registered
> address
>     > at
>     > > > > > Cannon
>     > > > > > > Bridge
>     > > > > > >     > > > House, 25
>     > > > > > >     > > >     >     > Dowgate Hill,
>     > > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
>     > Markets
>     > > > > > Limited
>     > > > > > >     > > (register
>     > > > > > >     > > >     > number
>     > > > > > >     > > >     >     > 195355) and IG
>     > > > > > >     > > >     >     >         >> > Index Limited (register
> number
>     > > > > 114059)
>     > > > > > > are
>     > > > > > >     > > > authorised
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     > regulated by
>     > > > > > >     > > >     >     >         >> the
>     > > > > > >     > > >     >     >         >> > Financial Conduct
> Authority.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> --
>     > > > > > >     > > >     >     >         >> -Regards,
>     > > > > > >     > > >     >     >         >> Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         >> (862) 250-7125
>     > > > > > >     > > >     >     >         >> The information contained
> in
>     > this
>     > > > > email
>     > > > > > is
>     > > > > > >     > > strictly
>     > > > > > >     > > >     >     > confidential and for
>     > > > > > >     > > >     >     >         >> the use of the addressee
> only,
>     > > > unless
>     > > > > > > otherwise
>     > > > > > >     > > >     > indicated. If
>     > > > > > >     > > >     >     > you are not
>     > > > > > >     > > >     >     >         >> the intended recipient,
> please
>     > do
>     > > > not
>     > > > > > > read,
>     > > > > > >     > copy,
>     > > > > > >     > > > use or
>     > > > > > >     > > >     >     > disclose to others
>     > > > > > >     > > >     >     >         >> this message or any
> attachment.
>     > > > Please
>     > > > > > > also
>     > > > > > >     > notify
>     > > > > > >     > > > the
>     > > > > > >     > > >     > sender
>     > > > > > >     > > >     >     > by replying
>     > > > > > >     > > >     >     >         >> to this email or by
> telephone
>     > > > (+44(020
>     > > > > > > 7896
>     > > > > > >     > 0011)
>     > > > > > >     > > > and then
>     > > > > > >     > > >     >     > delete the email
>     > > > > > >     > > >     >     >         >> and any copies of it.
> Opinions,
>     > > > > > > conclusion (etc)
>     > > > > > >     > > > that do
>     > > > > > >     > > >     > not
>     > > > > > >     > > >     >     > relate to the
>     > > > > > >     > > >     >     >         >> official business of this
>     > company
>     > > > > shall
>     > > > > > be
>     > > > > > >     > > > understood as
>     > > > > > >     > > >     >     > neither given nor
>     > > > > > >     > > >     >     >         >> endorsed by it. IG is a
> trading
>     > > name
>     > > > > of
>     > > > > > IG
>     > > > > > >     > Markets
>     > > > > > >     > > >     > Limited (a
>     > > > > > >     > > >     >     > company
>     > > > > > >     > > >     >     >         >> registered in England and
> Wales,
>     > > > > company
>     > > > > > > number
>     > > > > > >     > > > 04008957)
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     > IG Index
>     > > > > > >     > > >     >     >         >> Limited (a company
> registered in
>     > > > > England
>     > > > > > > and
>     > > > > > >     > > Wales,
>     > > > > > >     > > >     > company
>     > > > > > >     > > >     >     > number
>     > > > > > >     > > >     >     >         >> 01190902). Registered
> address at
>     > > > > Cannon
>     > > > > > > Bridge
>     > > > > > >     > > > House, 25
>     > > > > > >     > > >     >     > Dowgate Hill,
>     > > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> Markets
>     > > > > Limited
>     > > > > > >     > (register
>     > > > > > >     > > > number
>     > > > > > >     > > >     >     > 195355) and IG
>     > > > > > >     > > >     >     >         >> Index Limited (register
> number
>     > > > 114059)
>     > > > > > are
>     > > > > > >     > > > authorised and
>     > > > > > >     > > >     >     > regulated by the
>     > > > > > >     > > >     >     >         >> Financial Conduct
> Authority.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > --
>     > > > > > >     > > >     >     >         > -Regards,
>     > > > > > >     > > >     >     >         > Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         > (862) 250-7125
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         --
>     > > > > > >     > > >     >     >         -Regards,
>     > > > > > >     > > >     >     >         Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         (862) 250-7125
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     The information contained in
> this email
>     > > is
>     > > > > > > strictly
>     > > > > > >     > > > confidential
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     > for the use of the addressee only,
> unless
>     > > > > otherwise
>     > > > > > >     > > indicated.
>     > > > > > >     > > > If
>     > > > > > >     > > >     > you are
>     > > > > > >     > > >     >     > not the intended recipient, please
> do not
>     > > read,
>     > > > > > > copy, use
>     > > > > > >     > or
>     > > > > > >     > > >     > disclose to
>     > > > > > >     > > >     >     > others this message or any
> attachment.
>     > Please
>     > > > > also
>     > > > > > > notify
>     > > > > > >     > the
>     > > > > > >     > > > sender
>     > > > > > >     > > >     > by
>     > > > > > >     > > >     >     > replying to this email or by
> telephone
>     > > (+44(020
>     > > > > > 7896
>     > > > > > > 0011)
>     > > > > > >     > > and
>     > > > > > >     > > > then
>     > > > > > >     > > >     > delete
>     > > > > > >     > > >     >     > the email and any copies of it.
> Opinions,
>     > > > > > conclusion
>     > > > > > > (etc)
>     > > > > > >     > > > that do
>     > > > > > >     > > >     > not
>     > > > > > >     > > >     >     > relate to the official business of
> this
>     > > company
>     > > > > > > shall be
>     > > > > > >     > > > understood
>     > > > > > >     > > >     > as
>     > > > > > >     > > >     >     > neither given nor endorsed by it. IG
> is a
>     > > > trading
>     > > > > > > name of
>     > > > > > >     > IG
>     > > > > > >     > > > Markets
>     > > > > > >     > > >     >     > Limited (a company registered in
> England
>     > and
>     > > > > Wales,
>     > > > > > > company
>     > > > > > >     > > > number
>     > > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> company
>     > > > > > registered
>     > > > > > > in
>     > > > > > >     > > > England and
>     > > > > > >     > > >     > Wales,
>     > > > > > >     > > >     >     > company number 01190902). Registered
>     > address
>     > > at
>     > > > > > > Cannon
>     > > > > > >     > Bridge
>     > > > > > >     > > > House,
>     > > > > > >     > > >     > 25
>     > > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both
> IG
>     > > Markets
>     > > > > > > Limited
>     > > > > > >     > > > (register
>     > > > > > >     > > >     > number
>     > > > > > >     > > >     >     > 195355) and IG Index Limited
> (register
>     > number
>     > > > > > > 114059) are
>     > > > > > >     > > > authorised
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     > regulated by the Financial Conduct
>     > Authority.
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > The information contained in this email is
>     > strictly
>     > > > > > > confidential
>     > > > > > >     > > and
>     > > > > > >     > > > for
>     > > > > > >     > > >     > the use of the addressee only, unless
> otherwise
>     > > > > > indicated.
>     > > > > > > If you
>     > > > > > >     > > > are not
>     > > > > > >     > > >     > the intended recipient, please do not
> read, copy,
>     > > use
>     > > > > or
>     > > > > > > disclose
>     > > > > > >     > > to
>     > > > > > >     > > > others
>     > > > > > >     > > >     > this message or any attachment. Please also
>     > notify
>     > > > the
>     > > > > > > sender by
>     > > > > > >     > > > replying
>     > > > > > >     > > >     > to this email or by telephone (+44(020
> 7896 0011
>     > > <020%207896%200011>) and
>     > > > > > then
>     > > > > > > delete
>     > > > > > >     > > > the email
>     > > > > > >     > > >     > and any copies of it. Opinions, conclusion
> (etc)
>     > > that
>     > > > > do
>     > > > > > > not
>     > > > > > >     > relate
>     > > > > > >     > > > to the
>     > > > > > >     > > >     > official business of this company shall be
>     > > understood
>     > > > > as
>     > > > > > > neither
>     > > > > > >     > > > given nor
>     > > > > > >     > > >     > endorsed by it. IG is a trading name of IG
>     > Markets
>     > > > > > Limited
>     > > > > > > (a
>     > > > > > >     > > company
>     > > > > > >     > > >     > registered in England and Wales, company
> number
>     > > > > 04008957)
>     > > > > > > and IG
>     > > > > > >     > > > Index
>     > > > > > >     > > >     > Limited (a company registered in England
> and
>     > Wales,
>     > > > > > company
>     > > > > > >     > number
>     > > > > > >     > > >     > 01190902). Registered address at Cannon
> Bridge
>     > > House,
>     > > > > 25
>     > > > > > > Dowgate
>     > > > > > >     > > > Hill,
>     > > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
>     > (register
>     > > > > number
>     > > > > > > 195355)
>     > > > > > >     > > > and IG
>     > > > > > >     > > >     > Index Limited (register number 114059) are
>     > > authorised
>     > > > > and
>     > > > > > >     > regulated
>     > > > > > >     > > > by the
>     > > > > > >     > > >     > Financial Conduct Authority.
>     > > > > > >     > > >     >
>     > > > > > >     > > >
>     > > > > > >     > > >
>     > > > > > >     > > > The information contained in this email is
> strictly
>     > > > > > confidential
>     > > > > > > and
>     > > > > > >     > for
>     > > > > > >     > > > the use of the addressee only, unless otherwise
>     > > indicated.
>     > > > If
>     > > > > > > you are
>     > > > > > >     > not
>     > > > > > >     > > > the intended recipient, please do not read,
> copy, use
>     > or
>     > > > > > > disclose to
>     > > > > > >     > > others
>     > > > > > >     > > > this message or any attachment. Please also
> notify the
>     > > > sender
>     > > > > > by
>     > > > > > >     > replying
>     > > > > > >     > > > to this email or by telephone (+44(020 7896 0011
>     > > <020%207896%200011>) and then
>     > > > > > > delete the
>     > > > > > >     > > email
>     > > > > > >     > > > and any copies of it. Opinions, conclusion (etc)
> that
>     > do
>     > > > not
>     > > > > > > relate to
>     > > > > > >     > > the
>     > > > > > >     > > > official business of this company shall be
> understood
>     > as
>     > > > > > neither
>     > > > > > > given
>     > > > > > >     > > nor
>     > > > > > >     > > > endorsed by it. IG is a trading name of IG
> Markets
>     > > Limited
>     > > > (a
>     > > > > > > company
>     > > > > > >     > > > registered in England and Wales, company number
>     > 04008957)
>     > > > and
>     > > > > > IG
>     > > > > > > Index
>     > > > > > >     > > > Limited (a company registered in England and
> Wales,
>     > > company
>     > > > > > > number
>     > > > > > >     > > > 01190902). Registered address at Cannon Bridge
> House,
>     > 25
>     > > > > > Dowgate
>     > > > > > > Hill,
>     > > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
> (register
>     > number
>     > > > > > > 195355) and
>     > > > > > >     > IG
>     > > > > > >     > > > Index Limited (register number 114059) are
> authorised
>     > and
>     > > > > > > regulated by
>     > > > > > >     > > the
>     > > > > > >     > > > Financial Conduct Authority.
>     > > > > > >     > > >
>     > > > > > >     > > The information contained in this email is strictly
>     > > > > confidential
>     > > > > > > and for
>     > > > > > >     > > the use of the addressee only, unless otherwise
>     > indicated.
>     > > If
>     > > > > you
>     > > > > > > are not
>     > > > > > >     > > the intended recipient, please do not read, copy,
> use or
>     > > > > disclose
>     > > > > > > to
>     > > > > > >     > others
>     > > > > > >     > > this message or any attachment. Please also notify
> the
>     > > sender
>     > > > > by
>     > > > > > > replying
>     > > > > > >     > > to this email or by telephone (+44(020 7896 0011
>     > > <020%207896%200011>) and then
>     > > > > delete
>     > > > > > > the
>     > > > > > >     > email
>     > > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> that do
>     > > not
>     > > > > > > relate to
>     > > > > > >     > the
>     > > > > > >     > > official business of this company shall be
> understood as
>     > > > > neither
>     > > > > > > given
>     > > > > > >     > nor
>     > > > > > >     > > endorsed by it. IG is a trading name of IG Markets
>     > Limited
>     > > (a
>     > > > > > > company
>     > > > > > >     > > registered in England and Wales, company number
> 04008957)
>     > > and
>     > > > > IG
>     > > > > > > Index
>     > > > > > >     > > Limited (a company registered in England and Wales,
>     > company
>     > > > > > number
>     > > > > > >     > > 01190902). Registered address at Cannon Bridge
> House, 25
>     > > > > Dowgate
>     > > > > > > Hill,
>     > > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > > > 195355)
>     > > > > > > and IG
>     > > > > > >     > > Index Limited (register number 114059) are
> authorised and
>     > > > > > > regulated by
>     > > > > > >     > the
>     > > > > > >     > > Financial Conduct Authority.
>     > > > > > >     > >
>     > > > > > >     > >
>     > > > > > >     > The information contained in this email is strictly
>     > > > confidential
>     > > > > > and
>     > > > > > > for
>     > > > > > >     > the use of the addressee only, unless otherwise
> indicated.
>     > If
>     > > > you
>     > > > > > > are not
>     > > > > > >     > the intended recipient, please do not read, copy,
> use or
>     > > > disclose
>     > > > > > to
>     > > > > > > others
>     > > > > > >     > this message or any attachment. Please also notify
> the
>     > sender
>     > > > by
>     > > > > > > replying
>     > > > > > >     > to this email or by telephone (+44(020 7896 0011
>     > > <020%207896%200011>) and then
>     > > > delete
>     > > > > > > the email
>     > > > > > >     > and any copies of it. Opinions, conclusion (etc)
> that do
>     > not
>     > > > > relate
>     > > > > > > to the
>     > > > > > >     > official business of this company shall be
> understood as
>     > > > neither
>     > > > > > > given nor
>     > > > > > >     > endorsed by it. IG is a trading name of IG Markets
> Limited
>     > (a
>     > > > > > company
>     > > > > > >     > registered in England and Wales, company number
> 04008957)
>     > and
>     > > > IG
>     > > > > > > Index
>     > > > > > >     > Limited (a company registered in England and Wales,
> company
>     > > > > number
>     > > > > > >     > 01190902). Registered address at Cannon Bridge
> House, 25
>     > > > Dowgate
>     > > > > > > Hill,
>     > > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > > 195355)
>     > > > > > > and IG
>     > > > > > >     > Index Limited (register number 114059) are
> authorised and
>     > > > > regulated
>     > > > > > > by the
>     > > > > > >     > Financial Conduct Authority.
>     > > > > > >     >
>     > > > > > >     >
>     > > > > > >
>     > > > > > >
>     > > > > > > The information contained in this email is strictly
> confidential
>     > > and
>     > > > > for
>     > > > > > > the use of the addressee only, unless otherwise indicated.
> If you
>     > > are
>     > > > > not
>     > > > > > > the intended recipient, please do not read, copy, use or
> disclose
>     > > to
>     > > > > > others
>     > > > > > > this message or any attachment. Please also notify the
> sender by
>     > > > > replying
>     > > > > > > to this email or by telephone (+44(020 7896 0011
>     > > <020%207896%200011>) and then delete the
>     > > > > > email
>     > > > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
>     > relate
>     > > > to
>     > > > > > the
>     > > > > > > official business of this company shall be understood as
> neither
>     > > > given
>     > > > > > nor
>     > > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
>     > > company
>     > > > > > > registered in England and Wales, company number 04008957)
> and IG
>     > > > Index
>     > > > > > > Limited (a company registered in England and Wales, company
>     > number
>     > > > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > > > Hill,
>     > > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
>     > > and
>     > > > > IG
>     > > > > > > Index Limited (register number 114059) are authorised and
>     > regulated
>     > > > by
>     > > > > > the
>     > > > > > > Financial Conduct Authority.
>     > > > > > >
>     > > > > > The information contained in this email is strictly
> confidential
>     > and
>     > > > for
>     > > > > > the use of the addressee only, unless otherwise indicated.
> If you
>     > are
>     > > > not
>     > > > > > the intended recipient, please do not read, copy, use or
> disclose
>     > to
>     > > > > others
>     > > > > > this message or any attachment. Please also notify the
> sender by
>     > > > replying
>     > > > > > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>)
>     > > and then delete the
>     > > > > email
>     > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > > to
>     > > > > the
>     > > > > > official business of this company shall be understood as
> neither
>     > > given
>     > > > > nor
>     > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>     > company
>     > > > > > registered in England and Wales, company number 04008957)
> and IG
>     > > Index
>     > > > > > Limited (a company registered in England and Wales, company
> number
>     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > > Hill,
>     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
>     > and
>     > > > IG
>     > > > > > Index Limited (register number 114059) are authorised and
> regulated
>     > > by
>     > > > > the
>     > > > > > Financial Conduct Authority.
>     > > > > >
>     > > > > The information contained in this email is strictly
> confidential and
>     > > for
>     > > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > > not
>     > > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > > others
>     > > > > this message or any attachment. Please also notify the sender
> by
>     > > replying
>     > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > > and then delete the
>     > > > email
>     > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > to
>     > > > the
>     > > > > official business of this company shall be understood as
> neither
>     > given
>     > > > nor
>     > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > > registered in England and Wales, company number 04008957) and
> IG
>     > Index
>     > > > > Limited (a company registered in England and Wales, company
> number
>     > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > Hill,
>     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > > IG
>     > > > > Index Limited (register number 114059) are authorised and
> regulated
>     > by
>     > > > the
>     > > > > Financial Conduct Authority.
>     > > > >
>     > > > The information contained in this email is strictly confidential
> and
>     > for
>     > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > not
>     > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > others
>     > > > this message or any attachment. Please also notify the sender by
>     > replying
>     > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > > and then delete the email
>     > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > > the
>     > > > official business of this company shall be understood as neither
> given
>     > > nor
>     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > > Limited (a company registered in England and Wales, company
> number
>     > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > IG
>     > > > Index Limited (register number 114059) are authorised and
> regulated by
>     > > the
>     > > > Financial Conduct Authority.
>     > > >
>     > >
>     > The information contained in this email is strictly confidential and
> for
>     > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > the intended recipient, please do not read, copy, use or disclose to
> others
>     > this message or any attachment. Please also notify the sender by
> replying
>     > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
>     > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
>     > official business of this company shall be understood as neither
> given nor
>     > endorsed by it. IG is a trading name of IG Markets Limited (a company
>     > registered in England and Wales, company number 04008957) and IG
> Index
>     > Limited (a company registered in England and Wales, company number
>     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > Index Limited (register number 114059) are authorised and regulated
> by the
>     > Financial Conduct Authority.
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Ismael Juma <is...@juma.me.uk>]

Hi Michael,

Yes, something along those lines. Although I'd probably have the public
constructor take a `boolean` and have a private constructor for the
`Boolean` (this way, we can retire the config at some point without
affecting the API exposed by the class). Also, I would just hardcode it in
`send` instead of using an interceptor.

It would be interesting to get some thoughts on whether people think this
is OK given the constraints.

A note: message format version bumps are relatively disruptive for users
(and they impose a non-trivial maintenance cost). So, as Jun said, it would
be good to avoid bumping it many times in rapid succession. The last bump
happened in 0.10.0.0, which was released in May 2016. My personal
preference would be to wait until the May 2017 release for the next message
format bump.

We have a number of KIPs currently being discussed that require a message
format bump: KIP-82, KIP-87, KIP-98 and KIP-101. Given that a lot of people
are going to be on holiday soon and taking into account the complexity of
the mentioned KIPs, it seems that we could, at best, get one of them in for
the January release. And it seems suboptimal to bump the message format
version for just one KIP when we know we have three others lined up. I hope
that explains my reasoning. Does it make sense at all?

Ismael



On Fri, Dec 16, 2016 at 1:53 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Ismael
>
>
> So I understand what is being suggested, is we allow on the ProducerRecord
> at time of construction to be nullable aka big B, Boolean, where old
> constructors set it to null.
>
>
>     /**
>      * Creates a record with a specified timestamp to be sent to a
> specified topic and partition
>      *
>      * @param topic The topic the record will be appended to
>      * @param partition The partition to which the record should be sent
>      * @param tombstone if the record is a tombstone marker.
>      * @param timestamp The timestamp of the record
>      * @param key The key that will be included in the record
>      * @param value The record contents
>      */
>     public ProducerRecord(String topic, Integer partition, Boolean
> tombstone, Long timestamp, K key, V value) {
>         if (topic == null)
>             throw new IllegalArgumentException("Topic cannot be null");
>         if (timestamp != null && timestamp < 0)
>             throw new IllegalArgumentException("Invalid timestamp " +
> timestamp);
>         this.topic = topic;
>         this.partition = partition;
>         this.tombstone = tombstone;
>         this.key = key;
>         this.value = value;
>         this.timestamp = timestamp;
>     }
>
>     public ProducerRecord(String topic, Integer partition, Long timestamp,
> K key, V value)
>     {
>         this(topic, partition, null, timestamp, key, value);
>     }
>
>
> Then within the KafkaProducer an enforced/defaulter interceptor or simply
> hardcoded in the on send if it detects the tombstone is null, it sets the
> tombstone to true or false based on a configuration property of the
> KafkaProducer?
>
> Aka in this method, or by adding a default interceptor ->
>
>     @Override
>     public Future<RecordMetadata> send(ProducerRecord<K, V> record,
> Callback callback) {
>         // intercept the record, which can be potentially modified; this
> method does not throw exceptions
>         ProducerRecord<K, V> interceptedRecord = this.interceptors == null
> ? record : this.interceptors.onSend(record);
>         return doSend(interceptedRecord, callback);
>     }
>
> e.g sample default interceptor method we can either just add this to the
> interceptors by default or code this same logic in the send/doSend logic.
>
> public class TombstoneInterceptor implements ProducerInterceptor
> {
>    boolean defaultNullValueTombstoneMarker;
>
>    @Override
>    public void configure(Map<String, ?> configs)
>    {
>       defaultNullValueTombstoneMarker = (Boolean)configs.get("
> tombstone.default.null.value");
>    }
>
>    @Override
>    public ProducerRecord onSend(ProducerRecord record)
>    {
>       if (record.tombstone() == null){
>          boolean tombstone = false;
>          if (record.value() == null){
>             tombstone = defaultNullValueTombstoneMarker;
>          }
>          return new ProducerRecord(record.topic(), record.partition(),
> tombstone, record.timestamp(), record.key(), record.value());
>       }
>       return record;
>    }
>
>    @Override
>    public void onAcknowledgement(RecordMetadata metadata, Exception
> exception)
>    {
>
>    }
>
>    @Override
>    public void close()
>    {
>
>    }
> }
>
> Is this a correct understanding?
>
> Cheers
> Mike
>
> On 16/12/2016, 21:16, "ismaelj@gmail.com on behalf of Ismael Juma" <
> ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
>
>     Hi Michael,
>
>     I am not sure about using a new policy. One other option (suggested by
>     Gwen) is that the behaviour of the producer with regards to tombstones
> when
>     you pass a `null` value and don't use the constructor that takes a
>     tombstone can be influenced by a producer config. Not ideal, but would
>     limit the complexity of maintaining compatibility while providing a
> path
>     for the (what I think) are the right semantics.
>
>     Ismael
>
>     On Tue, Dec 13, 2016 at 8:32 AM, Michael Pearce <Michael.Pearce@ig.com
> >
>     wrote:
>
>     > Hi Ismael
>     >
>     > Did you see our email this morning, what's your thoughts on this
> approach
>     > to instead we simply have a brand new policy?
>     >
>     > Cheers
>     > Mike
>     >
>     >
>     > Sent using OWA for iPhone
>     > ________________________________________
>     > From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael
> Juma <
>     > ismael@juma.me.uk>
>     > Sent: Tuesday, December 13, 2016 11:30:05 AM
>     > To: dev@kafka.apache.org
>     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     >
>     > Yes, this is actually tricky to do in a way where we both have the
> desired
>     > semantics and maintain compatibility. When someone creates a
>     > `ProducerRecord` with a `null` value today, the producer doesn't
> know if
>     > it's meant to be a tombstone or not. For V3 messages, it's easy when
> the
>     > constructor that takes a tombstone is used. However, if any other
>     > constructor is used, it's not clear. A couple of options I can think
> of,
>     > none of them particularly nice:
>     >
>     > 1. Have a third state where tombstone = unknown and the broker would
> set
>     > the tombstone bit if the value was null and the topic was compacted.
> People
>     > that wanted to pass a non-null value for the tombstone would have to
> use
>     > the constructor that takes a tombstone. The drawbacks: third state
> for
>     > tombstone in message format, message conversion at the broker for a
> common
>     > case.
>     >
>     > 2. Extend MetadataResponse to optionally include topic configs,
> which would
>     > make it possible for the producer to be smarter about setting the
>     > tombstone. It would only do it if a tombstone was not passed
> explicitly,
>     > the value was null and the topic was compacted. The main drawback is
> that
>     > the producer would be getting a bit more data for each topic even
> though it
>     > probably won't use it most of the time. Extending MetadataResponse to
>     > return topic configs would be useful for other reasons as well, so
> that
>     > part seems OK.
>     >
>     > In addition, for both proposals, we could consider adding warnings
> to the
>     > documentation that the behaviour of the constructors that don't take
> a
>     > tombstone would change in the next major release so that tombstone =
> false.
>     > Not sure if this would be worth it though.
>     >
>     > Ismael
>     >
>     > On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
> ewen@confluent.io
>     > >
>     > wrote:
>     >
>     > > Michael,
>     > >
>     > > It kind of depends on how you want to interpret the tombstone
> flag. If
>     > it's
>     > > purely a producer-facing Kafka-level thing that we treat as
> internal to
>     > the
>     > > broker and log cleaner once the record is sent, then your approach
> makes
>     > > sense. You're just moving copying the null-indicates-delete
> behavior of
>     > the
>     > > old constructor into the tombstone flag.
>     > >
>     > > However, if you want this change to more generally decouple the
> idea of
>     > > deletion and null values, then you are sometimes converting what
> might
>     > be a
>     > > completely valid null value that doesn't indicate deletion into a
>     > > tombstone. Downstream applications could potentially handle these
> cases
>     > > differently given the separation of deletion from value.
>     > >
>     > > I guess the question is if we want to try to support the latter
> even for
>     > > topics where we have older produce requests. An example where this
> could
>     > > come up is in something like a CDC Connector. If we try to support
> the
>     > > semantic difference, a connector might write changes to Kafka
> using the
>     > > tombstone flag to indicate when a row was truly deleted (vs an
> update
>     > that
>     > > sets it to null but still present; this probably makes more sense
> for CDC
>     > > from document stores or extracting single columns). There are
> various
>     > > reasons we might want to maintain the full log and not turn
> compaction on
>     > > (or just use a time-based retention policy), but downstream
> applications
>     > > might care to know the difference between a delete and a null
> value. In
>     > > fact both versions of the same log (compacted and time-retention)
> could
>     > be
>     > > useful and I don't think it'll be uncommon to maintain both or use
> KIP-71
>     > > to maintain a hybrid compacted/retention topic.
>     > >
>     > > -Ewen
>     > >
>     > > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > > wrote:
>     > >
>     > > > Hi Jay,
>     > > >
>     > > > Why wouldn't that work, the tombstone value is only looked at by
> the
>     > > > broker, on a topic configured for compaction as such is benign
> on non
>     > > > compacted topics. This is as much as sending a null value
> currently
>     > > >
>     > > >
>     > > > Regards
>     > > > Mike
>     > > >
>     > > >
>     > > >
>     > > > Sent using OWA for iPhone
>     > > > ________________________________________
>     > > > From: Jay Kreps <ja...@confluent.io>
>     > > > Sent: Sunday, December 11, 2016 8:58:53 PM
>     > > > To: dev@kafka.apache.org
>     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > >
>     > > > Hey Michael,
>     > > >
>     > > > I'm not quite sure that works as that would translate ALL null
> values
>     > to
>     > > > tombstones, even for non-compacted topics that use null as an
>     > acceptable
>     > > > value sent by the producer and expected by the consumer.
>     > > >
>     > > > -Jay
>     > > >
>     > > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
> Michael.Pearce@ig.com
>     > >
>     > > > wrote:
>     > > >
>     > > > > Hi Ewen,
>     > > > >
>     > > > > I think the easiest way to show this is with code.
>     > > > >
>     > > > > As you can see we keep the existing behaviour for code/binaries
>     > calling
>     > > > > the pre-existing constructors, whereby if the value is null the
>     > > tombstone
>     > > > > is set to true.
>     > > > >
>     > > > > Regards
>     > > > > Mike
>     > > > >
>     > > > >
>     > > > >
>     > > > >     /**
>     > > > >      * Creates a record with a specified timestamp to be sent
> to a
>     > > > > specified topic and partition
>     > > > >      *
>     > > > >      * @param topic The topic the record will be appended to
>     > > > >      * @param partition The partition to which the record
> should be
>     > > sent
>     > > > >      * @param timestamp The timestamp of the record
>     > > > >      * @param tombstone if the record should be treated as a
>     > tombstone
>     > > if
>     > > > > the topic is compacted
>     > > > >      * @param key The key that will be included in the record
>     > > > >      * @param value The record contents
>     > > > >      */
>     > > > >     public ProducerRecord(String topic, Integer partition,
> Boolean
>     > > > > tombstone, Long timestamp, K key, V value) {
>     > > > >         if (topic == null)
>     > > > >             throw new IllegalArgumentException("Topic cannot
> be
>     > > null.");
>     > > > >         if (timestamp != null && timestamp < 0)
>     > > > >             throw new IllegalArgumentException(
>     > > > >                     String.format("Invalid timestamp: %d.
> Timestamp
>     > > > should
>     > > > > always be non-negative or null.", timestamp));
>     > > > >         if (partition != null && partition < 0)
>     > > > >             throw new IllegalArgumentException(
>     > > > >                     String.format("Invalid partition: %d.
> Partition
>     > > > number
>     > > > > should always be non-negative or null.", partition));
>     > > > >         if (!tombstone && value == null){
>     > > > >             throw new IllegalArgumentException(
>     > > > >                     String.format("Tombstone must be true if
> null
>     > > > value"));
>     > > > >         }
>     > > > >         this.topic = topic;
>     > > > >         this.partition = partition;
>     > > > >         this.tombstone = tombstone;
>     > > > >         this.key = key;
>     > > > >         this.value = value;
>     > > > >         this.timestamp = timestamp;
>     > > > >     }
>     > > > >
>     > > > >     public ProducerRecord(String topic, Integer partition, Long
>     > > > timestamp,
>     > > > > K key, V value) {
>     > > > >         this(topic, partition, value == null, timestamp, key,
> value);
>     > > > >     }
>     > > > > ________________________________________
>     > > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
>     > > > > Sent: Sunday, December 11, 2016 5:45 AM
>     > > > > To: dev@kafka.apache.org
>     > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > > >
>     > > > > It seemed like this was addressed in the migration section,
> wasn't
>     > it?
>     > > > The
>     > > > > V2 vs V3 requests and the fact that the broker will downgrade
> the
>     > > message
>     > > > > format (losing zero copy) if you issues a V2 request to a
> broker
>     > using
>     > > V3
>     > > > > format handles compatibility, does it not? The existing
> consumers
>     > won't
>     > > > see
>     > > > > the extra metadata in the value, but they will get a null
> instead and
>     > > > treat
>     > > > > it as a tombstone. Certainly there is a performance impact,
> but it
>     > > seemed
>     > > > > compatible.
>     > > > >
>     > > > > I'm worried about this though:
>     > > > >
>     > > > >
>     > > > >    - *NOTE *: With the new version of producer client using
>     > > > ProduceRequest
>     > > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not
> set) and
>     > > null
>     > > > > value
>     > > > >    should not be allowed as the older version of consumer using
>     > > > > FetchRequest
>     > > > >    V2 will think of this as a tombstone when its actually not.
>     > > > >
>     > > > >
>     > > > > Unless I'm misunderstanding, this ends up breaking binary
>     > compatibility
>     > > > for
>     > > > > the Java API. It sounds like this suggests that passing a null
> value
>     > to
>     > > > the
>     > > > > existing ProducerRecord constructors would generate an
> exception
>     > since
>     > > > you
>     > > > > didn't explicitly enable the tombstone (via whatever new
> constructor
>     > is
>     > > > > provided). But that means you can't swap in a newer client jar
>     > without
>     > > > > recompiling and get the same behavior if your app deletes keys
> using
>     > > the
>     > > > > current approach because your app will start throwing
> exceptions.
>     > Maybe
>     > > > > this is a tradeoff we're ok with, but we've tried pretty hard
> to
>     > avoid
>     > > > > breaking compatibility like this so far -- it makes picking up
> bug
>     > > fixes
>     > > > in
>     > > > > the clients harder for users of frameworks that have to pin to
>     > earlier
>     > > > > default versions for compatibility.
>     > > > >
>     > > > > But then later the KIP says:
>     > > > >
>     > > > >
>     > > > >    - The old constructors for ProducerRecord without this
> parameter
>     > > will
>     > > > be
>     > > > >    kept but updated so that their default behaviour if setting
> null
>     > > value
>     > > > > will
>     > > > >    be the tombstone will be set to true to keep existing
> behaviour.
>     > > > >
>     > > > >
>     > > > > So maybe I am misinterpreting something.
>     > > > >
>     > > > > And just a nit re: motivation section, item 6 would be clearer
> for a
>     > > > union
>     > > > > schema with null (or optional schemas in other formats), e.g.
> [null,
>     > > > > string], in which case losing the schema truly is losing
> information
>     > > > > (whereas null is already the only valid value for a pure null
>     > schema).
>     > > > >
>     > > > > -Ewen
>     > > > >
>     > > > >
>     > > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
>     > Michael.Pearce@ig.com
>     > > >
>     > > > > wrote:
>     > > > >
>     > > > > > Hi Jay,
>     > > > > >
>     > > > > > Good point this detail is missing in the KIP write up. Ive
> added
>     > this
>     > > > > now.
>     > > > > >
>     > > > > > Essentially simply just upgrading the clients we do not
> expect any
>     > > > client
>     > > > > > app code change needed.
>     > > > > >
>     > > > > > Cheers
>     > > > > > Mike
>     > > > > > ________________________________________
>     > > > > > From: Jay Kreps <ja...@confluent.io>
>     > > > > > Sent: Saturday, December 10, 2016 10:51 PM
>     > > > > > To: dev@kafka.apache.org
>     > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > > > >
>     > > > > > Michael,
>     > > > > >
>     > > > > > The compatibility section goes through the migration path,
> but
>     > isn't
>     > > > the
>     > > > > > bigger compatibility issue with existing apps? There are many
>     > > (probably
>     > > > > > thousands) of apps in production that use this feature and
> send
>     > null
>     > > to
>     > > > > > mean delete. It seems like this would break compatibility
> with
>     > them,
>     > > > and
>     > > > > > they would have to be rewritten to right?
>     > > > > >
>     > > > > > -Jay
>     > > > > >
>     > > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
>     > > Michael.Pearce@ig.com
>     > > > >
>     > > > > > wrote:
>     > > > > >
>     > > > > > > Hi Jun,
>     > > > > > >
>     > > > > > > 4) On v3 we honour the tombstone. As such we expect it to
> be set
>     > > > > > correctly
>     > > > > > > as per the KIP.
>     > > > > > >
>     > > > > > > 4.1) why would we want to produce an error when its v3?
> This is
>     > the
>     > > > > exact
>     > > > > > > purpose to support non-null tombstone’s
>     > > > > > > 4.2) again here im unclear on the question on the v3,
> produce
>     > > request
>     > > > > we
>     > > > > > > expect the tombstone flag to be set correctly.
>     > > > > > >
>     > > > > > > 4.4) compaction only occurs on compacted topics, the bit
> makes no
>     > > > > > > difference and not looked at on un-compacted (time/size
> based
>     > > > > eviction).
>     > > > > > >
>     > > > > > >
>     > > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
>     > > > > > >
>     > > > > > >     Hi, Michael,
>     > > > > > >
>     > > > > > >     4. Then, I think I misunderstood this point. Could you
>     > document
>     > > > the
>     > > > > > >     following points in the wiki?
>     > > > > > >     4.1 If producer V3 sets tombstone, but provides a
> non-null
>     > > value,
>     > > > > > does
>     > > > > > > the
>     > > > > > >     send() get an error or does the producer automatically
> set
>     > the
>     > > > > value
>     > > > > > to
>     > > > > > >     null?
>     > > > > > >     4.2 If producer V3 doesn't set tombstone, but provides
> a null
>     > > > > value,
>     > > > > > > does
>     > > > > > >     the send() get an error or does the producer
> automatically
>     > sets
>     > > > the
>     > > > > > >     tombstone?
>     > > > > > >     4.3 Does the broker only expect messages that (a) have
> no
>     > > > tombstone
>     > > > > > and
>     > > > > > >     non-null value; (b) have tombstone and null value and
> reject
>     > > the
>     > > > > > > messages
>     > > > > > >     with an error code otherwise?
>     > > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
>     > compacted
>     > > on
>     > > > > > not?
>     > > > > > >
>     > > > > > >     Thanks,
>     > > > > > >
>     > > > > > >     Jun
>     > > > > > >
>     > > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
>     > > > > > Michael.Pearce@ig.com
>     > > > > > > >
>     > > > > > >     wrote:
>     > > > > > >
>     > > > > > >     > Not at all.  This only acts on compacted topics just
> as
>     > what
>     > > > > occurs
>     > > > > > > today
>     > > > > > >     >
>     > > > > > >     > Sent using OWA for iPhone
>     > > > > > >     > ________________________________________
>     > > > > > >     > From: Jun Rao <ju...@confluent.io>
>     > > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
>     > > > > > >     > To: dev@kafka.apache.org
>     > > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone Flag
>     > > > > > >     >
>     > > > > > >     > Hi, Michael,
>     > > > > > >     >
>     > > > > > >     > 4. Hmm, does that mean the new client library can
> never
>     > send
>     > > a
>     > > > > null
>     > > > > > > message
>     > > > > > >     > even to a regular topic? This seems like a change of
> the
>     > > > existing
>     > > > > > > behavior.
>     > > > > > >     >
>     > > > > > >     > Thanks,
>     > > > > > >     >
>     > > > > > >     > Jun
>     > > > > > >     >
>     > > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
>     > > > > > > Michael.Pearce@ig.com>
>     > > > > > >     > wrote:
>     > > > > > >     >
>     > > > > > >     > > Hi Jun,
>     > > > > > >     > >
>     > > > > > >     > > Re 4) That's because we expect the tombstone value
> to be
>     > > set
>     > > > > > > correctly if
>     > > > > > >     > > message bit is 2, as such if an older client sends
> in on
>     > > old
>     > > > > > > message the
>     > > > > > >     > > message is upcast and the bit is set correctly.
> And such
>     > no
>     > > > > > longer
>     > > > > > > need
>     > > > > > >     > to
>     > > > > > >     > > check the value. Mayuresh can you confirm my
> thinking and
>     > > > > > > understanding
>     > > > > > >     > of
>     > > > > > >     > > what we've discussed?
>     > > > > > >     > >
>     > > > > > >     > > The second point I understand what you're getting
> at now
>     > my
>     > > > > > > apologies.
>     > > > > > >     > Yes
>     > > > > > >     > > this makes sense to save on touching the message,
> if
>     > we're
>     > > > the
>     > > > > > > only kip
>     > > > > > >     > > going in, in this release.
>     > > > > > >     > >
>     > > > > > >     > > Cheers
>     > > > > > >     > > Mike
>     > > > > > >     > >
>     > > > > > >     > > Sent using OWA for iPhone
>     > > > > > >     > > ________________________________________
>     > > > > > >     > > From: Jun Rao <ju...@confluent.io>
>     > > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>     > > > > > >     > > To: dev@kafka.apache.org
>     > > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone
>     > Flag
>     > > > > > >     > >
>     > > > > > >     > > Hi, Michael,
>     > > > > > >     > >
>     > > > > > >     > > 4. Is this updated in the wiki? The text "If the
> magic
>     > byte
>     > > > on
>     > > > > > > message is
>     > > > > > >     > > 2, the broker should use the tombstone bit for log
>     > > > compaction."
>     > > > > > > doesn't
>     > > > > > >     > > seem to have changed.
>     > > > > > >     > >
>     > > > > > >     > > 2. My point is that if we change the message
> format just
>     > > for
>     > > > > this
>     > > > > > > KIP, we
>     > > > > > >     > > should consider whether it's worth optimizing the
> down
>     > > > > conversion
>     > > > > > > path
>     > > > > > >     > > (i.e., decide whether a conversion is needed by
> just
>     > > looking
>     > > > at
>     > > > > > the
>     > > > > > >     > > tombstone bit in the wrapper message) since
> tombstone
>     > will
>     > > be
>     > > > > > used
>     > > > > > >     > rarely.
>     > > > > > >     > > However, if the message format change here is
> combined
>     > with
>     > > > > other
>     > > > > > > KIPs,
>     > > > > > >     > > then this optimization likely won't be needed. The
> latter
>     > > > > > probably
>     > > > > > > makes
>     > > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you
> think?
>     > > > > > >     > >
>     > > > > > >     > > Other than those, +1 from me,
>     > > > > > >     > >
>     > > > > > >     > > Thanks,
>     > > > > > >     > >
>     > > > > > >     > > Jun
>     > > > > > >     > >
>     > > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
>     > > > > > > Michael.Pearce@ig.com>
>     > > > > > >     > > wrote:
>     > > > > > >     > >
>     > > > > > >     > > > Hi Jun
>     > > > > > >     > > >
>     > > > > > >     > > > do we have your vote on this now?
>     > > > > > >     > > >
>     > > > > > >     > > > Any other concerns?
>     > > > > > >     > > >
>     > > > > > >     > > > Cheers
>     > > > > > >     > > > Mike
>     > > > > > >     > > >
>     > > > > > >     > > > Sent using OWA for iPhone
>     > > > > > >     > > > ________________________________________
>     > > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
>     > > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>     > > > > > >     > > > To: dev@kafka.apache.org
>     > > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
> Tombstone
>     > > Flag
>     > > > > > >     > > >
>     > > > > > >     > > > Hi Jun,
>     > > > > > >     > > >
>     > > > > > >     > > > Have updated. Thanks again for the feedback.
>     > > > > > >     > > >
>     > > > > > >     > > > Agree yes we should align up when it gets to
> that, I
>     > > assume
>     > > > > > > you’ve
>     > > > > > >     > > flagged
>     > > > > > >     > > > the same in the other KIP?
>     > > > > > >     > > >
>     > > > > > >     > > > I think for now let’s get this KIP approved,
> then we
>     > can
>     > > > > start
>     > > > > > > the work
>     > > > > > >     > > to
>     > > > > > >     > > > get an initial PR, then we can discuss how to
> align the
>     > > two
>     > > > > if
>     > > > > > > needed.
>     > > > > > >     > > >
>     > > > > > >     > > > Cheers,
>     > > > > > >     > > > Mike
>     > > > > > >     > > >
>     > > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <
> jun@confluent.io>
>     > > wrote:
>     > > > > > >     > > >
>     > > > > > >     > > >     Hi, Michael,
>     > > > > > >     > > >
>     > > > > > >     > > >     For 2), this is fine. Could you update the
> KIP wiki
>     > > to
>     > > > > make
>     > > > > > > this
>     > > > > > >     > and
>     > > > > > >     > > > other
>     > > > > > >     > > >     points clearer? Other than that, the KIP
> looks good
>     > > to
>     > > > > me.
>     > > > > > >     > > >
>     > > > > > >     > > >     An orthogonal thing is that there are other
> KIPs
>     > such
>     > > > as
>     > > > > > > KIP-98
>     > > > > > >     > that
>     > > > > > >     > > > also
>     > > > > > >     > > >     intend to change the message format. If they
> all
>     > get
>     > > > > > > approved, we
>     > > > > > >     > > > should
>     > > > > > >     > > >     think about whether it's better to just bump
> up the
>     > > > magic
>     > > > > > > byte once
>     > > > > > >     > > to
>     > > > > > >     > > >     incorporate multiple format changes like we
> did in
>     > > > > > > KIP-31/KIP-32.
>     > > > > > >     > > >
>     > > > > > >     > > >     Thanks,
>     > > > > > >     > > >
>     > > > > > >     > > >     Jun
>     > > > > > >     > > >
>     > > > > > >     > > >
>     > > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael
> Pearce <
>     > > > > > >     > > Michael.Pearce@ig.com>
>     > > > > > >     > > >     wrote:
>     > > > > > >     > > >
>     > > > > > >     > > >     > Hi Jao
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > Thanks for the response. Sorry for slow
> reply,
>     > both
>     > > > > with
>     > > > > > > personal
>     > > > > > >     > > > sickness
>     > > > > > >     > > >     > and also battling some critical issues
>     > encountered
>     > > > > since
>     > > > > > >     > upgrading
>     > > > > > >     > > to
>     > > > > > >     > > >     > 0.10.1.0
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > 1) Thans for spotting, Document error
> where we
>     > > > branched
>     > > > > > > this KIP
>     > > > > > >     > > from
>     > > > > > >     > > >     > KIP-82, will get that removed.
>     > > > > > >     > > >     > 2) Intent is to do this just at the record
>     > message
>     > > > > level.
>     > > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
>     > > > corrected.
>     > > > > > >     > > >     > 4) As per discussion thread we will support
>     > > > tombstone +
>     > > > > > > null
>     > > > > > >     > value,
>     > > > > > >     > > >     > tombstone + non null value, no tombstone +
> null
>     > > > value.
>     > > > > > >     > > >     > 5) I believe this was in the discussion
> thread,
>     > > > > @Mayuresh
>     > > > > > > is this
>     > > > > > >     > > >     > something we’ve overlooked? I thought we
> would
>     > down
>     > > > > > > convert and
>     > > > > > >     > > > remove the
>     > > > > > >     > > >     > value so the old consumer had existing
> behavior,
>     > or
>     > > > is
>     > > > > > > there
>     > > > > > >     > > > something we
>     > > > > > >     > > >     > haven’t thought about?
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > Cheers
>     > > > > > >     > > >     > Mike
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
>     > jun@confluent.io>
>     > > > > > wrote:
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     Hi, Michael,
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     Thanks for the KIP. A few comments
> below.
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     1. The message format change contains
>     > > > > "HeadersLength
>     > > > > > >     > Headers".
>     > > > > > >     > > > Is that
>     > > > > > >     > > >     >     intended?
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     2. For compressed messageset, is the
>     > tombstone
>     > > > bit
>     > > > > > > only set
>     > > > > > >     > at
>     > > > > > >     > > > the
>     > > > > > >     > > >     > shallow
>     > > > > > >     > > >     >     level? Do we always leave that bit in
> the
>     > > wrapper
>     > > > > > > message
>     > > > > > >     > > unset?
>     > > > > > >     > > > An
>     > > > > > >     > > >     >     alternative is to set the tombstone
> bit in
>     > the
>     > > > > > wrapper
>     > > > > > > if at
>     > > > > > >     > > > least one
>     > > > > > >     > > >     >     inner message has the tombstone bit
> set. This
>     > > > makes
>     > > > > > > things a
>     > > > > > >     > > bit
>     > > > > > >     > > > more
>     > > > > > >     > > >     >     complicated, but we could potentially
> exploit
>     > > > that
>     > > > > > for
>     > > > > > >     > > > optimizing down
>     > > > > > >     > > >     >     conversion. For example, we only need
> to
>     > > convert
>     > > > > > > messages
>     > > > > > >     > with
>     > > > > > >     > > > magic 2
>     > > > > > >     > > >     > to
>     > > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit
> is set
>     > > > > > > (conversion is
>     > > > > > >     > > > always
>     > > > > > >     > > >     > needed
>     > > > > > >     > > >     >     from magic 2 to magic 0). Not sure if
> the
>     > > > > > optimization
>     > > > > > > is
>     > > > > > >     > worth
>     > > > > > >     > > > the
>     > > > > > >     > > >     >     complexity though.
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     3. The referencing of the new version
> of
>     > > > > > >     > > > ProducerRequest/FetchRequest
>     > > > > > >     > > >     > is
>     > > > > > >     > > >     >     inconsistent (v4 vs v3). Since our
> convention
>     > > > > starts
>     > > > > > at
>     > > > > > >     > version
>     > > > > > >     > > > at 0, I
>     > > > > > >     > > >     >     think the new version would be 3.
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     4. "If the magic byte on message is 2,
> the
>     > > broker
>     > > > > > > should use
>     > > > > > >     > > the
>     > > > > > >     > > >     > tombstone
>     > > > > > >     > > >     >     bit for log compaction." What about
> null
>     > value?
>     > > > My
>     > > > > > >     > > understanding
>     > > > > > >     > > > is
>     > > > > > >     > > >     > that
>     > > > > > >     > > >     >     null value will be treated the same as
>     > setting
>     > > > the
>     > > > > > > tombstone
>     > > > > > >     > > bit.
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     5. For the migration path, it would be
> useful
>     > > to
>     > > > > > > describe the
>     > > > > > >     > > > down
>     > > > > > >     > > >     >     conversion path to consumers (i.e.,
> brokers
>     > on
>     > > > > > message
>     > > > > > > format
>     > > > > > >     > > > 0.10.2
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     consumers on older version).
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     Thanks,
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     Jun
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM,
> Michael
>     > > Pearce <
>     > > > > > >     > > > Michael.Pearce@ig.com
>     > > > > > >     > > >     > >
>     > > > > > >     > > >     >     wrote:
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >     > Hi All,
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > We have been discussing in the below
> thread
>     > > and
>     > > > > > final
>     > > > > > >     > changes
>     > > > > > >     > > > have
>     > > > > > >     > > >     > been
>     > > > > > >     > > >     >     > made to the KIP wiki based on these
>     > > > discussions.
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > We would now like to put to the vote
> the
>     > > > > following
>     > > > > > > KIP:
>     > > > > > >     > > >     >     > https://cwiki.apache.org/
>     > > > > > > confluence/display/KAFKA/KIP-
>     > > > > > >     > 87+-+
>     > > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > This kip is for having a distinct
>     > compaction
>     > > > > > > attribute
>     > > > > > >     > > > “tombstone”
>     > > > > > >     > > >     > flag
>     > > > > > >     > > >     >     > instead of relying on null value,
> allowing
>     > > > > non-null
>     > > > > > > value
>     > > > > > >     > > > delete
>     > > > > > >     > > >     > messages.
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > Many thanks,
>     > > > > > >     > > >     >     > Michael
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael
> Pearce" <
>     > > > > > >     > > Michael.Pearce@ig.com>
>     > > > > > >     > > >     > wrote:
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     Hi Mayuresh,
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     LGTM. Ive just made one small
>     > adjustment
>     > > > > > > updating the
>     > > > > > >     > > wire
>     > > > > > >     > > >     > protocol to
>     > > > > > >     > > >     >     > show the magic byte bump.
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     Do we think we’re good to put to
> a
>     > vote?
>     > > Is
>     > > > > > > there any
>     > > > > > >     > > > other bits
>     > > > > > >     > > >     >     > needing discussion?
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     Cheers
>     > > > > > >     > > >     >     >     Mike
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
>     > Gharat" <
>     > > > > > >     > > >     > gharatmayuresh15@gmail.com>
>     > > > > > >     > > >     >     > wrote:
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         Hi Michael,
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         I have updated the migration
>     > section
>     > > of
>     > > > > the
>     > > > > > > KIP.
>     > > > > > >     > Can
>     > > > > > >     > > > you
>     > > > > > >     > > >     > please
>     > > > > > >     > > >     >     > take a look?
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         Thanks,
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         Mayuresh
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07
> AM,
>     > > > Mayuresh
>     > > > > > > Gharat <
>     > > > > > >     > > >     >     > gharatmayuresh15@gmail.com
>     > > > > > >     > > >     >     >         > wrote:
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         > Hi Michael,
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > That whilst sending
> tombstone and
>     > > non
>     > > > > > null
>     > > > > > > value,
>     > > > > > >     > > the
>     > > > > > >     > > >     > consumer
>     > > > > > >     > > >     >     > can expect
>     > > > > > >     > > >     >     >         > only to receive the
> non-null
>     > > message
>     > > > > only
>     > > > > > > in step
>     > > > > > >     > > > (3) is
>     > > > > > >     > > >     > this
>     > > > > > >     > > >     >     > correct?
>     > > > > > >     > > >     >     >         > ---> I do agree with you
> here.
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > Becket, Ismael : can you
> guys
>     > > review
>     > > > > the
>     > > > > > >     > migration
>     > > > > > >     > > > plan
>     > > > > > >     > > >     > listed
>     > > > > > >     > > >     >     > above using
>     > > > > > >     > > >     >     >         > magic byte?
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > Thanks,
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > Mayuresh
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at
> 8:58 AM,
>     > > > > Michael
>     > > > > > > Pearce <
>     > > > > > >     > > >     >     > Michael.Pearce@ig.com>
>     > > > > > >     > > >     >     >         > wrote:
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         >> Many thanks for this
> Mayuresh. I
>     > > > don't
>     > > > > > > have any
>     > > > > > >     > > >     > objections.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> I assume we should state:
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> That whilst sending
> tombstone
>     > and
>     > > > non
>     > > > > > null
>     > > > > > >     > value,
>     > > > > > >     > > > the
>     > > > > > >     > > >     > consumer
>     > > > > > >     > > >     >     > can expect
>     > > > > > >     > > >     >     >         >> only to receive the
> non-null
>     > > message
>     > > > > > only
>     > > > > > > in
>     > > > > > >     > step
>     > > > > > >     > > > (3) is
>     > > > > > >     > > >     > this
>     > > > > > >     > > >     >     > correct?
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Cheers
>     > > > > > >     > > >     >     >         >> Mike
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Sent using OWA for iPhone
>     > > > > > >     > > >     >     >         >>
> ______________________________
>     > > > > > __________
>     > > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
>     > > > > > >     > gharatmayuresh15@gmail.com
>     > > > > > >     > > >
>     > > > > > >     > > >     >     >         >> Sent: Thursday, November
> 17,
>     > 2016
>     > > > > > 5:18:41
>     > > > > > > PM
>     > > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
>     > > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS]
> KIP-87 -
>     > > Add
>     > > > > > > Compaction
>     > > > > > >     > > > Tombstone
>     > > > > > >     > > >     > Flag
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Hi Ismael,
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Thanks for the
> explanation.
>     > > > > > >     > > >     >     >         >> Specially I like this
> part where
>     > > in
>     > > > > you
>     > > > > > >     > mentioned
>     > > > > > >     > > > we can
>     > > > > > >     > > >     > get
>     > > > > > >     > > >     >     > rid of the
>     > > > > > >     > > >     >     >         >> older null value support
> for log
>     > > > > > > compaction
>     > > > > > >     > later
>     > > > > > >     > > > on,
>     > > > > > >     > > >     > here :
>     > > > > > >     > > >     >     >         >> We can't change semantics
> of the
>     > > > > message
>     > > > > > > format
>     > > > > > >     > > > without
>     > > > > > >     > > >     > having
>     > > > > > >     > > >     >     > a long
>     > > > > > >     > > >     >     >         >> transition period. And we
> can't
>     > > rely
>     > > > > > >     > > >     >     >         >> on people reading
> documentation
>     > or
>     > > > > > acting
>     > > > > > > on a
>     > > > > > >     > > > warning for
>     > > > > > >     > > >     >     > something so
>     > > > > > >     > > >     >     >         >> fundamental. As such, my
> take is
>     > > > that
>     > > > > we
>     > > > > > > need to
>     > > > > > >     > > > bump the
>     > > > > > >     > > >     > magic
>     > > > > > >     > > >     >     > byte. The
>     > > > > > >     > > >     >     >         >> good news is
>     > > > > > >     > > >     >     >         >> that we don't have to
> support
>     > all
>     > > > > > versions
>     > > > > > >     > > forever.
>     > > > > > >     > > > We
>     > > > > > >     > > >     > have
>     > > > > > >     > > >     >     > said that we
>     > > > > > >     > > >     >     >         >> will support direct
> upgrades
>     > for 2
>     > > > > > years.
>     > > > > > > That
>     > > > > > >     > > > means that
>     > > > > > >     > > >     >     > message format
>     > > > > > >     > > >     >     >         >> version n could, in
> theory, be
>     > > > > removed 2
>     > > > > > > years
>     > > > > > >     > > > after the
>     > > > > > >     > > >     > it's
>     > > > > > >     > > >     >     > introduced.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Just a heads up, I would
> like to
>     > > > > mention
>     > > > > > > that
>     > > > > > >     > even
>     > > > > > >     > > > without
>     > > > > > >     > > >     >     > bumping magic
>     > > > > > >     > > >     >     >         >> byte, we will *NOT* loose
> zero
>     > > copy
>     > > > as
>     > > > > > in
>     > > > > > > the
>     > > > > > >     > > > client(x+1)
>     > > > > > >     > > >     > in my
>     > > > > > >     > > >     >     >         >> explanation
>     > > > > > >     > > >     >     >         >> above will convert
> internally a
>     > > null
>     > > > > > > value to
>     > > > > > >     > > have a
>     > > > > > >     > > >     > tombstone
>     > > > > > >     > > >     >     > bit set and
>     > > > > > >     > > >     >     >         >> a tombstone bit set to
> have a
>     > null
>     > > > > value
>     > > > > > >     > > > automatically
>     > > > > > >     > > >     >     > internally and by
>     > > > > > >     > > >     >     >         >> the time we move to
> version
>     > (x+2),
>     > > > the
>     > > > > > > clients
>     > > > > > >     > > > would have
>     > > > > > >     > > >     >     > upgraded.
>     > > > > > >     > > >     >     >         >> Obviously if we support a
>     > request
>     > > > from
>     > > > > > >     > > consumer(x),
>     > > > > > >     > > > we
>     > > > > > >     > > >     > will
>     > > > > > >     > > >     >     > loose zero
>     > > > > > >     > > >     >     >         >> copy
>     > > > > > >     > > >     >     >         >> but that is the same case
> with
>     > > magic
>     > > > > > byte.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> But if magic byte bump
> makes
>     > life
>     > > > > easier
>     > > > > > > for
>     > > > > > >     > > > transition
>     > > > > > >     > > >     > for the
>     > > > > > >     > > >     >     > above
>     > > > > > >     > > >     >     >         >> reasons that you
> explained, I am
>     > > OK
>     > > > > with
>     > > > > > > it
>     > > > > > >     > since
>     > > > > > >     > > > we are
>     > > > > > >     > > >     > going
>     > > > > > >     > > >     >     > to meet the
>     > > > > > >     > > >     >     >         >> end goal down the road :)
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> On a side note can we
> update the
>     > > doc
>     > > > > > here
>     > > > > > > on
>     > > > > > >     > magic
>     > > > > > >     > > > byte
>     > > > > > >     > > >     > to say
>     > > > > > >     > > >     >     > that "*it
>     > > > > > >     > > >     >     >         >> should be bumped whenever
> the
>     > > > message
>     > > > > > > format is
>     > > > > > >     > > > changed
>     > > > > > >     > > >     > or the
>     > > > > > >     > > >     >     >         >> interpretation of message
> format
>     > > > > (usage
>     > > > > > > of the
>     > > > > > >     > > > reserved
>     > > > > > >     > > >     > bits as
>     > > > > > >     > > >     >     > well) is
>     > > > > > >     > > >     >     >         >> changed*".
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Hi Michael,
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Here is the update plan
> that we
>     > > > > > discussed
>     > > > > > >     > offline
>     > > > > > >     > > >     > yesterday :
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Currently the magic-byte
> which
>     > > > > > > corresponds to
>     > > > > > >     > the
>     > > > > > >     > > >     >     > "message.format.version"
>     > > > > > >     > > >     >     >         >> is set to 1.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> 1) On broker it will be
> set to 1
>     > > > > > > initially.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> 2) When a producer client
> sends
>     > a
>     > > > > > message
>     > > > > > > with
>     > > > > > >     > > > magic-byte
>     > > > > > >     > > >     > = 2,
>     > > > > > >     > > >     >     > since the
>     > > > > > >     > > >     >     >         >> broker is on magic-byte =
> 1, we
>     > > will
>     > > > > > down
>     > > > > > >     > convert
>     > > > > > >     > > > it,
>     > > > > > >     > > >     > which
>     > > > > > >     > > >     >     > means if the
>     > > > > > >     > > >     >     >         >> tombstone bit is set, the
> value
>     > > will
>     > > > > be
>     > > > > > > set to
>     > > > > > >     > > > null. A
>     > > > > > >     > > >     > consumer
>     > > > > > >     > > >     >     >         >> understanding magic-byte
> = 1,
>     > will
>     > > > > still
>     > > > > > > work
>     > > > > > >     > with
>     > > > > > >     > > > this. A
>     > > > > > >     > > >     >     > consumer
>     > > > > > >     > > >     >     >         >> working
>     > > > > > >     > > >     >     >         >> with magic-byte =2 will
> also be
>     > > able
>     > > > > to
>     > > > > > >     > understand
>     > > > > > >     > > > this,
>     > > > > > >     > > >     > since
>     > > > > > >     > > >     >     > it
>     > > > > > >     > > >     >     >         >> understands the tombstone.
>     > > > > > >     > > >     >     >         >> Now there is still the
> question
>     > of
>     > > > > > > supporting a
>     > > > > > >     > > >     > non-tombstone
>     > > > > > >     > > >     >     > and null
>     > > > > > >     > > >     >     >         >> value from producer
> client with
>     > > > > > > magic-byte = 2.*
>     > > > > > >     > > (I
>     > > > > > >     > > > am
>     > > > > > >     > > >     > not sure
>     > > > > > >     > > >     >     > if we
>     > > > > > >     > > >     >     >         >> should support this.
>     > Ismael/Becket
>     > > > can
>     > > > > > > comment
>     > > > > > >     > > > here)*
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> 3) When almost all the
> clients
>     > > have
>     > > > > > > upgraded,
>     > > > > > >     > the
>     > > > > > >     > > >     >     > message.format.version
>     > > > > > >     > > >     >     >         >> on
>     > > > > > >     > > >     >     >         >> the broker can be changed
> to 2,
>     > > > where
>     > > > > in
>     > > > > > > the
>     > > > > > >     > down
>     > > > > > >     > > >     > conversion in
>     > > > > > >     > > >     >     > the above
>     > > > > > >     > > >     >     >         >> step will not happen. If
> at this
>     > > > point
>     > > > > > we
>     > > > > > > get a
>     > > > > > >     > > > consumer
>     > > > > > >     > > >     >     > request from a
>     > > > > > >     > > >     >     >         >> older consumer, we might
> have to
>     > > > down
>     > > > > > > convert
>     > > > > > >     > > where
>     > > > > > >     > > > in we
>     > > > > > >     > > >     > loose
>     > > > > > >     > > >     >     > zero copy,
>     > > > > > >     > > >     >     >         >> but these cases should be
> rare.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Becket can you review
> this plan
>     > > and
>     > > > > add
>     > > > > > > more
>     > > > > > >     > > > details if I
>     > > > > > >     > > >     > have
>     > > > > > >     > > >     >     >         >> missed/wronged something,
> before
>     > > we
>     > > > > put
>     > > > > > > it on
>     > > > > > >     > KIP.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Thanks,
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> Mayuresh
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at
> 11:07
>     > PM,
>     > > > > > Michael
>     > > > > > >     > Pearce <
>     > > > > > >     > > >     >     > Michael.Pearce@ig.com>
>     > > > > > >     > > >     >     >         >> wrote:
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> > Thanks guys, for
> discussing
>     > this
>     > > > > > > offline and
>     > > > > > >     > > > getting
>     > > > > > >     > > >     > some
>     > > > > > >     > > >     >     > consensus.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > So its clear for myself
> and
>     > > others
>     > > > > > what
>     > > > > > > is
>     > > > > > >     > > > proposed now
>     > > > > > >     > > >     > (i
>     > > > > > >     > > >     >     > think i
>     > > > > > >     > > >     >     >         >> > understand, but want to
> make
>     > > sure)
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Could i ask either
> directly
>     > > update
>     > > > > the
>     > > > > > > kip to
>     > > > > > >     > > > detail the
>     > > > > > >     > > >     >     > migration
>     > > > > > >     > > >     >     >         >> > strategy, or (re-)state
> your
>     > > > offline
>     > > > > > > discussed
>     > > > > > >     > > and
>     > > > > > >     > > >     > agreed
>     > > > > > >     > > >     >     > migration
>     > > > > > >     > > >     >     >         >> > strategy based on a
> magic byte
>     > > is
>     > > > in
>     > > > > > > this
>     > > > > > >     > > thread.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > The main original
> driver for
>     > the
>     > > > KIP
>     > > > > > > was to
>     > > > > > >     > > > support
>     > > > > > >     > > >     >     > compaction where
>     > > > > > >     > > >     >     >         >> value
>     > > > > > >     > > >     >     >         >> > isn't null, based off
> the
>     > > > > discussions
>     > > > > > on
>     > > > > > >     > KIP-82
>     > > > > > >     > > > thread.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > We should be able to
> support
>     > > > > > > non-tombstone +
>     > > > > > >     > > null
>     > > > > > >     > > > value
>     > > > > > >     > > >     > by the
>     > > > > > >     > > >     >     >         >> completion
>     > > > > > >     > > >     >     >         >> > of the KIP, as we noted
> when
>     > > > > > discussing
>     > > > > > > this
>     > > > > > >     > > kip,
>     > > > > > >     > > > having
>     > > > > > >     > > >     >     > logic based on
>     > > > > > >     > > >     >     >         >> a
>     > > > > > >     > > >     >     >         >> > null value isn't very
> clean
>     > and
>     > > > also
>     > > > > > > separates
>     > > > > > >     > > the
>     > > > > > >     > > >     > concerns.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > As discussed already
> though we
>     > > can
>     > > > > > > split this
>     > > > > > >     > > into
>     > > > > > >     > > >     > KIP-87a
>     > > > > > >     > > >     >     > and KIP-87b
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Where we look to deliver
>     > KIP-87a
>     > > > on
>     > > > > a
>     > > > > > >     > compacted
>     > > > > > >     > > > topic
>     > > > > > >     > > >     > (to
>     > > > > > >     > > >     >     > address the
>     > > > > > >     > > >     >     >         >> > immediate issues)
>     > > > > > >     > > >     >     >         >> > * tombstone + null value
>     > > > > > >     > > >     >     >         >> > * tombstone + non-null
> value
>     > > > > > >     > > >     >     >         >> > * non-tombstone +
> non-null
>     > value
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Then we can discuss once
>     > KIP-87a
>     > > > is
>     > > > > > > completed
>     > > > > > >     > > > options
>     > > > > > >     > > >     > later
>     > > > > > >     > > >     >     > and how we
>     > > > > > >     > > >     >     >         >> > support the second part
>     > KIP-87b
>     > > to
>     > > > > > > deliver:
>     > > > > > >     > > >     >     >         >> > * non-tombstone + null
> value
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Cheers
>     > > > > > >     > > >     >     >         >> > Mike
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
> ______________________________
>     > > > > > > __________
>     > > > > > >     > > >     >     >         >> > From: Becket Qin <
>     > > > > > becket.qin@gmail.com>
>     > > > > > >     > > >     >     >         >> > Sent: Thursday,
> November 17,
>     > > 2016
>     > > > > 1:43
>     > > > > > > AM
>     > > > > > >     > > >     >     >         >> > To:
> dev@kafka.apache.org
>     > > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS]
> KIP-87
>     > -
>     > > > Add
>     > > > > > > Compaction
>     > > > > > >     > > >     > Tombstone Flag
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Renu, Mayuresh and I
> had an
>     > > > offline
>     > > > > > >     > discussion,
>     > > > > > >     > > > and
>     > > > > > >     > > >     > following
>     > > > > > >     > > >     >     > is a brief
>     > > > > > >     > > >     >     >         >> > summary.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > 1. We agreed that not
> bumping
>     > up
>     > > > > magic
>     > > > > > > value
>     > > > > > >     > may
>     > > > > > >     > > > result
>     > > > > > >     > > >     > in
>     > > > > > >     > > >     >     > losing zero
>     > > > > > >     > > >     >     >         >> copy
>     > > > > > >     > > >     >     >         >> > during migration.
>     > > > > > >     > > >     >     >         >> > 2. Given that bumping
> up magic
>     > > > value
>     > > > > > is
>     > > > > > > almost
>     > > > > > >     > > > free and
>     > > > > > >     > > >     > has
>     > > > > > >     > > >     >     > benefit of
>     > > > > > >     > > >     >     >         >> > avoiding potential
> performance
>     > > > > issue.
>     > > > > > > It is
>     > > > > > >     > > > probably
>     > > > > > >     > > >     > worth
>     > > > > > >     > > >     >     > doing.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > One issue we still need
> to
>     > think
>     > > > > about
>     > > > > > > is
>     > > > > > >     > > whether
>     > > > > > >     > > > we
>     > > > > > >     > > >     > want to
>     > > > > > >     > > >     >     > support a
>     > > > > > >     > > >     >     >         >> > non-tombstone message
> with
>     > null
>     > > > > value.
>     > > > > > >     > > >     >     >         >> > Currently it is not
> supported
>     > by
>     > > > > > Kafka.
>     > > > > > > If we
>     > > > > > >     > > > allow a
>     > > > > > >     > > >     >     > non-tombstone null
>     > > > > > >     > > >     >     >         >> > value message to exist
> after
>     > > > KIP-87.
>     > > > > > The
>     > > > > > >     > problem
>     > > > > > >     > > > is
>     > > > > > >     > > >     > that such
>     > > > > > >     > > >     >     > message
>     > > > > > >     > > >     >     >         >> will
>     > > > > > >     > > >     >     >         >> > not be supported by the
>     > > consumers
>     > > > > > prior
>     > > > > > > to
>     > > > > > >     > > KIP-87.
>     > > > > > >     > > >     > Because a
>     > > > > > >     > > >     >     > null value
>     > > > > > >     > > >     >     >         >> > will always be
> interpreted to
>     > a
>     > > > > > > tombstone.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > One option is that we
> keep the
>     > > > > current
>     > > > > > > way,
>     > > > > > >     > i.e.
>     > > > > > >     > > > do not
>     > > > > > >     > > >     >     > support such
>     > > > > > >     > > >     >     >         >> > message. It would be
> good to
>     > > know
>     > > > if
>     > > > > > > there is
>     > > > > > >     > a
>     > > > > > >     > > >     > concrete use
>     > > > > > >     > > >     >     > case for
>     > > > > > >     > > >     >     >         >> such
>     > > > > > >     > > >     >     >         >> > message. If there is
> not, we
>     > can
>     > > > > > > probably just
>     > > > > > >     > > not
>     > > > > > >     > > >     > support it.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > Thanks,
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at
> 1:28
>     > PM,
>     > > > > > > Mayuresh
>     > > > > > >     > > Gharat <
>     > > > > > >     > > >     >     >         >> >
> gharatmayuresh15@gmail.com
>     > > > > > >     > > >     >     >         >> > > wrote:
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >> > > Hi Ismael,
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > This is something I
> can
>     > think
>     > > of
>     > > > > for
>     > > > > > >     > migration
>     > > > > > >     > > > plan:
>     > > > > > >     > > >     >     >         >> > > So the migration plan
> can
>     > look
>     > > > > > > something
>     > > > > > >     > like
>     > > > > > >     > > > this,
>     > > > > > >     > > >     > with up
>     > > > > > >     > > >     >     >         >> conversion :
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > 1) Currently lets say
> we
>     > have
>     > > > > Broker
>     > > > > > > at
>     > > > > > >     > > version
>     > > > > > >     > > > x.
>     > > > > > >     > > >     >     >         >> > > 2) Currently we have
> clients
>     > > at
>     > > > > > > version x.
>     > > > > > >     > > >     >     >         >> > > 3) a) We move the
> version to
>     > > > > > > Broker(x+1) :
>     > > > > > >     > > > supports
>     > > > > > >     > > >     > both
>     > > > > > >     > > >     >     > tombstone and
>     > > > > > >     > > >     >     >         >> > null
>     > > > > > >     > > >     >     >         >> > > for log compaction.
>     > > > > > >     > > >     >     >         >> > >     b) We upgrade the
> client
>     > > to
>     > > > > > > version
>     > > > > > >     > > > client(x+1) :
>     > > > > > >     > > >     > if in
>     > > > > > >     > > >     >     > the
>     > > > > > >     > > >     >     >         >> producer
>     > > > > > >     > > >     >     >         >> > > client(x+1) the value
> is set
>     > > to
>     > > > > > null,
>     > > > > > > we
>     > > > > > >     > will
>     > > > > > >     > > >     > automatically
>     > > > > > >     > > >     >     > set the
>     > > > > > >     > > >     >     >         >> > > Tombstone bit
> internally. If
>     > > the
>     > > > > > > producer
>     > > > > > >     > > > client(x+1)
>     > > > > > >     > > >     > sets
>     > > > > > >     > > >     >     > the
>     > > > > > >     > > >     >     >         >> tombstone
>     > > > > > >     > > >     >     >         >> > > itself, well and
> good. For
>     > > > > producer
>     > > > > > >     > client(x),
>     > > > > > >     > > > the
>     > > > > > >     > > >     > broker
>     > > > > > >     > > >     >     > will up
>     > > > > > >     > > >     >     >         >> convert
>     > > > > > >     > > >     >     >         >> > > to have the tombstone
> bit.
>     > > > > > > Broker(x+1) is
>     > > > > > >     > > > supporting
>     > > > > > >     > > >     > both.
>     > > > > > >     > > >     >     > Consumer
>     > > > > > >     > > >     >     >         >> > > client(x+1) will be
> aware of
>     > > > this
>     > > > > > and
>     > > > > > > should
>     > > > > > >     > > be
>     > > > > > >     > > > able
>     > > > > > >     > > >     > to
>     > > > > > >     > > >     >     > handle this.
>     > > > > > >     > > >     >     >         >> For
>     > > > > > >     > > >     >     >         >> > > consumer client(x) we
> will
>     > > down
>     > > > > > > convert the
>     > > > > > >     > > > message
>     > > > > > >     > > >     > on the
>     > > > > > >     > > >     >     > broker
>     > > > > > >     > > >     >     >         >> side.
>     > > > > > >     > > >     >     >         >> > >     c) At this point
> we will
>     > > > have
>     > > > > to
>     > > > > > >     > specify a
>     > > > > > >     > > >     > warning or
>     > > > > > >     > > >     >     > clearly
>     > > > > > >     > > >     >     >         >> specify
>     > > > > > >     > > >     >     >         >> > > in docs that this
> behavior
>     > is
>     > > > > about
>     > > > > > > to be
>     > > > > > >     > > > changed for
>     > > > > > >     > > >     > log
>     > > > > > >     > > >     >     > compaction.
>     > > > > > >     > > >     >     >         >> > > 4) a) In next release
> of the
>     > > > > > > Broker(x+2), we
>     > > > > > >     > > > say that
>     > > > > > >     > > >     > only
>     > > > > > >     > > >     >     > Tombstone
>     > > > > > >     > > >     >     >         >> is
>     > > > > > >     > > >     >     >         >> > > used for log
> compaction on
>     > the
>     > > > > > Broker
>     > > > > > > side.
>     > > > > > >     > > >     > Clients(x+1)
>     > > > > > >     > > >     >     > still is
>     > > > > > >     > > >     >     >         >> > > supported.
>     > > > > > >     > > >     >     >         >> > >     b) We upgrade the
> client
>     > > to
>     > > > > > > version
>     > > > > > >     > > > client(x+2) :
>     > > > > > >     > > >     > if
>     > > > > > >     > > >     >     > value is set
>     > > > > > >     > > >     >     >         >> to
>     > > > > > >     > > >     >     >         >> > > null, tombstone will
> not be
>     > > set
>     > > > > > >     > automatically.
>     > > > > > >     > > > The
>     > > > > > >     > > >     > client
>     > > > > > >     > > >     >     > will have to
>     > > > > > >     > > >     >     >         >> > call
>     > > > > > >     > > >     >     >         >> > > setTombstone() to
> actually
>     > set
>     > > > the
>     > > > > > >     > tombstone.
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > We should compare this
>     > > migration
>     > > > > > plan
>     > > > > > > with
>     > > > > > >     > the
>     > > > > > >     > > >     > migration
>     > > > > > >     > > >     >     > plan for
>     > > > > > >     > > >     >     >         >> magic
>     > > > > > >     > > >     >     >         >> > > byte bump and do
> whatever
>     > > looks
>     > > > > > good.
>     > > > > > >     > > >     >     >         >> > > I am just worried
> that if we
>     > > go
>     > > > > down
>     > > > > > > magic
>     > > > > > >     > > byte
>     > > > > > >     > > > route,
>     > > > > > >     > > >     >     > unless I am
>     > > > > > >     > > >     >     >         >> > missing
>     > > > > > >     > > >     >     >         >> > > something, it sounds
> like
>     > > kafka
>     > > > > will
>     > > > > > > be
>     > > > > > >     > stuck
>     > > > > > >     > > > with
>     > > > > > >     > > >     >     > supporting both
>     > > > > > >     > > >     >     >         >> null
>     > > > > > >     > > >     >     >         >> > > value and tombstone
> bit for
>     > > log
>     > > > > > > compaction
>     > > > > > >     > for
>     > > > > > >     > > > life
>     > > > > > >     > > >     > long,
>     > > > > > >     > > >     >     > which does
>     > > > > > >     > > >     >     >         >> not
>     > > > > > >     > > >     >     >         >> > > look like a good end
> state.
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > Thanks,
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > Mayuresh
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016
> at 9:32
>     > > AM,
>     > > > > > > Mayuresh
>     > > > > > >     > > > Gharat <
>     > > > > > >     > > >     >     >         >> > >
> gharatmayuresh15@gmail.com
>     > > > > > >     > > >     >     >         >> > > > wrote:
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > > Hi Ismael,
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > That's a very good
> point
>     > > > which I
>     > > > > > > might
>     > > > > > >     > have
>     > > > > > >     > > > not
>     > > > > > >     > > >     >     > considered earlier.
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > Here is a plan that
> I can
>     > > > think
>     > > > > > of:
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > Stage 1) The broker
> from
>     > now
>     > > > on,
>     > > > > > up
>     > > > > > >     > converts
>     > > > > > >     > > > the
>     > > > > > >     > > >     > message
>     > > > > > >     > > >     >     > to have the
>     > > > > > >     > > >     >     >         >> > > > tombstone marker.
> The log
>     > > > > > compaction
>     > > > > > >     > thread
>     > > > > > >     > > > does log
>     > > > > > >     > > >     >     > compaction
>     > > > > > >     > > >     >     >         >> based
>     > > > > > >     > > >     >     >         >> > on
>     > > > > > >     > > >     >     >         >> > > > both null and
> tombstone
>     > > > marker.
>     > > > > > > This is
>     > > > > > >     > our
>     > > > > > >     > > >     > transition
>     > > > > > >     > > >     >     > period.
>     > > > > > >     > > >     >     >         >> > > > Stage 2) The next
> release
>     > we
>     > > > > only
>     > > > > > > say that
>     > > > > > >     > > log
>     > > > > > >     > > >     > compaction
>     > > > > > >     > > >     >     > is based
>     > > > > > >     > > >     >     >         >> on
>     > > > > > >     > > >     >     >         >> > > > tombstone marker.
> (Open
>     > > source
>     > > > > > > kafka makes
>     > > > > > >     > > > this as a
>     > > > > > >     > > >     >     > policy). By
>     > > > > > >     > > >     >     >         >> this
>     > > > > > >     > > >     >     >         >> > > time,
>     > > > > > >     > > >     >     >         >> > > > the organization
> which is
>     > > > moving
>     > > > > > to
>     > > > > > > this
>     > > > > > >     > > > release
>     > > > > > >     > > >     > will be
>     > > > > > >     > > >     >     > sure that
>     > > > > > >     > > >     >     >         >> they
>     > > > > > >     > > >     >     >         >> > > > have gone through
> the
>     > entire
>     > > > > > > transition
>     > > > > > >     > > > period.
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > My only goal of
> doing this
>     > > is
>     > > > > that
>     > > > > > > Kafka
>     > > > > > >     > > > clearly
>     > > > > > >     > > >     >     > specifies the end
>     > > > > > >     > > >     >     >         >> > state
>     > > > > > >     > > >     >     >         >> > > > about what log
> compaction
>     > > > means
>     > > > > > (is
>     > > > > > > it
>     > > > > > >     > null
>     > > > > > >     > > > value
>     > > > > > >     > > >     > or a
>     > > > > > >     > > >     >     > tombstone
>     > > > > > >     > > >     >     >         >> > marker,
>     > > > > > >     > > >     >     >         >> > > > but not both).
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > What do you think?
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > Thanks,
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > Mayuresh
>     > > > > > >     > > >     >     >         >> > > > .
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > On Wed, Nov 16,
> 2016 at
>     > 9:17
>     > > > AM,
>     > > > > > > Ismael
>     > > > > > >     > > Juma <
>     > > > > > >     > > >     >     > ismael@juma.me.uk>
>     > > > > > >     > > >     >     >         >> > wrote:
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > >> One comment below.
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16,
> 2016 at
>     > > 5:08
>     > > > > PM,
>     > > > > > > Mayuresh
>     > > > > > >     > > > Gharat <
>     > > > > > >     > > >     >     >         >> > > >>
>     > gharatmayuresh15@gmail.com
>     > > > > > >     > > >     >     >         >> > > >> > wrote:
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >> >    - If we don't
> bump
>     > up
>     > > > the
>     > > > > > > magic
>     > > > > > >     > byte,
>     > > > > > >     > > > on the
>     > > > > > >     > > >     > broker
>     > > > > > >     > > >     >     > side, the
>     > > > > > >     > > >     >     >         >> > > broker
>     > > > > > >     > > >     >     >         >> > > >> >    will always
> have to
>     > > look
>     > > > > at
>     > > > > > > both
>     > > > > > >     > > > tombstone
>     > > > > > >     > > >     > bit and
>     > > > > > >     > > >     >     > the value
>     > > > > > >     > > >     >     >         >> when
>     > > > > > >     > > >     >     >         >> > > do
>     > > > > > >     > > >     >     >         >> > > >> the
>     > > > > > >     > > >     >     >         >> > > >> >    compaction.
> Assuming
>     > > we
>     > > > do
>     > > > > > > not bump
>     > > > > > >     > up
>     > > > > > >     > > > the
>     > > > > > >     > > >     > magic
>     > > > > > >     > > >     >     > byte,
>     > > > > > >     > > >     >     >         >> > > >> >    imagine the
> broker
>     > > sees
>     > > > a
>     > > > > > > message
>     > > > > > >     > > which
>     > > > > > >     > > > does
>     > > > > > >     > > >     > not
>     > > > > > >     > > >     >     > have a
>     > > > > > >     > > >     >     >         >> tombstone
>     > > > > > >     > > >     >     >         >> > > bit
>     > > > > > >     > > >     >     >         >> > > >> >    set. The
> broker does
>     > > not
>     > > > > > know
>     > > > > > > when
>     > > > > > >     > the
>     > > > > > >     > > >     > message was
>     > > > > > >     > > >     >     > produced
>     > > > > > >     > > >     >     >         >> (i.e.
>     > > > > > >     > > >     >     >         >> > > >> > whether
>     > > > > > >     > > >     >     >         >> > > >> >    the message
> has been
>     > > up
>     > > > > > > converted or
>     > > > > > >     > > > not), it
>     > > > > > >     > > >     > has
>     > > > > > >     > > >     >     > to take a
>     > > > > > >     > > >     >     >         >> > further
>     > > > > > >     > > >     >     >         >> > > >> > look at
>     > > > > > >     > > >     >     >         >> > > >> >    the value to
> see if
>     > it
>     > > > is
>     > > > > > > null or
>     > > > > > >     > not
>     > > > > > >     > > in
>     > > > > > >     > > >     > order to
>     > > > > > >     > > >     >     > determine
>     > > > > > >     > > >     >     >         >> if it
>     > > > > > >     > > >     >     >         >> > > is
>     > > > > > >     > > >     >     >         >> > > >> a
>     > > > > > >     > > >     >     >         >> > > >> >    tombstone.
> The same
>     > > > logic
>     > > > > > has
>     > > > > > > to be
>     > > > > > >     > > put
>     > > > > > >     > > > on the
>     > > > > > >     > > >     >     > consumer as
>     > > > > > >     > > >     >     >         >> well
>     > > > > > >     > > >     >     >         >> > > >> because
>     > > > > > >     > > >     >     >         >> > > >> > the
>     > > > > > >     > > >     >     >         >> > > >> >    consumer does
> not
>     > know
>     > > > if
>     > > > > > the
>     > > > > > >     > message
>     > > > > > >     > > > has
>     > > > > > >     > > >     > been up
>     > > > > > >     > > >     >     > converted or
>     > > > > > >     > > >     >     >         >> > not.
>     > > > > > >     > > >     >     >         >> > > >> >       - If we
> upconvert
>     > > > while
>     > > > > > >     > appending,
>     > > > > > >     > > > this is
>     > > > > > >     > > >     > not
>     > > > > > >     > > >     >     > the case,
>     > > > > > >     > > >     >     >         >> > right?
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >> If I understand you
>     > > > correctly,
>     > > > > > > this is
>     > > > > > >     > not
>     > > > > > >     > > >     > sufficient
>     > > > > > >     > > >     >     > because the
>     > > > > > >     > > >     >     >         >> log
>     > > > > > >     > > >     >     >         >> > > may
>     > > > > > >     > > >     >     >         >> > > >> have messages
> appended
>     > > before
>     > > > > it
>     > > > > > > was
>     > > > > > >     > > > upgraded to
>     > > > > > >     > > >     > include
>     > > > > > >     > > >     >     > KIP-87.
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >> Ismael
>     > > > > > >     > > >     >     >         >> > > >>
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > > > --
>     > > > > > >     > > >     >     >         >> > > > -Regards,
>     > > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         >> > > > (862) 250-7125
>     > > > > > >     > > >     >     >         >> > > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > > --
>     > > > > > >     > > >     >     >         >> > > -Regards,
>     > > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         >> > > (862) 250-7125
>     > > > > > >     > > >     >     >         >> > >
>     > > > > > >     > > >     >     >         >> > The information
> contained in
>     > > this
>     > > > > > email
>     > > > > > > is
>     > > > > > >     > > > strictly
>     > > > > > >     > > >     >     > confidential and for
>     > > > > > >     > > >     >     >         >> > the use of the
> addressee only,
>     > > > > unless
>     > > > > > >     > otherwise
>     > > > > > >     > > >     > indicated. If
>     > > > > > >     > > >     >     > you are
>     > > > > > >     > > >     >     >         >> not
>     > > > > > >     > > >     >     >         >> > the intended recipient,
> please
>     > > do
>     > > > > not
>     > > > > > > read,
>     > > > > > >     > > copy,
>     > > > > > >     > > > use or
>     > > > > > >     > > >     >     > disclose to
>     > > > > > >     > > >     >     >         >> others
>     > > > > > >     > > >     >     >         >> > this message or any
>     > attachment.
>     > > > > Please
>     > > > > > > also
>     > > > > > >     > > > notify the
>     > > > > > >     > > >     > sender
>     > > > > > >     > > >     >     > by
>     > > > > > >     > > >     >     >         >> replying
>     > > > > > >     > > >     >     >         >> > to this email or by
> telephone
>     > > > > (+44(020
>     > > > > > > 7896
>     > > > > > >     > > 0011)
>     > > > > > >     > > > and
>     > > > > > >     > > >     > then
>     > > > > > >     > > >     >     > delete the
>     > > > > > >     > > >     >     >         >> email
>     > > > > > >     > > >     >     >         >> > and any copies of it.
>     > Opinions,
>     > > > > > > conclusion
>     > > > > > >     > (etc)
>     > > > > > >     > > > that
>     > > > > > >     > > >     > do not
>     > > > > > >     > > >     >     > relate to
>     > > > > > >     > > >     >     >         >> the
>     > > > > > >     > > >     >     >         >> > official business of
> this
>     > > company
>     > > > > > shall
>     > > > > > > be
>     > > > > > >     > > > understood as
>     > > > > > >     > > >     >     > neither given
>     > > > > > >     > > >     >     >         >> nor
>     > > > > > >     > > >     >     >         >> > endorsed by it. IG is a
>     > trading
>     > > > name
>     > > > > > of
>     > > > > > > IG
>     > > > > > >     > > Markets
>     > > > > > >     > > >     > Limited (a
>     > > > > > >     > > >     >     > company
>     > > > > > >     > > >     >     >         >> > registered in England
> and
>     > Wales,
>     > > > > > company
>     > > > > > >     > number
>     > > > > > >     > > >     > 04008957) and
>     > > > > > >     > > >     >     > IG Index
>     > > > > > >     > > >     >     >         >> > Limited (a company
> registered
>     > in
>     > > > > > > England and
>     > > > > > >     > > > Wales,
>     > > > > > >     > > >     > company
>     > > > > > >     > > >     >     > number
>     > > > > > >     > > >     >     >         >> > 01190902). Registered
> address
>     > at
>     > > > > > Cannon
>     > > > > > > Bridge
>     > > > > > >     > > > House, 25
>     > > > > > >     > > >     >     > Dowgate Hill,
>     > > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
>     > Markets
>     > > > > > Limited
>     > > > > > >     > > (register
>     > > > > > >     > > >     > number
>     > > > > > >     > > >     >     > 195355) and IG
>     > > > > > >     > > >     >     >         >> > Index Limited (register
> number
>     > > > > 114059)
>     > > > > > > are
>     > > > > > >     > > > authorised
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     > regulated by
>     > > > > > >     > > >     >     >         >> the
>     > > > > > >     > > >     >     >         >> > Financial Conduct
> Authority.
>     > > > > > >     > > >     >     >         >> >
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >> --
>     > > > > > >     > > >     >     >         >> -Regards,
>     > > > > > >     > > >     >     >         >> Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         >> (862) 250-7125
>     > > > > > >     > > >     >     >         >> The information contained
> in
>     > this
>     > > > > email
>     > > > > > is
>     > > > > > >     > > strictly
>     > > > > > >     > > >     >     > confidential and for
>     > > > > > >     > > >     >     >         >> the use of the addressee
> only,
>     > > > unless
>     > > > > > > otherwise
>     > > > > > >     > > >     > indicated. If
>     > > > > > >     > > >     >     > you are not
>     > > > > > >     > > >     >     >         >> the intended recipient,
> please
>     > do
>     > > > not
>     > > > > > > read,
>     > > > > > >     > copy,
>     > > > > > >     > > > use or
>     > > > > > >     > > >     >     > disclose to others
>     > > > > > >     > > >     >     >         >> this message or any
> attachment.
>     > > > Please
>     > > > > > > also
>     > > > > > >     > notify
>     > > > > > >     > > > the
>     > > > > > >     > > >     > sender
>     > > > > > >     > > >     >     > by replying
>     > > > > > >     > > >     >     >         >> to this email or by
> telephone
>     > > > (+44(020
>     > > > > > > 7896
>     > > > > > >     > 0011)
>     > > > > > >     > > > and then
>     > > > > > >     > > >     >     > delete the email
>     > > > > > >     > > >     >     >         >> and any copies of it.
> Opinions,
>     > > > > > > conclusion (etc)
>     > > > > > >     > > > that do
>     > > > > > >     > > >     > not
>     > > > > > >     > > >     >     > relate to the
>     > > > > > >     > > >     >     >         >> official business of this
>     > company
>     > > > > shall
>     > > > > > be
>     > > > > > >     > > > understood as
>     > > > > > >     > > >     >     > neither given nor
>     > > > > > >     > > >     >     >         >> endorsed by it. IG is a
> trading
>     > > name
>     > > > > of
>     > > > > > IG
>     > > > > > >     > Markets
>     > > > > > >     > > >     > Limited (a
>     > > > > > >     > > >     >     > company
>     > > > > > >     > > >     >     >         >> registered in England and
> Wales,
>     > > > > company
>     > > > > > > number
>     > > > > > >     > > > 04008957)
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     > IG Index
>     > > > > > >     > > >     >     >         >> Limited (a company
> registered in
>     > > > > England
>     > > > > > > and
>     > > > > > >     > > Wales,
>     > > > > > >     > > >     > company
>     > > > > > >     > > >     >     > number
>     > > > > > >     > > >     >     >         >> 01190902). Registered
> address at
>     > > > > Cannon
>     > > > > > > Bridge
>     > > > > > >     > > > House, 25
>     > > > > > >     > > >     >     > Dowgate Hill,
>     > > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG
> Markets
>     > > > > Limited
>     > > > > > >     > (register
>     > > > > > >     > > > number
>     > > > > > >     > > >     >     > 195355) and IG
>     > > > > > >     > > >     >     >         >> Index Limited (register
> number
>     > > > 114059)
>     > > > > > are
>     > > > > > >     > > > authorised and
>     > > > > > >     > > >     >     > regulated by the
>     > > > > > >     > > >     >     >         >> Financial Conduct
> Authority.
>     > > > > > >     > > >     >     >         >>
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >         > --
>     > > > > > >     > > >     >     >         > -Regards,
>     > > > > > >     > > >     >     >         > Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         > (862) 250-7125
>     > > > > > >     > > >     >     >         >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >         --
>     > > > > > >     > > >     >     >         -Regards,
>     > > > > > >     > > >     >     >         Mayuresh R. Gharat
>     > > > > > >     > > >     >     >         (862) 250-7125
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >     The information contained in
> this email
>     > > is
>     > > > > > > strictly
>     > > > > > >     > > > confidential
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     > for the use of the addressee only,
> unless
>     > > > > otherwise
>     > > > > > >     > > indicated.
>     > > > > > >     > > > If
>     > > > > > >     > > >     > you are
>     > > > > > >     > > >     >     > not the intended recipient, please
> do not
>     > > read,
>     > > > > > > copy, use
>     > > > > > >     > or
>     > > > > > >     > > >     > disclose to
>     > > > > > >     > > >     >     > others this message or any
> attachment.
>     > Please
>     > > > > also
>     > > > > > > notify
>     > > > > > >     > the
>     > > > > > >     > > > sender
>     > > > > > >     > > >     > by
>     > > > > > >     > > >     >     > replying to this email or by
> telephone
>     > > (+44(020
>     > > > > > 7896
>     > > > > > > 0011)
>     > > > > > >     > > and
>     > > > > > >     > > > then
>     > > > > > >     > > >     > delete
>     > > > > > >     > > >     >     > the email and any copies of it.
> Opinions,
>     > > > > > conclusion
>     > > > > > > (etc)
>     > > > > > >     > > > that do
>     > > > > > >     > > >     > not
>     > > > > > >     > > >     >     > relate to the official business of
> this
>     > > company
>     > > > > > > shall be
>     > > > > > >     > > > understood
>     > > > > > >     > > >     > as
>     > > > > > >     > > >     >     > neither given nor endorsed by it. IG
> is a
>     > > > trading
>     > > > > > > name of
>     > > > > > >     > IG
>     > > > > > >     > > > Markets
>     > > > > > >     > > >     >     > Limited (a company registered in
> England
>     > and
>     > > > > Wales,
>     > > > > > > company
>     > > > > > >     > > > number
>     > > > > > >     > > >     >     > 04008957) and IG Index Limited (a
> company
>     > > > > > registered
>     > > > > > > in
>     > > > > > >     > > > England and
>     > > > > > >     > > >     > Wales,
>     > > > > > >     > > >     >     > company number 01190902). Registered
>     > address
>     > > at
>     > > > > > > Cannon
>     > > > > > >     > Bridge
>     > > > > > >     > > > House,
>     > > > > > >     > > >     > 25
>     > > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both
> IG
>     > > Markets
>     > > > > > > Limited
>     > > > > > >     > > > (register
>     > > > > > >     > > >     > number
>     > > > > > >     > > >     >     > 195355) and IG Index Limited
> (register
>     > number
>     > > > > > > 114059) are
>     > > > > > >     > > > authorised
>     > > > > > >     > > >     > and
>     > > > > > >     > > >     >     > regulated by the Financial Conduct
>     > Authority.
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >     >
>     > > > > > >     > > >     >
>     > > > > > >     > > >     >
>     > > > > > >     > > >     > The information contained in this email is
>     > strictly
>     > > > > > > confidential
>     > > > > > >     > > and
>     > > > > > >     > > > for
>     > > > > > >     > > >     > the use of the addressee only, unless
> otherwise
>     > > > > > indicated.
>     > > > > > > If you
>     > > > > > >     > > > are not
>     > > > > > >     > > >     > the intended recipient, please do not
> read, copy,
>     > > use
>     > > > > or
>     > > > > > > disclose
>     > > > > > >     > > to
>     > > > > > >     > > > others
>     > > > > > >     > > >     > this message or any attachment. Please also
>     > notify
>     > > > the
>     > > > > > > sender by
>     > > > > > >     > > > replying
>     > > > > > >     > > >     > to this email or by telephone (+44(020
> 7896 0011
>     > > <020%207896%200011>) and
>     > > > > > then
>     > > > > > > delete
>     > > > > > >     > > > the email
>     > > > > > >     > > >     > and any copies of it. Opinions, conclusion
> (etc)
>     > > that
>     > > > > do
>     > > > > > > not
>     > > > > > >     > relate
>     > > > > > >     > > > to the
>     > > > > > >     > > >     > official business of this company shall be
>     > > understood
>     > > > > as
>     > > > > > > neither
>     > > > > > >     > > > given nor
>     > > > > > >     > > >     > endorsed by it. IG is a trading name of IG
>     > Markets
>     > > > > > Limited
>     > > > > > > (a
>     > > > > > >     > > company
>     > > > > > >     > > >     > registered in England and Wales, company
> number
>     > > > > 04008957)
>     > > > > > > and IG
>     > > > > > >     > > > Index
>     > > > > > >     > > >     > Limited (a company registered in England
> and
>     > Wales,
>     > > > > > company
>     > > > > > >     > number
>     > > > > > >     > > >     > 01190902). Registered address at Cannon
> Bridge
>     > > House,
>     > > > > 25
>     > > > > > > Dowgate
>     > > > > > >     > > > Hill,
>     > > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
>     > (register
>     > > > > number
>     > > > > > > 195355)
>     > > > > > >     > > > and IG
>     > > > > > >     > > >     > Index Limited (register number 114059) are
>     > > authorised
>     > > > > and
>     > > > > > >     > regulated
>     > > > > > >     > > > by the
>     > > > > > >     > > >     > Financial Conduct Authority.
>     > > > > > >     > > >     >
>     > > > > > >     > > >
>     > > > > > >     > > >
>     > > > > > >     > > > The information contained in this email is
> strictly
>     > > > > > confidential
>     > > > > > > and
>     > > > > > >     > for
>     > > > > > >     > > > the use of the addressee only, unless otherwise
>     > > indicated.
>     > > > If
>     > > > > > > you are
>     > > > > > >     > not
>     > > > > > >     > > > the intended recipient, please do not read,
> copy, use
>     > or
>     > > > > > > disclose to
>     > > > > > >     > > others
>     > > > > > >     > > > this message or any attachment. Please also
> notify the
>     > > > sender
>     > > > > > by
>     > > > > > >     > replying
>     > > > > > >     > > > to this email or by telephone (+44(020 7896 0011
>     > > <020%207896%200011>) and then
>     > > > > > > delete the
>     > > > > > >     > > email
>     > > > > > >     > > > and any copies of it. Opinions, conclusion (etc)
> that
>     > do
>     > > > not
>     > > > > > > relate to
>     > > > > > >     > > the
>     > > > > > >     > > > official business of this company shall be
> understood
>     > as
>     > > > > > neither
>     > > > > > > given
>     > > > > > >     > > nor
>     > > > > > >     > > > endorsed by it. IG is a trading name of IG
> Markets
>     > > Limited
>     > > > (a
>     > > > > > > company
>     > > > > > >     > > > registered in England and Wales, company number
>     > 04008957)
>     > > > and
>     > > > > > IG
>     > > > > > > Index
>     > > > > > >     > > > Limited (a company registered in England and
> Wales,
>     > > company
>     > > > > > > number
>     > > > > > >     > > > 01190902). Registered address at Cannon Bridge
> House,
>     > 25
>     > > > > > Dowgate
>     > > > > > > Hill,
>     > > > > > >     > > > London EC4R 2YA. Both IG Markets Limited
> (register
>     > number
>     > > > > > > 195355) and
>     > > > > > >     > IG
>     > > > > > >     > > > Index Limited (register number 114059) are
> authorised
>     > and
>     > > > > > > regulated by
>     > > > > > >     > > the
>     > > > > > >     > > > Financial Conduct Authority.
>     > > > > > >     > > >
>     > > > > > >     > > The information contained in this email is strictly
>     > > > > confidential
>     > > > > > > and for
>     > > > > > >     > > the use of the addressee only, unless otherwise
>     > indicated.
>     > > If
>     > > > > you
>     > > > > > > are not
>     > > > > > >     > > the intended recipient, please do not read, copy,
> use or
>     > > > > disclose
>     > > > > > > to
>     > > > > > >     > others
>     > > > > > >     > > this message or any attachment. Please also notify
> the
>     > > sender
>     > > > > by
>     > > > > > > replying
>     > > > > > >     > > to this email or by telephone (+44(020 7896 0011
>     > > <020%207896%200011>) and then
>     > > > > delete
>     > > > > > > the
>     > > > > > >     > email
>     > > > > > >     > > and any copies of it. Opinions, conclusion (etc)
> that do
>     > > not
>     > > > > > > relate to
>     > > > > > >     > the
>     > > > > > >     > > official business of this company shall be
> understood as
>     > > > > neither
>     > > > > > > given
>     > > > > > >     > nor
>     > > > > > >     > > endorsed by it. IG is a trading name of IG Markets
>     > Limited
>     > > (a
>     > > > > > > company
>     > > > > > >     > > registered in England and Wales, company number
> 04008957)
>     > > and
>     > > > > IG
>     > > > > > > Index
>     > > > > > >     > > Limited (a company registered in England and Wales,
>     > company
>     > > > > > number
>     > > > > > >     > > 01190902). Registered address at Cannon Bridge
> House, 25
>     > > > > Dowgate
>     > > > > > > Hill,
>     > > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > > > 195355)
>     > > > > > > and IG
>     > > > > > >     > > Index Limited (register number 114059) are
> authorised and
>     > > > > > > regulated by
>     > > > > > >     > the
>     > > > > > >     > > Financial Conduct Authority.
>     > > > > > >     > >
>     > > > > > >     > >
>     > > > > > >     > The information contained in this email is strictly
>     > > > confidential
>     > > > > > and
>     > > > > > > for
>     > > > > > >     > the use of the addressee only, unless otherwise
> indicated.
>     > If
>     > > > you
>     > > > > > > are not
>     > > > > > >     > the intended recipient, please do not read, copy,
> use or
>     > > > disclose
>     > > > > > to
>     > > > > > > others
>     > > > > > >     > this message or any attachment. Please also notify
> the
>     > sender
>     > > > by
>     > > > > > > replying
>     > > > > > >     > to this email or by telephone (+44(020 7896 0011
>     > > <020%207896%200011>) and then
>     > > > delete
>     > > > > > > the email
>     > > > > > >     > and any copies of it. Opinions, conclusion (etc)
> that do
>     > not
>     > > > > relate
>     > > > > > > to the
>     > > > > > >     > official business of this company shall be
> understood as
>     > > > neither
>     > > > > > > given nor
>     > > > > > >     > endorsed by it. IG is a trading name of IG Markets
> Limited
>     > (a
>     > > > > > company
>     > > > > > >     > registered in England and Wales, company number
> 04008957)
>     > and
>     > > > IG
>     > > > > > > Index
>     > > > > > >     > Limited (a company registered in England and Wales,
> company
>     > > > > number
>     > > > > > >     > 01190902). Registered address at Cannon Bridge
> House, 25
>     > > > Dowgate
>     > > > > > > Hill,
>     > > > > > >     > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > > > 195355)
>     > > > > > > and IG
>     > > > > > >     > Index Limited (register number 114059) are
> authorised and
>     > > > > regulated
>     > > > > > > by the
>     > > > > > >     > Financial Conduct Authority.
>     > > > > > >     >
>     > > > > > >     >
>     > > > > > >
>     > > > > > >
>     > > > > > > The information contained in this email is strictly
> confidential
>     > > and
>     > > > > for
>     > > > > > > the use of the addressee only, unless otherwise indicated.
> If you
>     > > are
>     > > > > not
>     > > > > > > the intended recipient, please do not read, copy, use or
> disclose
>     > > to
>     > > > > > others
>     > > > > > > this message or any attachment. Please also notify the
> sender by
>     > > > > replying
>     > > > > > > to this email or by telephone (+44(020 7896 0011
>     > > <020%207896%200011>) and then delete the
>     > > > > > email
>     > > > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
>     > relate
>     > > > to
>     > > > > > the
>     > > > > > > official business of this company shall be understood as
> neither
>     > > > given
>     > > > > > nor
>     > > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
>     > > company
>     > > > > > > registered in England and Wales, company number 04008957)
> and IG
>     > > > Index
>     > > > > > > Limited (a company registered in England and Wales, company
>     > number
>     > > > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > > > Hill,
>     > > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
>     > > and
>     > > > > IG
>     > > > > > > Index Limited (register number 114059) are authorised and
>     > regulated
>     > > > by
>     > > > > > the
>     > > > > > > Financial Conduct Authority.
>     > > > > > >
>     > > > > > The information contained in this email is strictly
> confidential
>     > and
>     > > > for
>     > > > > > the use of the addressee only, unless otherwise indicated.
> If you
>     > are
>     > > > not
>     > > > > > the intended recipient, please do not read, copy, use or
> disclose
>     > to
>     > > > > others
>     > > > > > this message or any attachment. Please also notify the
> sender by
>     > > > replying
>     > > > > > to this email or by telephone (+44(020 7896 0011
>     > <020%207896%200011>)
>     > > and then delete the
>     > > > > email
>     > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > > to
>     > > > > the
>     > > > > > official business of this company shall be understood as
> neither
>     > > given
>     > > > > nor
>     > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>     > company
>     > > > > > registered in England and Wales, company number 04008957)
> and IG
>     > > Index
>     > > > > > Limited (a company registered in England and Wales, company
> number
>     > > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > > Hill,
>     > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
>     > and
>     > > > IG
>     > > > > > Index Limited (register number 114059) are authorised and
> regulated
>     > > by
>     > > > > the
>     > > > > > Financial Conduct Authority.
>     > > > > >
>     > > > > The information contained in this email is strictly
> confidential and
>     > > for
>     > > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > > not
>     > > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > > others
>     > > > > this message or any attachment. Please also notify the sender
> by
>     > > replying
>     > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > > and then delete the
>     > > > email
>     > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > to
>     > > > the
>     > > > > official business of this company shall be understood as
> neither
>     > given
>     > > > nor
>     > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > > registered in England and Wales, company number 04008957) and
> IG
>     > Index
>     > > > > Limited (a company registered in England and Wales, company
> number
>     > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > Hill,
>     > > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > > IG
>     > > > > Index Limited (register number 114059) are authorised and
> regulated
>     > by
>     > > > the
>     > > > > Financial Conduct Authority.
>     > > > >
>     > > > The information contained in this email is strictly confidential
> and
>     > for
>     > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > not
>     > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > others
>     > > > this message or any attachment. Please also notify the sender by
>     > replying
>     > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
>     > > and then delete the email
>     > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > > the
>     > > > official business of this company shall be understood as neither
> given
>     > > nor
>     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > > Limited (a company registered in England and Wales, company
> number
>     > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > IG
>     > > > Index Limited (register number 114059) are authorised and
> regulated by
>     > > the
>     > > > Financial Conduct Authority.
>     > > >
>     > >
>     > The information contained in this email is strictly confidential and
> for
>     > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > the intended recipient, please do not read, copy, use or disclose to
> others
>     > this message or any attachment. Please also notify the sender by
> replying
>     > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
>     > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
>     > official business of this company shall be understood as neither
> given nor
>     > endorsed by it. IG is a trading name of IG Markets Limited (a company
>     > registered in England and Wales, company number 04008957) and IG
> Index
>     > Limited (a company registered in England and Wales, company number
>     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > Index Limited (register number 114059) are authorised and regulated
> by the
>     > Financial Conduct Authority.
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Ismael


So I understand what is being suggested, is we allow on the ProducerRecord at time of construction to be nullable aka big B, Boolean, where old constructors set it to null.


    /**
     * Creates a record with a specified timestamp to be sent to a specified topic and partition
     *
     * @param topic The topic the record will be appended to
     * @param partition The partition to which the record should be sent
     * @param tombstone if the record is a tombstone marker.
     * @param timestamp The timestamp of the record
     * @param key The key that will be included in the record
     * @param value The record contents
     */
    public ProducerRecord(String topic, Integer partition, Boolean tombstone, Long timestamp, K key, V value) {
        if (topic == null)
            throw new IllegalArgumentException("Topic cannot be null");
        if (timestamp != null && timestamp < 0)
            throw new IllegalArgumentException("Invalid timestamp " + timestamp);
        this.topic = topic;
        this.partition = partition;
        this.tombstone = tombstone;
        this.key = key;
        this.value = value;
        this.timestamp = timestamp;
    }

    public ProducerRecord(String topic, Integer partition, Long timestamp, K key, V value)
    {
        this(topic, partition, null, timestamp, key, value);
    }


Then within the KafkaProducer an enforced/defaulter interceptor or simply hardcoded in the on send if it detects the tombstone is null, it sets the tombstone to true or false based on a configuration property of the KafkaProducer?

Aka in this method, or by adding a default interceptor ->

    @Override
    public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
        // intercept the record, which can be potentially modified; this method does not throw exceptions
        ProducerRecord<K, V> interceptedRecord = this.interceptors == null ? record : this.interceptors.onSend(record);
        return doSend(interceptedRecord, callback);
    }

e.g sample default interceptor method we can either just add this to the interceptors by default or code this same logic in the send/doSend logic.

public class TombstoneInterceptor implements ProducerInterceptor
{
   boolean defaultNullValueTombstoneMarker;

   @Override
   public void configure(Map<String, ?> configs)
   {
      defaultNullValueTombstoneMarker = (Boolean)configs.get("tombstone.default.null.value");
   }

   @Override
   public ProducerRecord onSend(ProducerRecord record)
   {
      if (record.tombstone() == null){
         boolean tombstone = false;
         if (record.value() == null){
            tombstone = defaultNullValueTombstoneMarker;
         }
         return new ProducerRecord(record.topic(), record.partition(), tombstone, record.timestamp(), record.key(), record.value());
      }
      return record;
   }

   @Override
   public void onAcknowledgement(RecordMetadata metadata, Exception exception)
   {

   }

   @Override
   public void close()
   {

   }
}

Is this a correct understanding?

Cheers
Mike

On 16/12/2016, 21:16, "ismaelj@gmail.com on behalf of Ismael Juma" <ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:

    Hi Michael,

    I am not sure about using a new policy. One other option (suggested by
    Gwen) is that the behaviour of the producer with regards to tombstones when
    you pass a `null` value and don't use the constructor that takes a
    tombstone can be influenced by a producer config. Not ideal, but would
    limit the complexity of maintaining compatibility while providing a path
    for the (what I think) are the right semantics.

    Ismael

    On Tue, Dec 13, 2016 at 8:32 AM, Michael Pearce <Mi...@ig.com>
    wrote:

    > Hi Ismael
    >
    > Did you see our email this morning, what's your thoughts on this approach
    > to instead we simply have a brand new policy?
    >
    > Cheers
    > Mike
    >
    >
    > Sent using OWA for iPhone
    > ________________________________________
    > From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <
    > ismael@juma.me.uk>
    > Sent: Tuesday, December 13, 2016 11:30:05 AM
    > To: dev@kafka.apache.org
    > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >
    > Yes, this is actually tricky to do in a way where we both have the desired
    > semantics and maintain compatibility. When someone creates a
    > `ProducerRecord` with a `null` value today, the producer doesn't know if
    > it's meant to be a tombstone or not. For V3 messages, it's easy when the
    > constructor that takes a tombstone is used. However, if any other
    > constructor is used, it's not clear. A couple of options I can think of,
    > none of them particularly nice:
    >
    > 1. Have a third state where tombstone = unknown and the broker would set
    > the tombstone bit if the value was null and the topic was compacted. People
    > that wanted to pass a non-null value for the tombstone would have to use
    > the constructor that takes a tombstone. The drawbacks: third state for
    > tombstone in message format, message conversion at the broker for a common
    > case.
    >
    > 2. Extend MetadataResponse to optionally include topic configs, which would
    > make it possible for the producer to be smarter about setting the
    > tombstone. It would only do it if a tombstone was not passed explicitly,
    > the value was null and the topic was compacted. The main drawback is that
    > the producer would be getting a bit more data for each topic even though it
    > probably won't use it most of the time. Extending MetadataResponse to
    > return topic configs would be useful for other reasons as well, so that
    > part seems OK.
    >
    > In addition, for both proposals, we could consider adding warnings to the
    > documentation that the behaviour of the constructors that don't take a
    > tombstone would change in the next major release so that tombstone = false.
    > Not sure if this would be worth it though.
    >
    > Ismael
    >
    > On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <ewen@confluent.io
    > >
    > wrote:
    >
    > > Michael,
    > >
    > > It kind of depends on how you want to interpret the tombstone flag. If
    > it's
    > > purely a producer-facing Kafka-level thing that we treat as internal to
    > the
    > > broker and log cleaner once the record is sent, then your approach makes
    > > sense. You're just moving copying the null-indicates-delete behavior of
    > the
    > > old constructor into the tombstone flag.
    > >
    > > However, if you want this change to more generally decouple the idea of
    > > deletion and null values, then you are sometimes converting what might
    > be a
    > > completely valid null value that doesn't indicate deletion into a
    > > tombstone. Downstream applications could potentially handle these cases
    > > differently given the separation of deletion from value.
    > >
    > > I guess the question is if we want to try to support the latter even for
    > > topics where we have older produce requests. An example where this could
    > > come up is in something like a CDC Connector. If we try to support the
    > > semantic difference, a connector might write changes to Kafka using the
    > > tombstone flag to indicate when a row was truly deleted (vs an update
    > that
    > > sets it to null but still present; this probably makes more sense for CDC
    > > from document stores or extracting single columns). There are various
    > > reasons we might want to maintain the full log and not turn compaction on
    > > (or just use a time-based retention policy), but downstream applications
    > > might care to know the difference between a delete and a null value. In
    > > fact both versions of the same log (compacted and time-retention) could
    > be
    > > useful and I don't think it'll be uncommon to maintain both or use KIP-71
    > > to maintain a hybrid compacted/retention topic.
    > >
    > > -Ewen
    > >
    > > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <Mi...@ig.com>
    > > wrote:
    > >
    > > > Hi Jay,
    > > >
    > > > Why wouldn't that work, the tombstone value is only looked at by the
    > > > broker, on a topic configured for compaction as such is benign on non
    > > > compacted topics. This is as much as sending a null value currently
    > > >
    > > >
    > > > Regards
    > > > Mike
    > > >
    > > >
    > > >
    > > > Sent using OWA for iPhone
    > > > ________________________________________
    > > > From: Jay Kreps <ja...@confluent.io>
    > > > Sent: Sunday, December 11, 2016 8:58:53 PM
    > > > To: dev@kafka.apache.org
    > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > >
    > > > Hey Michael,
    > > >
    > > > I'm not quite sure that works as that would translate ALL null values
    > to
    > > > tombstones, even for non-compacted topics that use null as an
    > acceptable
    > > > value sent by the producer and expected by the consumer.
    > > >
    > > > -Jay
    > > >
    > > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Michael.Pearce@ig.com
    > >
    > > > wrote:
    > > >
    > > > > Hi Ewen,
    > > > >
    > > > > I think the easiest way to show this is with code.
    > > > >
    > > > > As you can see we keep the existing behaviour for code/binaries
    > calling
    > > > > the pre-existing constructors, whereby if the value is null the
    > > tombstone
    > > > > is set to true.
    > > > >
    > > > > Regards
    > > > > Mike
    > > > >
    > > > >
    > > > >
    > > > >     /**
    > > > >      * Creates a record with a specified timestamp to be sent to a
    > > > > specified topic and partition
    > > > >      *
    > > > >      * @param topic The topic the record will be appended to
    > > > >      * @param partition The partition to which the record should be
    > > sent
    > > > >      * @param timestamp The timestamp of the record
    > > > >      * @param tombstone if the record should be treated as a
    > tombstone
    > > if
    > > > > the topic is compacted
    > > > >      * @param key The key that will be included in the record
    > > > >      * @param value The record contents
    > > > >      */
    > > > >     public ProducerRecord(String topic, Integer partition, Boolean
    > > > > tombstone, Long timestamp, K key, V value) {
    > > > >         if (topic == null)
    > > > >             throw new IllegalArgumentException("Topic cannot be
    > > null.");
    > > > >         if (timestamp != null && timestamp < 0)
    > > > >             throw new IllegalArgumentException(
    > > > >                     String.format("Invalid timestamp: %d. Timestamp
    > > > should
    > > > > always be non-negative or null.", timestamp));
    > > > >         if (partition != null && partition < 0)
    > > > >             throw new IllegalArgumentException(
    > > > >                     String.format("Invalid partition: %d. Partition
    > > > number
    > > > > should always be non-negative or null.", partition));
    > > > >         if (!tombstone && value == null){
    > > > >             throw new IllegalArgumentException(
    > > > >                     String.format("Tombstone must be true if null
    > > > value"));
    > > > >         }
    > > > >         this.topic = topic;
    > > > >         this.partition = partition;
    > > > >         this.tombstone = tombstone;
    > > > >         this.key = key;
    > > > >         this.value = value;
    > > > >         this.timestamp = timestamp;
    > > > >     }
    > > > >
    > > > >     public ProducerRecord(String topic, Integer partition, Long
    > > > timestamp,
    > > > > K key, V value) {
    > > > >         this(topic, partition, value == null, timestamp, key, value);
    > > > >     }
    > > > > ________________________________________
    > > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
    > > > > Sent: Sunday, December 11, 2016 5:45 AM
    > > > > To: dev@kafka.apache.org
    > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > >
    > > > > It seemed like this was addressed in the migration section, wasn't
    > it?
    > > > The
    > > > > V2 vs V3 requests and the fact that the broker will downgrade the
    > > message
    > > > > format (losing zero copy) if you issues a V2 request to a broker
    > using
    > > V3
    > > > > format handles compatibility, does it not? The existing consumers
    > won't
    > > > see
    > > > > the extra metadata in the value, but they will get a null instead and
    > > > treat
    > > > > it as a tombstone. Certainly there is a performance impact, but it
    > > seemed
    > > > > compatible.
    > > > >
    > > > > I'm worried about this though:
    > > > >
    > > > >
    > > > >    - *NOTE *: With the new version of producer client using
    > > > ProduceRequest
    > > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and
    > > null
    > > > > value
    > > > >    should not be allowed as the older version of consumer using
    > > > > FetchRequest
    > > > >    V2 will think of this as a tombstone when its actually not.
    > > > >
    > > > >
    > > > > Unless I'm misunderstanding, this ends up breaking binary
    > compatibility
    > > > for
    > > > > the Java API. It sounds like this suggests that passing a null value
    > to
    > > > the
    > > > > existing ProducerRecord constructors would generate an exception
    > since
    > > > you
    > > > > didn't explicitly enable the tombstone (via whatever new constructor
    > is
    > > > > provided). But that means you can't swap in a newer client jar
    > without
    > > > > recompiling and get the same behavior if your app deletes keys using
    > > the
    > > > > current approach because your app will start throwing exceptions.
    > Maybe
    > > > > this is a tradeoff we're ok with, but we've tried pretty hard to
    > avoid
    > > > > breaking compatibility like this so far -- it makes picking up bug
    > > fixes
    > > > in
    > > > > the clients harder for users of frameworks that have to pin to
    > earlier
    > > > > default versions for compatibility.
    > > > >
    > > > > But then later the KIP says:
    > > > >
    > > > >
    > > > >    - The old constructors for ProducerRecord without this parameter
    > > will
    > > > be
    > > > >    kept but updated so that their default behaviour if setting null
    > > value
    > > > > will
    > > > >    be the tombstone will be set to true to keep existing behaviour.
    > > > >
    > > > >
    > > > > So maybe I am misinterpreting something.
    > > > >
    > > > > And just a nit re: motivation section, item 6 would be clearer for a
    > > > union
    > > > > schema with null (or optional schemas in other formats), e.g. [null,
    > > > > string], in which case losing the schema truly is losing information
    > > > > (whereas null is already the only valid value for a pure null
    > schema).
    > > > >
    > > > > -Ewen
    > > > >
    > > > >
    > > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
    > Michael.Pearce@ig.com
    > > >
    > > > > wrote:
    > > > >
    > > > > > Hi Jay,
    > > > > >
    > > > > > Good point this detail is missing in the KIP write up. Ive added
    > this
    > > > > now.
    > > > > >
    > > > > > Essentially simply just upgrading the clients we do not expect any
    > > > client
    > > > > > app code change needed.
    > > > > >
    > > > > > Cheers
    > > > > > Mike
    > > > > > ________________________________________
    > > > > > From: Jay Kreps <ja...@confluent.io>
    > > > > > Sent: Saturday, December 10, 2016 10:51 PM
    > > > > > To: dev@kafka.apache.org
    > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > > >
    > > > > > Michael,
    > > > > >
    > > > > > The compatibility section goes through the migration path, but
    > isn't
    > > > the
    > > > > > bigger compatibility issue with existing apps? There are many
    > > (probably
    > > > > > thousands) of apps in production that use this feature and send
    > null
    > > to
    > > > > > mean delete. It seems like this would break compatibility with
    > them,
    > > > and
    > > > > > they would have to be rewritten to right?
    > > > > >
    > > > > > -Jay
    > > > > >
    > > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
    > > Michael.Pearce@ig.com
    > > > >
    > > > > > wrote:
    > > > > >
    > > > > > > Hi Jun,
    > > > > > >
    > > > > > > 4) On v3 we honour the tombstone. As such we expect it to be set
    > > > > > correctly
    > > > > > > as per the KIP.
    > > > > > >
    > > > > > > 4.1) why would we want to produce an error when its v3? This is
    > the
    > > > > exact
    > > > > > > purpose to support non-null tombstone’s
    > > > > > > 4.2) again here im unclear on the question on the v3, produce
    > > request
    > > > > we
    > > > > > > expect the tombstone flag to be set correctly.
    > > > > > >
    > > > > > > 4.4) compaction only occurs on compacted topics, the bit makes no
    > > > > > > difference and not looked at on un-compacted (time/size based
    > > > > eviction).
    > > > > > >
    > > > > > >
    > > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
    > > > > > >
    > > > > > >     Hi, Michael,
    > > > > > >
    > > > > > >     4. Then, I think I misunderstood this point. Could you
    > document
    > > > the
    > > > > > >     following points in the wiki?
    > > > > > >     4.1 If producer V3 sets tombstone, but provides a non-null
    > > value,
    > > > > > does
    > > > > > > the
    > > > > > >     send() get an error or does the producer automatically set
    > the
    > > > > value
    > > > > > to
    > > > > > >     null?
    > > > > > >     4.2 If producer V3 doesn't set tombstone, but provides a null
    > > > > value,
    > > > > > > does
    > > > > > >     the send() get an error or does the producer automatically
    > sets
    > > > the
    > > > > > >     tombstone?
    > > > > > >     4.3 Does the broker only expect messages that (a) have no
    > > > tombstone
    > > > > > and
    > > > > > >     non-null value; (b) have tombstone and null value and reject
    > > the
    > > > > > > messages
    > > > > > >     with an error code otherwise?
    > > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
    > compacted
    > > on
    > > > > > not?
    > > > > > >
    > > > > > >     Thanks,
    > > > > > >
    > > > > > >     Jun
    > > > > > >
    > > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
    > > > > > Michael.Pearce@ig.com
    > > > > > > >
    > > > > > >     wrote:
    > > > > > >
    > > > > > >     > Not at all.  This only acts on compacted topics just as
    > what
    > > > > occurs
    > > > > > > today
    > > > > > >     >
    > > > > > >     > Sent using OWA for iPhone
    > > > > > >     > ________________________________________
    > > > > > >     > From: Jun Rao <ju...@confluent.io>
    > > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
    > > > > > >     > To: dev@kafka.apache.org
    > > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > > > >     >
    > > > > > >     > Hi, Michael,
    > > > > > >     >
    > > > > > >     > 4. Hmm, does that mean the new client library can never
    > send
    > > a
    > > > > null
    > > > > > > message
    > > > > > >     > even to a regular topic? This seems like a change of the
    > > > existing
    > > > > > > behavior.
    > > > > > >     >
    > > > > > >     > Thanks,
    > > > > > >     >
    > > > > > >     > Jun
    > > > > > >     >
    > > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
    > > > > > > Michael.Pearce@ig.com>
    > > > > > >     > wrote:
    > > > > > >     >
    > > > > > >     > > Hi Jun,
    > > > > > >     > >
    > > > > > >     > > Re 4) That's because we expect the tombstone value to be
    > > set
    > > > > > > correctly if
    > > > > > >     > > message bit is 2, as such if an older client sends in on
    > > old
    > > > > > > message the
    > > > > > >     > > message is upcast and the bit is set correctly. And such
    > no
    > > > > > longer
    > > > > > > need
    > > > > > >     > to
    > > > > > >     > > check the value. Mayuresh can you confirm my thinking and
    > > > > > > understanding
    > > > > > >     > of
    > > > > > >     > > what we've discussed?
    > > > > > >     > >
    > > > > > >     > > The second point I understand what you're getting at now
    > my
    > > > > > > apologies.
    > > > > > >     > Yes
    > > > > > >     > > this makes sense to save on touching the message, if
    > we're
    > > > the
    > > > > > > only kip
    > > > > > >     > > going in, in this release.
    > > > > > >     > >
    > > > > > >     > > Cheers
    > > > > > >     > > Mike
    > > > > > >     > >
    > > > > > >     > > Sent using OWA for iPhone
    > > > > > >     > > ________________________________________
    > > > > > >     > > From: Jun Rao <ju...@confluent.io>
    > > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
    > > > > > >     > > To: dev@kafka.apache.org
    > > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
    > Flag
    > > > > > >     > >
    > > > > > >     > > Hi, Michael,
    > > > > > >     > >
    > > > > > >     > > 4. Is this updated in the wiki? The text "If the magic
    > byte
    > > > on
    > > > > > > message is
    > > > > > >     > > 2, the broker should use the tombstone bit for log
    > > > compaction."
    > > > > > > doesn't
    > > > > > >     > > seem to have changed.
    > > > > > >     > >
    > > > > > >     > > 2. My point is that if we change the message format just
    > > for
    > > > > this
    > > > > > > KIP, we
    > > > > > >     > > should consider whether it's worth optimizing the down
    > > > > conversion
    > > > > > > path
    > > > > > >     > > (i.e., decide whether a conversion is needed by just
    > > looking
    > > > at
    > > > > > the
    > > > > > >     > > tombstone bit in the wrapper message) since tombstone
    > will
    > > be
    > > > > > used
    > > > > > >     > rarely.
    > > > > > >     > > However, if the message format change here is combined
    > with
    > > > > other
    > > > > > > KIPs,
    > > > > > >     > > then this optimization likely won't be needed. The latter
    > > > > > probably
    > > > > > > makes
    > > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
    > > > > > >     > >
    > > > > > >     > > Other than those, +1 from me,
    > > > > > >     > >
    > > > > > >     > > Thanks,
    > > > > > >     > >
    > > > > > >     > > Jun
    > > > > > >     > >
    > > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
    > > > > > > Michael.Pearce@ig.com>
    > > > > > >     > > wrote:
    > > > > > >     > >
    > > > > > >     > > > Hi Jun
    > > > > > >     > > >
    > > > > > >     > > > do we have your vote on this now?
    > > > > > >     > > >
    > > > > > >     > > > Any other concerns?
    > > > > > >     > > >
    > > > > > >     > > > Cheers
    > > > > > >     > > > Mike
    > > > > > >     > > >
    > > > > > >     > > > Sent using OWA for iPhone
    > > > > > >     > > > ________________________________________
    > > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
    > > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
    > > > > > >     > > > To: dev@kafka.apache.org
    > > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
    > > Flag
    > > > > > >     > > >
    > > > > > >     > > > Hi Jun,
    > > > > > >     > > >
    > > > > > >     > > > Have updated. Thanks again for the feedback.
    > > > > > >     > > >
    > > > > > >     > > > Agree yes we should align up when it gets to that, I
    > > assume
    > > > > > > you’ve
    > > > > > >     > > flagged
    > > > > > >     > > > the same in the other KIP?
    > > > > > >     > > >
    > > > > > >     > > > I think for now let’s get this KIP approved, then we
    > can
    > > > > start
    > > > > > > the work
    > > > > > >     > > to
    > > > > > >     > > > get an initial PR, then we can discuss how to align the
    > > two
    > > > > if
    > > > > > > needed.
    > > > > > >     > > >
    > > > > > >     > > > Cheers,
    > > > > > >     > > > Mike
    > > > > > >     > > >
    > > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io>
    > > wrote:
    > > > > > >     > > >
    > > > > > >     > > >     Hi, Michael,
    > > > > > >     > > >
    > > > > > >     > > >     For 2), this is fine. Could you update the KIP wiki
    > > to
    > > > > make
    > > > > > > this
    > > > > > >     > and
    > > > > > >     > > > other
    > > > > > >     > > >     points clearer? Other than that, the KIP looks good
    > > to
    > > > > me.
    > > > > > >     > > >
    > > > > > >     > > >     An orthogonal thing is that there are other KIPs
    > such
    > > > as
    > > > > > > KIP-98
    > > > > > >     > that
    > > > > > >     > > > also
    > > > > > >     > > >     intend to change the message format. If they all
    > get
    > > > > > > approved, we
    > > > > > >     > > > should
    > > > > > >     > > >     think about whether it's better to just bump up the
    > > > magic
    > > > > > > byte once
    > > > > > >     > > to
    > > > > > >     > > >     incorporate multiple format changes like we did in
    > > > > > > KIP-31/KIP-32.
    > > > > > >     > > >
    > > > > > >     > > >     Thanks,
    > > > > > >     > > >
    > > > > > >     > > >     Jun
    > > > > > >     > > >
    > > > > > >     > > >
    > > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
    > > > > > >     > > Michael.Pearce@ig.com>
    > > > > > >     > > >     wrote:
    > > > > > >     > > >
    > > > > > >     > > >     > Hi Jao
    > > > > > >     > > >     >
    > > > > > >     > > >     > Thanks for the response. Sorry for slow reply,
    > both
    > > > > with
    > > > > > > personal
    > > > > > >     > > > sickness
    > > > > > >     > > >     > and also battling some critical issues
    > encountered
    > > > > since
    > > > > > >     > upgrading
    > > > > > >     > > to
    > > > > > >     > > >     > 0.10.1.0
    > > > > > >     > > >     >
    > > > > > >     > > >     > 1) Thans for spotting, Document error where we
    > > > branched
    > > > > > > this KIP
    > > > > > >     > > from
    > > > > > >     > > >     > KIP-82, will get that removed.
    > > > > > >     > > >     > 2) Intent is to do this just at the record
    > message
    > > > > level.
    > > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
    > > > corrected.
    > > > > > >     > > >     > 4) As per discussion thread we will support
    > > > tombstone +
    > > > > > > null
    > > > > > >     > value,
    > > > > > >     > > >     > tombstone + non null value, no tombstone + null
    > > > value.
    > > > > > >     > > >     > 5) I believe this was in the discussion thread,
    > > > > @Mayuresh
    > > > > > > is this
    > > > > > >     > > >     > something we’ve overlooked? I thought we would
    > down
    > > > > > > convert and
    > > > > > >     > > > remove the
    > > > > > >     > > >     > value so the old consumer had existing behavior,
    > or
    > > > is
    > > > > > > there
    > > > > > >     > > > something we
    > > > > > >     > > >     > haven’t thought about?
    > > > > > >     > > >     >
    > > > > > >     > > >     > Cheers
    > > > > > >     > > >     > Mike
    > > > > > >     > > >     >
    > > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
    > jun@confluent.io>
    > > > > > wrote:
    > > > > > >     > > >     >
    > > > > > >     > > >     >     Hi, Michael,
    > > > > > >     > > >     >
    > > > > > >     > > >     >     Thanks for the KIP. A few comments below.
    > > > > > >     > > >     >
    > > > > > >     > > >     >     1. The message format change contains
    > > > > "HeadersLength
    > > > > > >     > Headers".
    > > > > > >     > > > Is that
    > > > > > >     > > >     >     intended?
    > > > > > >     > > >     >
    > > > > > >     > > >     >     2. For compressed messageset, is the
    > tombstone
    > > > bit
    > > > > > > only set
    > > > > > >     > at
    > > > > > >     > > > the
    > > > > > >     > > >     > shallow
    > > > > > >     > > >     >     level? Do we always leave that bit in the
    > > wrapper
    > > > > > > message
    > > > > > >     > > unset?
    > > > > > >     > > > An
    > > > > > >     > > >     >     alternative is to set the tombstone bit in
    > the
    > > > > > wrapper
    > > > > > > if at
    > > > > > >     > > > least one
    > > > > > >     > > >     >     inner message has the tombstone bit set. This
    > > > makes
    > > > > > > things a
    > > > > > >     > > bit
    > > > > > >     > > > more
    > > > > > >     > > >     >     complicated, but we could potentially exploit
    > > > that
    > > > > > for
    > > > > > >     > > > optimizing down
    > > > > > >     > > >     >     conversion. For example, we only need to
    > > convert
    > > > > > > messages
    > > > > > >     > with
    > > > > > >     > > > magic 2
    > > > > > >     > > >     > to
    > > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
    > > > > > > (conversion is
    > > > > > >     > > > always
    > > > > > >     > > >     > needed
    > > > > > >     > > >     >     from magic 2 to magic 0). Not sure if the
    > > > > > optimization
    > > > > > > is
    > > > > > >     > worth
    > > > > > >     > > > the
    > > > > > >     > > >     >     complexity though.
    > > > > > >     > > >     >
    > > > > > >     > > >     >     3. The referencing of the new version of
    > > > > > >     > > > ProducerRequest/FetchRequest
    > > > > > >     > > >     > is
    > > > > > >     > > >     >     inconsistent (v4 vs v3). Since our convention
    > > > > starts
    > > > > > at
    > > > > > >     > version
    > > > > > >     > > > at 0, I
    > > > > > >     > > >     >     think the new version would be 3.
    > > > > > >     > > >     >
    > > > > > >     > > >     >     4. "If the magic byte on message is 2, the
    > > broker
    > > > > > > should use
    > > > > > >     > > the
    > > > > > >     > > >     > tombstone
    > > > > > >     > > >     >     bit for log compaction." What about null
    > value?
    > > > My
    > > > > > >     > > understanding
    > > > > > >     > > > is
    > > > > > >     > > >     > that
    > > > > > >     > > >     >     null value will be treated the same as
    > setting
    > > > the
    > > > > > > tombstone
    > > > > > >     > > bit.
    > > > > > >     > > >     >
    > > > > > >     > > >     >     5. For the migration path, it would be useful
    > > to
    > > > > > > describe the
    > > > > > >     > > > down
    > > > > > >     > > >     >     conversion path to consumers (i.e., brokers
    > on
    > > > > > message
    > > > > > > format
    > > > > > >     > > > 0.10.2
    > > > > > >     > > >     > and
    > > > > > >     > > >     >     consumers on older version).
    > > > > > >     > > >     >
    > > > > > >     > > >     >     Thanks,
    > > > > > >     > > >     >
    > > > > > >     > > >     >     Jun
    > > > > > >     > > >     >
    > > > > > >     > > >     >
    > > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael
    > > Pearce <
    > > > > > >     > > > Michael.Pearce@ig.com
    > > > > > >     > > >     > >
    > > > > > >     > > >     >     wrote:
    > > > > > >     > > >     >
    > > > > > >     > > >     >     > Hi All,
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     > We have been discussing in the below thread
    > > and
    > > > > > final
    > > > > > >     > changes
    > > > > > >     > > > have
    > > > > > >     > > >     > been
    > > > > > >     > > >     >     > made to the KIP wiki based on these
    > > > discussions.
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     > We would now like to put to the vote the
    > > > > following
    > > > > > > KIP:
    > > > > > >     > > >     >     > https://cwiki.apache.org/
    > > > > > > confluence/display/KAFKA/KIP-
    > > > > > >     > 87+-+
    > > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     > This kip is for having a distinct
    > compaction
    > > > > > > attribute
    > > > > > >     > > > “tombstone”
    > > > > > >     > > >     > flag
    > > > > > >     > > >     >     > instead of relying on null value, allowing
    > > > > non-null
    > > > > > > value
    > > > > > >     > > > delete
    > > > > > >     > > >     > messages.
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     > Many thanks,
    > > > > > >     > > >     >     > Michael
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
    > > > > > >     > > Michael.Pearce@ig.com>
    > > > > > >     > > >     > wrote:
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >     Hi Mayuresh,
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >     LGTM. Ive just made one small
    > adjustment
    > > > > > > updating the
    > > > > > >     > > wire
    > > > > > >     > > >     > protocol to
    > > > > > >     > > >     >     > show the magic byte bump.
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >     Do we think we’re good to put to a
    > vote?
    > > Is
    > > > > > > there any
    > > > > > >     > > > other bits
    > > > > > >     > > >     >     > needing discussion?
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >     Cheers
    > > > > > >     > > >     >     >     Mike
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
    > Gharat" <
    > > > > > >     > > >     > gharatmayuresh15@gmail.com>
    > > > > > >     > > >     >     > wrote:
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >         Hi Michael,
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >         I have updated the migration
    > section
    > > of
    > > > > the
    > > > > > > KIP.
    > > > > > >     > Can
    > > > > > >     > > > you
    > > > > > >     > > >     > please
    > > > > > >     > > >     >     > take a look?
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >         Thanks,
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >         Mayuresh
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM,
    > > > Mayuresh
    > > > > > > Gharat <
    > > > > > >     > > >     >     > gharatmayuresh15@gmail.com
    > > > > > >     > > >     >     >         > wrote:
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >         > Hi Michael,
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         > That whilst sending tombstone and
    > > non
    > > > > > null
    > > > > > > value,
    > > > > > >     > > the
    > > > > > >     > > >     > consumer
    > > > > > >     > > >     >     > can expect
    > > > > > >     > > >     >     >         > only to receive the non-null
    > > message
    > > > > only
    > > > > > > in step
    > > > > > >     > > > (3) is
    > > > > > >     > > >     > this
    > > > > > >     > > >     >     > correct?
    > > > > > >     > > >     >     >         > ---> I do agree with you here.
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         > Becket, Ismael : can you guys
    > > review
    > > > > the
    > > > > > >     > migration
    > > > > > >     > > > plan
    > > > > > >     > > >     > listed
    > > > > > >     > > >     >     > above using
    > > > > > >     > > >     >     >         > magic byte?
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         > Thanks,
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         > Mayuresh
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
    > > > > Michael
    > > > > > > Pearce <
    > > > > > >     > > >     >     > Michael.Pearce@ig.com>
    > > > > > >     > > >     >     >         > wrote:
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         >> Many thanks for this Mayuresh. I
    > > > don't
    > > > > > > have any
    > > > > > >     > > >     > objections.
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> I assume we should state:
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> That whilst sending tombstone
    > and
    > > > non
    > > > > > null
    > > > > > >     > value,
    > > > > > >     > > > the
    > > > > > >     > > >     > consumer
    > > > > > >     > > >     >     > can expect
    > > > > > >     > > >     >     >         >> only to receive the non-null
    > > message
    > > > > > only
    > > > > > > in
    > > > > > >     > step
    > > > > > >     > > > (3) is
    > > > > > >     > > >     > this
    > > > > > >     > > >     >     > correct?
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Cheers
    > > > > > >     > > >     >     >         >> Mike
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Sent using OWA for iPhone
    > > > > > >     > > >     >     >         >> ______________________________
    > > > > > __________
    > > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
    > > > > > >     > gharatmayuresh15@gmail.com
    > > > > > >     > > >
    > > > > > >     > > >     >     >         >> Sent: Thursday, November 17,
    > 2016
    > > > > > 5:18:41
    > > > > > > PM
    > > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
    > > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 -
    > > Add
    > > > > > > Compaction
    > > > > > >     > > > Tombstone
    > > > > > >     > > >     > Flag
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Hi Ismael,
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Thanks for the explanation.
    > > > > > >     > > >     >     >         >> Specially I like this part where
    > > in
    > > > > you
    > > > > > >     > mentioned
    > > > > > >     > > > we can
    > > > > > >     > > >     > get
    > > > > > >     > > >     >     > rid of the
    > > > > > >     > > >     >     >         >> older null value support for log
    > > > > > > compaction
    > > > > > >     > later
    > > > > > >     > > > on,
    > > > > > >     > > >     > here :
    > > > > > >     > > >     >     >         >> We can't change semantics of the
    > > > > message
    > > > > > > format
    > > > > > >     > > > without
    > > > > > >     > > >     > having
    > > > > > >     > > >     >     > a long
    > > > > > >     > > >     >     >         >> transition period. And we can't
    > > rely
    > > > > > >     > > >     >     >         >> on people reading documentation
    > or
    > > > > > acting
    > > > > > > on a
    > > > > > >     > > > warning for
    > > > > > >     > > >     >     > something so
    > > > > > >     > > >     >     >         >> fundamental. As such, my take is
    > > > that
    > > > > we
    > > > > > > need to
    > > > > > >     > > > bump the
    > > > > > >     > > >     > magic
    > > > > > >     > > >     >     > byte. The
    > > > > > >     > > >     >     >         >> good news is
    > > > > > >     > > >     >     >         >> that we don't have to support
    > all
    > > > > > versions
    > > > > > >     > > forever.
    > > > > > >     > > > We
    > > > > > >     > > >     > have
    > > > > > >     > > >     >     > said that we
    > > > > > >     > > >     >     >         >> will support direct upgrades
    > for 2
    > > > > > years.
    > > > > > > That
    > > > > > >     > > > means that
    > > > > > >     > > >     >     > message format
    > > > > > >     > > >     >     >         >> version n could, in theory, be
    > > > > removed 2
    > > > > > > years
    > > > > > >     > > > after the
    > > > > > >     > > >     > it's
    > > > > > >     > > >     >     > introduced.
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Just a heads up, I would like to
    > > > > mention
    > > > > > > that
    > > > > > >     > even
    > > > > > >     > > > without
    > > > > > >     > > >     >     > bumping magic
    > > > > > >     > > >     >     >         >> byte, we will *NOT* loose zero
    > > copy
    > > > as
    > > > > > in
    > > > > > > the
    > > > > > >     > > > client(x+1)
    > > > > > >     > > >     > in my
    > > > > > >     > > >     >     >         >> explanation
    > > > > > >     > > >     >     >         >> above will convert internally a
    > > null
    > > > > > > value to
    > > > > > >     > > have a
    > > > > > >     > > >     > tombstone
    > > > > > >     > > >     >     > bit set and
    > > > > > >     > > >     >     >         >> a tombstone bit set to have a
    > null
    > > > > value
    > > > > > >     > > > automatically
    > > > > > >     > > >     >     > internally and by
    > > > > > >     > > >     >     >         >> the time we move to version
    > (x+2),
    > > > the
    > > > > > > clients
    > > > > > >     > > > would have
    > > > > > >     > > >     >     > upgraded.
    > > > > > >     > > >     >     >         >> Obviously if we support a
    > request
    > > > from
    > > > > > >     > > consumer(x),
    > > > > > >     > > > we
    > > > > > >     > > >     > will
    > > > > > >     > > >     >     > loose zero
    > > > > > >     > > >     >     >         >> copy
    > > > > > >     > > >     >     >         >> but that is the same case with
    > > magic
    > > > > > byte.
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> But if magic byte bump makes
    > life
    > > > > easier
    > > > > > > for
    > > > > > >     > > > transition
    > > > > > >     > > >     > for the
    > > > > > >     > > >     >     > above
    > > > > > >     > > >     >     >         >> reasons that you explained, I am
    > > OK
    > > > > with
    > > > > > > it
    > > > > > >     > since
    > > > > > >     > > > we are
    > > > > > >     > > >     > going
    > > > > > >     > > >     >     > to meet the
    > > > > > >     > > >     >     >         >> end goal down the road :)
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> On a side note can we update the
    > > doc
    > > > > > here
    > > > > > > on
    > > > > > >     > magic
    > > > > > >     > > > byte
    > > > > > >     > > >     > to say
    > > > > > >     > > >     >     > that "*it
    > > > > > >     > > >     >     >         >> should be bumped whenever the
    > > > message
    > > > > > > format is
    > > > > > >     > > > changed
    > > > > > >     > > >     > or the
    > > > > > >     > > >     >     >         >> interpretation of message format
    > > > > (usage
    > > > > > > of the
    > > > > > >     > > > reserved
    > > > > > >     > > >     > bits as
    > > > > > >     > > >     >     > well) is
    > > > > > >     > > >     >     >         >> changed*".
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Hi Michael,
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Here is the update plan that we
    > > > > > discussed
    > > > > > >     > offline
    > > > > > >     > > >     > yesterday :
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Currently the magic-byte which
    > > > > > > corresponds to
    > > > > > >     > the
    > > > > > >     > > >     >     > "message.format.version"
    > > > > > >     > > >     >     >         >> is set to 1.
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> 1) On broker it will be set to 1
    > > > > > > initially.
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> 2) When a producer client sends
    > a
    > > > > > message
    > > > > > > with
    > > > > > >     > > > magic-byte
    > > > > > >     > > >     > = 2,
    > > > > > >     > > >     >     > since the
    > > > > > >     > > >     >     >         >> broker is on magic-byte = 1, we
    > > will
    > > > > > down
    > > > > > >     > convert
    > > > > > >     > > > it,
    > > > > > >     > > >     > which
    > > > > > >     > > >     >     > means if the
    > > > > > >     > > >     >     >         >> tombstone bit is set, the value
    > > will
    > > > > be
    > > > > > > set to
    > > > > > >     > > > null. A
    > > > > > >     > > >     > consumer
    > > > > > >     > > >     >     >         >> understanding magic-byte = 1,
    > will
    > > > > still
    > > > > > > work
    > > > > > >     > with
    > > > > > >     > > > this. A
    > > > > > >     > > >     >     > consumer
    > > > > > >     > > >     >     >         >> working
    > > > > > >     > > >     >     >         >> with magic-byte =2 will also be
    > > able
    > > > > to
    > > > > > >     > understand
    > > > > > >     > > > this,
    > > > > > >     > > >     > since
    > > > > > >     > > >     >     > it
    > > > > > >     > > >     >     >         >> understands the tombstone.
    > > > > > >     > > >     >     >         >> Now there is still the question
    > of
    > > > > > > supporting a
    > > > > > >     > > >     > non-tombstone
    > > > > > >     > > >     >     > and null
    > > > > > >     > > >     >     >         >> value from producer client with
    > > > > > > magic-byte = 2.*
    > > > > > >     > > (I
    > > > > > >     > > > am
    > > > > > >     > > >     > not sure
    > > > > > >     > > >     >     > if we
    > > > > > >     > > >     >     >         >> should support this.
    > Ismael/Becket
    > > > can
    > > > > > > comment
    > > > > > >     > > > here)*
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> 3) When almost all the clients
    > > have
    > > > > > > upgraded,
    > > > > > >     > the
    > > > > > >     > > >     >     > message.format.version
    > > > > > >     > > >     >     >         >> on
    > > > > > >     > > >     >     >         >> the broker can be changed to 2,
    > > > where
    > > > > in
    > > > > > > the
    > > > > > >     > down
    > > > > > >     > > >     > conversion in
    > > > > > >     > > >     >     > the above
    > > > > > >     > > >     >     >         >> step will not happen. If at this
    > > > point
    > > > > > we
    > > > > > > get a
    > > > > > >     > > > consumer
    > > > > > >     > > >     >     > request from a
    > > > > > >     > > >     >     >         >> older consumer, we might have to
    > > > down
    > > > > > > convert
    > > > > > >     > > where
    > > > > > >     > > > in we
    > > > > > >     > > >     > loose
    > > > > > >     > > >     >     > zero copy,
    > > > > > >     > > >     >     >         >> but these cases should be rare.
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Becket can you review this plan
    > > and
    > > > > add
    > > > > > > more
    > > > > > >     > > > details if I
    > > > > > >     > > >     > have
    > > > > > >     > > >     >     >         >> missed/wronged something, before
    > > we
    > > > > put
    > > > > > > it on
    > > > > > >     > KIP.
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Thanks,
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> Mayuresh
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07
    > PM,
    > > > > > Michael
    > > > > > >     > Pearce <
    > > > > > >     > > >     >     > Michael.Pearce@ig.com>
    > > > > > >     > > >     >     >         >> wrote:
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> > Thanks guys, for discussing
    > this
    > > > > > > offline and
    > > > > > >     > > > getting
    > > > > > >     > > >     > some
    > > > > > >     > > >     >     > consensus.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > So its clear for myself and
    > > others
    > > > > > what
    > > > > > > is
    > > > > > >     > > > proposed now
    > > > > > >     > > >     > (i
    > > > > > >     > > >     >     > think i
    > > > > > >     > > >     >     >         >> > understand, but want to make
    > > sure)
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > Could i ask either directly
    > > update
    > > > > the
    > > > > > > kip to
    > > > > > >     > > > detail the
    > > > > > >     > > >     >     > migration
    > > > > > >     > > >     >     >         >> > strategy, or (re-)state your
    > > > offline
    > > > > > > discussed
    > > > > > >     > > and
    > > > > > >     > > >     > agreed
    > > > > > >     > > >     >     > migration
    > > > > > >     > > >     >     >         >> > strategy based on a magic byte
    > > is
    > > > in
    > > > > > > this
    > > > > > >     > > thread.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > The main original driver for
    > the
    > > > KIP
    > > > > > > was to
    > > > > > >     > > > support
    > > > > > >     > > >     >     > compaction where
    > > > > > >     > > >     >     >         >> value
    > > > > > >     > > >     >     >         >> > isn't null, based off the
    > > > > discussions
    > > > > > on
    > > > > > >     > KIP-82
    > > > > > >     > > > thread.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > We should be able to support
    > > > > > > non-tombstone +
    > > > > > >     > > null
    > > > > > >     > > > value
    > > > > > >     > > >     > by the
    > > > > > >     > > >     >     >         >> completion
    > > > > > >     > > >     >     >         >> > of the KIP, as we noted when
    > > > > > discussing
    > > > > > > this
    > > > > > >     > > kip,
    > > > > > >     > > > having
    > > > > > >     > > >     >     > logic based on
    > > > > > >     > > >     >     >         >> a
    > > > > > >     > > >     >     >         >> > null value isn't very clean
    > and
    > > > also
    > > > > > > separates
    > > > > > >     > > the
    > > > > > >     > > >     > concerns.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > As discussed already though we
    > > can
    > > > > > > split this
    > > > > > >     > > into
    > > > > > >     > > >     > KIP-87a
    > > > > > >     > > >     >     > and KIP-87b
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > Where we look to deliver
    > KIP-87a
    > > > on
    > > > > a
    > > > > > >     > compacted
    > > > > > >     > > > topic
    > > > > > >     > > >     > (to
    > > > > > >     > > >     >     > address the
    > > > > > >     > > >     >     >         >> > immediate issues)
    > > > > > >     > > >     >     >         >> > * tombstone + null value
    > > > > > >     > > >     >     >         >> > * tombstone + non-null value
    > > > > > >     > > >     >     >         >> > * non-tombstone + non-null
    > value
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > Then we can discuss once
    > KIP-87a
    > > > is
    > > > > > > completed
    > > > > > >     > > > options
    > > > > > >     > > >     > later
    > > > > > >     > > >     >     > and how we
    > > > > > >     > > >     >     >         >> > support the second part
    > KIP-87b
    > > to
    > > > > > > deliver:
    > > > > > >     > > >     >     >         >> > * non-tombstone + null value
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > Cheers
    > > > > > >     > > >     >     >         >> > Mike
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > ______________________________
    > > > > > > __________
    > > > > > >     > > >     >     >         >> > From: Becket Qin <
    > > > > > becket.qin@gmail.com>
    > > > > > >     > > >     >     >         >> > Sent: Thursday, November 17,
    > > 2016
    > > > > 1:43
    > > > > > > AM
    > > > > > >     > > >     >     >         >> > To: dev@kafka.apache.org
    > > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87
    > -
    > > > Add
    > > > > > > Compaction
    > > > > > >     > > >     > Tombstone Flag
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had an
    > > > offline
    > > > > > >     > discussion,
    > > > > > >     > > > and
    > > > > > >     > > >     > following
    > > > > > >     > > >     >     > is a brief
    > > > > > >     > > >     >     >         >> > summary.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > 1. We agreed that not bumping
    > up
    > > > > magic
    > > > > > > value
    > > > > > >     > may
    > > > > > >     > > > result
    > > > > > >     > > >     > in
    > > > > > >     > > >     >     > losing zero
    > > > > > >     > > >     >     >         >> copy
    > > > > > >     > > >     >     >         >> > during migration.
    > > > > > >     > > >     >     >         >> > 2. Given that bumping up magic
    > > > value
    > > > > > is
    > > > > > > almost
    > > > > > >     > > > free and
    > > > > > >     > > >     > has
    > > > > > >     > > >     >     > benefit of
    > > > > > >     > > >     >     >         >> > avoiding potential performance
    > > > > issue.
    > > > > > > It is
    > > > > > >     > > > probably
    > > > > > >     > > >     > worth
    > > > > > >     > > >     >     > doing.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > One issue we still need to
    > think
    > > > > about
    > > > > > > is
    > > > > > >     > > whether
    > > > > > >     > > > we
    > > > > > >     > > >     > want to
    > > > > > >     > > >     >     > support a
    > > > > > >     > > >     >     >         >> > non-tombstone message with
    > null
    > > > > value.
    > > > > > >     > > >     >     >         >> > Currently it is not supported
    > by
    > > > > > Kafka.
    > > > > > > If we
    > > > > > >     > > > allow a
    > > > > > >     > > >     >     > non-tombstone null
    > > > > > >     > > >     >     >         >> > value message to exist after
    > > > KIP-87.
    > > > > > The
    > > > > > >     > problem
    > > > > > >     > > > is
    > > > > > >     > > >     > that such
    > > > > > >     > > >     >     > message
    > > > > > >     > > >     >     >         >> will
    > > > > > >     > > >     >     >         >> > not be supported by the
    > > consumers
    > > > > > prior
    > > > > > > to
    > > > > > >     > > KIP-87.
    > > > > > >     > > >     > Because a
    > > > > > >     > > >     >     > null value
    > > > > > >     > > >     >     >         >> > will always be interpreted to
    > a
    > > > > > > tombstone.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > One option is that we keep the
    > > > > current
    > > > > > > way,
    > > > > > >     > i.e.
    > > > > > >     > > > do not
    > > > > > >     > > >     >     > support such
    > > > > > >     > > >     >     >         >> > message. It would be good to
    > > know
    > > > if
    > > > > > > there is
    > > > > > >     > a
    > > > > > >     > > >     > concrete use
    > > > > > >     > > >     >     > case for
    > > > > > >     > > >     >     >         >> such
    > > > > > >     > > >     >     >         >> > message. If there is not, we
    > can
    > > > > > > probably just
    > > > > > >     > > not
    > > > > > >     > > >     > support it.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > Thanks,
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28
    > PM,
    > > > > > > Mayuresh
    > > > > > >     > > Gharat <
    > > > > > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
    > > > > > >     > > >     >     >         >> > > wrote:
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >> > > Hi Ismael,
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > > This is something I can
    > think
    > > of
    > > > > for
    > > > > > >     > migration
    > > > > > >     > > > plan:
    > > > > > >     > > >     >     >         >> > > So the migration plan can
    > look
    > > > > > > something
    > > > > > >     > like
    > > > > > >     > > > this,
    > > > > > >     > > >     > with up
    > > > > > >     > > >     >     >         >> conversion :
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > > 1) Currently lets say we
    > have
    > > > > Broker
    > > > > > > at
    > > > > > >     > > version
    > > > > > >     > > > x.
    > > > > > >     > > >     >     >         >> > > 2) Currently we have clients
    > > at
    > > > > > > version x.
    > > > > > >     > > >     >     >         >> > > 3) a) We move the version to
    > > > > > > Broker(x+1) :
    > > > > > >     > > > supports
    > > > > > >     > > >     > both
    > > > > > >     > > >     >     > tombstone and
    > > > > > >     > > >     >     >         >> > null
    > > > > > >     > > >     >     >         >> > > for log compaction.
    > > > > > >     > > >     >     >         >> > >     b) We upgrade the client
    > > to
    > > > > > > version
    > > > > > >     > > > client(x+1) :
    > > > > > >     > > >     > if in
    > > > > > >     > > >     >     > the
    > > > > > >     > > >     >     >         >> producer
    > > > > > >     > > >     >     >         >> > > client(x+1) the value is set
    > > to
    > > > > > null,
    > > > > > > we
    > > > > > >     > will
    > > > > > >     > > >     > automatically
    > > > > > >     > > >     >     > set the
    > > > > > >     > > >     >     >         >> > > Tombstone bit internally. If
    > > the
    > > > > > > producer
    > > > > > >     > > > client(x+1)
    > > > > > >     > > >     > sets
    > > > > > >     > > >     >     > the
    > > > > > >     > > >     >     >         >> tombstone
    > > > > > >     > > >     >     >         >> > > itself, well and good. For
    > > > > producer
    > > > > > >     > client(x),
    > > > > > >     > > > the
    > > > > > >     > > >     > broker
    > > > > > >     > > >     >     > will up
    > > > > > >     > > >     >     >         >> convert
    > > > > > >     > > >     >     >         >> > > to have the tombstone bit.
    > > > > > > Broker(x+1) is
    > > > > > >     > > > supporting
    > > > > > >     > > >     > both.
    > > > > > >     > > >     >     > Consumer
    > > > > > >     > > >     >     >         >> > > client(x+1) will be aware of
    > > > this
    > > > > > and
    > > > > > > should
    > > > > > >     > > be
    > > > > > >     > > > able
    > > > > > >     > > >     > to
    > > > > > >     > > >     >     > handle this.
    > > > > > >     > > >     >     >         >> For
    > > > > > >     > > >     >     >         >> > > consumer client(x) we will
    > > down
    > > > > > > convert the
    > > > > > >     > > > message
    > > > > > >     > > >     > on the
    > > > > > >     > > >     >     > broker
    > > > > > >     > > >     >     >         >> side.
    > > > > > >     > > >     >     >         >> > >     c) At this point we will
    > > > have
    > > > > to
    > > > > > >     > specify a
    > > > > > >     > > >     > warning or
    > > > > > >     > > >     >     > clearly
    > > > > > >     > > >     >     >         >> specify
    > > > > > >     > > >     >     >         >> > > in docs that this behavior
    > is
    > > > > about
    > > > > > > to be
    > > > > > >     > > > changed for
    > > > > > >     > > >     > log
    > > > > > >     > > >     >     > compaction.
    > > > > > >     > > >     >     >         >> > > 4) a) In next release of the
    > > > > > > Broker(x+2), we
    > > > > > >     > > > say that
    > > > > > >     > > >     > only
    > > > > > >     > > >     >     > Tombstone
    > > > > > >     > > >     >     >         >> is
    > > > > > >     > > >     >     >         >> > > used for log compaction on
    > the
    > > > > > Broker
    > > > > > > side.
    > > > > > >     > > >     > Clients(x+1)
    > > > > > >     > > >     >     > still is
    > > > > > >     > > >     >     >         >> > > supported.
    > > > > > >     > > >     >     >         >> > >     b) We upgrade the client
    > > to
    > > > > > > version
    > > > > > >     > > > client(x+2) :
    > > > > > >     > > >     > if
    > > > > > >     > > >     >     > value is set
    > > > > > >     > > >     >     >         >> to
    > > > > > >     > > >     >     >         >> > > null, tombstone will not be
    > > set
    > > > > > >     > automatically.
    > > > > > >     > > > The
    > > > > > >     > > >     > client
    > > > > > >     > > >     >     > will have to
    > > > > > >     > > >     >     >         >> > call
    > > > > > >     > > >     >     >         >> > > setTombstone() to actually
    > set
    > > > the
    > > > > > >     > tombstone.
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > > We should compare this
    > > migration
    > > > > > plan
    > > > > > > with
    > > > > > >     > the
    > > > > > >     > > >     > migration
    > > > > > >     > > >     >     > plan for
    > > > > > >     > > >     >     >         >> magic
    > > > > > >     > > >     >     >         >> > > byte bump and do whatever
    > > looks
    > > > > > good.
    > > > > > >     > > >     >     >         >> > > I am just worried that if we
    > > go
    > > > > down
    > > > > > > magic
    > > > > > >     > > byte
    > > > > > >     > > > route,
    > > > > > >     > > >     >     > unless I am
    > > > > > >     > > >     >     >         >> > missing
    > > > > > >     > > >     >     >         >> > > something, it sounds like
    > > kafka
    > > > > will
    > > > > > > be
    > > > > > >     > stuck
    > > > > > >     > > > with
    > > > > > >     > > >     >     > supporting both
    > > > > > >     > > >     >     >         >> null
    > > > > > >     > > >     >     >         >> > > value and tombstone bit for
    > > log
    > > > > > > compaction
    > > > > > >     > for
    > > > > > >     > > > life
    > > > > > >     > > >     > long,
    > > > > > >     > > >     >     > which does
    > > > > > >     > > >     >     >         >> not
    > > > > > >     > > >     >     >         >> > > look like a good end state.
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > > Thanks,
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > > Mayuresh
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32
    > > AM,
    > > > > > > Mayuresh
    > > > > > >     > > > Gharat <
    > > > > > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
    > > > > > >     > > >     >     >         >> > > > wrote:
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > > > Hi Ismael,
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > That's a very good point
    > > > which I
    > > > > > > might
    > > > > > >     > have
    > > > > > >     > > > not
    > > > > > >     > > >     >     > considered earlier.
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > Here is a plan that I can
    > > > think
    > > > > > of:
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > Stage 1) The broker from
    > now
    > > > on,
    > > > > > up
    > > > > > >     > converts
    > > > > > >     > > > the
    > > > > > >     > > >     > message
    > > > > > >     > > >     >     > to have the
    > > > > > >     > > >     >     >         >> > > > tombstone marker. The log
    > > > > > compaction
    > > > > > >     > thread
    > > > > > >     > > > does log
    > > > > > >     > > >     >     > compaction
    > > > > > >     > > >     >     >         >> based
    > > > > > >     > > >     >     >         >> > on
    > > > > > >     > > >     >     >         >> > > > both null and tombstone
    > > > marker.
    > > > > > > This is
    > > > > > >     > our
    > > > > > >     > > >     > transition
    > > > > > >     > > >     >     > period.
    > > > > > >     > > >     >     >         >> > > > Stage 2) The next release
    > we
    > > > > only
    > > > > > > say that
    > > > > > >     > > log
    > > > > > >     > > >     > compaction
    > > > > > >     > > >     >     > is based
    > > > > > >     > > >     >     >         >> on
    > > > > > >     > > >     >     >         >> > > > tombstone marker. (Open
    > > source
    > > > > > > kafka makes
    > > > > > >     > > > this as a
    > > > > > >     > > >     >     > policy). By
    > > > > > >     > > >     >     >         >> this
    > > > > > >     > > >     >     >         >> > > time,
    > > > > > >     > > >     >     >         >> > > > the organization which is
    > > > moving
    > > > > > to
    > > > > > > this
    > > > > > >     > > > release
    > > > > > >     > > >     > will be
    > > > > > >     > > >     >     > sure that
    > > > > > >     > > >     >     >         >> they
    > > > > > >     > > >     >     >         >> > > > have gone through the
    > entire
    > > > > > > transition
    > > > > > >     > > > period.
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > My only goal of doing this
    > > is
    > > > > that
    > > > > > > Kafka
    > > > > > >     > > > clearly
    > > > > > >     > > >     >     > specifies the end
    > > > > > >     > > >     >     >         >> > state
    > > > > > >     > > >     >     >         >> > > > about what log compaction
    > > > means
    > > > > > (is
    > > > > > > it
    > > > > > >     > null
    > > > > > >     > > > value
    > > > > > >     > > >     > or a
    > > > > > >     > > >     >     > tombstone
    > > > > > >     > > >     >     >         >> > marker,
    > > > > > >     > > >     >     >         >> > > > but not both).
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > What do you think?
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > Thanks,
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > Mayuresh
    > > > > > >     > > >     >     >         >> > > > .
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at
    > 9:17
    > > > AM,
    > > > > > > Ismael
    > > > > > >     > > Juma <
    > > > > > >     > > >     >     > ismael@juma.me.uk>
    > > > > > >     > > >     >     >         >> > wrote:
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > >> One comment below.
    > > > > > >     > > >     >     >         >> > > >>
    > > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at
    > > 5:08
    > > > > PM,
    > > > > > > Mayuresh
    > > > > > >     > > > Gharat <
    > > > > > >     > > >     >     >         >> > > >>
    > gharatmayuresh15@gmail.com
    > > > > > >     > > >     >     >         >> > > >> > wrote:
    > > > > > >     > > >     >     >         >> > > >>
    > > > > > >     > > >     >     >         >> > > >> >    - If we don't bump
    > up
    > > > the
    > > > > > > magic
    > > > > > >     > byte,
    > > > > > >     > > > on the
    > > > > > >     > > >     > broker
    > > > > > >     > > >     >     > side, the
    > > > > > >     > > >     >     >         >> > > broker
    > > > > > >     > > >     >     >         >> > > >> >    will always have to
    > > look
    > > > > at
    > > > > > > both
    > > > > > >     > > > tombstone
    > > > > > >     > > >     > bit and
    > > > > > >     > > >     >     > the value
    > > > > > >     > > >     >     >         >> when
    > > > > > >     > > >     >     >         >> > > do
    > > > > > >     > > >     >     >         >> > > >> the
    > > > > > >     > > >     >     >         >> > > >> >    compaction. Assuming
    > > we
    > > > do
    > > > > > > not bump
    > > > > > >     > up
    > > > > > >     > > > the
    > > > > > >     > > >     > magic
    > > > > > >     > > >     >     > byte,
    > > > > > >     > > >     >     >         >> > > >> >    imagine the broker
    > > sees
    > > > a
    > > > > > > message
    > > > > > >     > > which
    > > > > > >     > > > does
    > > > > > >     > > >     > not
    > > > > > >     > > >     >     > have a
    > > > > > >     > > >     >     >         >> tombstone
    > > > > > >     > > >     >     >         >> > > bit
    > > > > > >     > > >     >     >         >> > > >> >    set. The broker does
    > > not
    > > > > > know
    > > > > > > when
    > > > > > >     > the
    > > > > > >     > > >     > message was
    > > > > > >     > > >     >     > produced
    > > > > > >     > > >     >     >         >> (i.e.
    > > > > > >     > > >     >     >         >> > > >> > whether
    > > > > > >     > > >     >     >         >> > > >> >    the message has been
    > > up
    > > > > > > converted or
    > > > > > >     > > > not), it
    > > > > > >     > > >     > has
    > > > > > >     > > >     >     > to take a
    > > > > > >     > > >     >     >         >> > further
    > > > > > >     > > >     >     >         >> > > >> > look at
    > > > > > >     > > >     >     >         >> > > >> >    the value to see if
    > it
    > > > is
    > > > > > > null or
    > > > > > >     > not
    > > > > > >     > > in
    > > > > > >     > > >     > order to
    > > > > > >     > > >     >     > determine
    > > > > > >     > > >     >     >         >> if it
    > > > > > >     > > >     >     >         >> > > is
    > > > > > >     > > >     >     >         >> > > >> a
    > > > > > >     > > >     >     >         >> > > >> >    tombstone. The same
    > > > logic
    > > > > > has
    > > > > > > to be
    > > > > > >     > > put
    > > > > > >     > > > on the
    > > > > > >     > > >     >     > consumer as
    > > > > > >     > > >     >     >         >> well
    > > > > > >     > > >     >     >         >> > > >> because
    > > > > > >     > > >     >     >         >> > > >> > the
    > > > > > >     > > >     >     >         >> > > >> >    consumer does not
    > know
    > > > if
    > > > > > the
    > > > > > >     > message
    > > > > > >     > > > has
    > > > > > >     > > >     > been up
    > > > > > >     > > >     >     > converted or
    > > > > > >     > > >     >     >         >> > not.
    > > > > > >     > > >     >     >         >> > > >> >       - If we upconvert
    > > > while
    > > > > > >     > appending,
    > > > > > >     > > > this is
    > > > > > >     > > >     > not
    > > > > > >     > > >     >     > the case,
    > > > > > >     > > >     >     >         >> > right?
    > > > > > >     > > >     >     >         >> > > >>
    > > > > > >     > > >     >     >         >> > > >>
    > > > > > >     > > >     >     >         >> > > >> If I understand you
    > > > correctly,
    > > > > > > this is
    > > > > > >     > not
    > > > > > >     > > >     > sufficient
    > > > > > >     > > >     >     > because the
    > > > > > >     > > >     >     >         >> log
    > > > > > >     > > >     >     >         >> > > may
    > > > > > >     > > >     >     >         >> > > >> have messages appended
    > > before
    > > > > it
    > > > > > > was
    > > > > > >     > > > upgraded to
    > > > > > >     > > >     > include
    > > > > > >     > > >     >     > KIP-87.
    > > > > > >     > > >     >     >         >> > > >>
    > > > > > >     > > >     >     >         >> > > >> Ismael
    > > > > > >     > > >     >     >         >> > > >>
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > > > --
    > > > > > >     > > >     >     >         >> > > > -Regards,
    > > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
    > > > > > >     > > >     >     >         >> > > > (862) 250-7125
    > > > > > >     > > >     >     >         >> > > >
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > > --
    > > > > > >     > > >     >     >         >> > > -Regards,
    > > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
    > > > > > >     > > >     >     >         >> > > (862) 250-7125
    > > > > > >     > > >     >     >         >> > >
    > > > > > >     > > >     >     >         >> > The information contained in
    > > this
    > > > > > email
    > > > > > > is
    > > > > > >     > > > strictly
    > > > > > >     > > >     >     > confidential and for
    > > > > > >     > > >     >     >         >> > the use of the addressee only,
    > > > > unless
    > > > > > >     > otherwise
    > > > > > >     > > >     > indicated. If
    > > > > > >     > > >     >     > you are
    > > > > > >     > > >     >     >         >> not
    > > > > > >     > > >     >     >         >> > the intended recipient, please
    > > do
    > > > > not
    > > > > > > read,
    > > > > > >     > > copy,
    > > > > > >     > > > use or
    > > > > > >     > > >     >     > disclose to
    > > > > > >     > > >     >     >         >> others
    > > > > > >     > > >     >     >         >> > this message or any
    > attachment.
    > > > > Please
    > > > > > > also
    > > > > > >     > > > notify the
    > > > > > >     > > >     > sender
    > > > > > >     > > >     >     > by
    > > > > > >     > > >     >     >         >> replying
    > > > > > >     > > >     >     >         >> > to this email or by telephone
    > > > > (+44(020
    > > > > > > 7896
    > > > > > >     > > 0011)
    > > > > > >     > > > and
    > > > > > >     > > >     > then
    > > > > > >     > > >     >     > delete the
    > > > > > >     > > >     >     >         >> email
    > > > > > >     > > >     >     >         >> > and any copies of it.
    > Opinions,
    > > > > > > conclusion
    > > > > > >     > (etc)
    > > > > > >     > > > that
    > > > > > >     > > >     > do not
    > > > > > >     > > >     >     > relate to
    > > > > > >     > > >     >     >         >> the
    > > > > > >     > > >     >     >         >> > official business of this
    > > company
    > > > > > shall
    > > > > > > be
    > > > > > >     > > > understood as
    > > > > > >     > > >     >     > neither given
    > > > > > >     > > >     >     >         >> nor
    > > > > > >     > > >     >     >         >> > endorsed by it. IG is a
    > trading
    > > > name
    > > > > > of
    > > > > > > IG
    > > > > > >     > > Markets
    > > > > > >     > > >     > Limited (a
    > > > > > >     > > >     >     > company
    > > > > > >     > > >     >     >         >> > registered in England and
    > Wales,
    > > > > > company
    > > > > > >     > number
    > > > > > >     > > >     > 04008957) and
    > > > > > >     > > >     >     > IG Index
    > > > > > >     > > >     >     >         >> > Limited (a company registered
    > in
    > > > > > > England and
    > > > > > >     > > > Wales,
    > > > > > >     > > >     > company
    > > > > > >     > > >     >     > number
    > > > > > >     > > >     >     >         >> > 01190902). Registered address
    > at
    > > > > > Cannon
    > > > > > > Bridge
    > > > > > >     > > > House, 25
    > > > > > >     > > >     >     > Dowgate Hill,
    > > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
    > Markets
    > > > > > Limited
    > > > > > >     > > (register
    > > > > > >     > > >     > number
    > > > > > >     > > >     >     > 195355) and IG
    > > > > > >     > > >     >     >         >> > Index Limited (register number
    > > > > 114059)
    > > > > > > are
    > > > > > >     > > > authorised
    > > > > > >     > > >     > and
    > > > > > >     > > >     >     > regulated by
    > > > > > >     > > >     >     >         >> the
    > > > > > >     > > >     >     >         >> > Financial Conduct Authority.
    > > > > > >     > > >     >     >         >> >
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >> --
    > > > > > >     > > >     >     >         >> -Regards,
    > > > > > >     > > >     >     >         >> Mayuresh R. Gharat
    > > > > > >     > > >     >     >         >> (862) 250-7125
    > > > > > >     > > >     >     >         >> The information contained in
    > this
    > > > > email
    > > > > > is
    > > > > > >     > > strictly
    > > > > > >     > > >     >     > confidential and for
    > > > > > >     > > >     >     >         >> the use of the addressee only,
    > > > unless
    > > > > > > otherwise
    > > > > > >     > > >     > indicated. If
    > > > > > >     > > >     >     > you are not
    > > > > > >     > > >     >     >         >> the intended recipient, please
    > do
    > > > not
    > > > > > > read,
    > > > > > >     > copy,
    > > > > > >     > > > use or
    > > > > > >     > > >     >     > disclose to others
    > > > > > >     > > >     >     >         >> this message or any attachment.
    > > > Please
    > > > > > > also
    > > > > > >     > notify
    > > > > > >     > > > the
    > > > > > >     > > >     > sender
    > > > > > >     > > >     >     > by replying
    > > > > > >     > > >     >     >         >> to this email or by telephone
    > > > (+44(020
    > > > > > > 7896
    > > > > > >     > 0011)
    > > > > > >     > > > and then
    > > > > > >     > > >     >     > delete the email
    > > > > > >     > > >     >     >         >> and any copies of it. Opinions,
    > > > > > > conclusion (etc)
    > > > > > >     > > > that do
    > > > > > >     > > >     > not
    > > > > > >     > > >     >     > relate to the
    > > > > > >     > > >     >     >         >> official business of this
    > company
    > > > > shall
    > > > > > be
    > > > > > >     > > > understood as
    > > > > > >     > > >     >     > neither given nor
    > > > > > >     > > >     >     >         >> endorsed by it. IG is a trading
    > > name
    > > > > of
    > > > > > IG
    > > > > > >     > Markets
    > > > > > >     > > >     > Limited (a
    > > > > > >     > > >     >     > company
    > > > > > >     > > >     >     >         >> registered in England and Wales,
    > > > > company
    > > > > > > number
    > > > > > >     > > > 04008957)
    > > > > > >     > > >     > and
    > > > > > >     > > >     >     > IG Index
    > > > > > >     > > >     >     >         >> Limited (a company registered in
    > > > > England
    > > > > > > and
    > > > > > >     > > Wales,
    > > > > > >     > > >     > company
    > > > > > >     > > >     >     > number
    > > > > > >     > > >     >     >         >> 01190902). Registered address at
    > > > > Cannon
    > > > > > > Bridge
    > > > > > >     > > > House, 25
    > > > > > >     > > >     >     > Dowgate Hill,
    > > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
    > > > > Limited
    > > > > > >     > (register
    > > > > > >     > > > number
    > > > > > >     > > >     >     > 195355) and IG
    > > > > > >     > > >     >     >         >> Index Limited (register number
    > > > 114059)
    > > > > > are
    > > > > > >     > > > authorised and
    > > > > > >     > > >     >     > regulated by the
    > > > > > >     > > >     >     >         >> Financial Conduct Authority.
    > > > > > >     > > >     >     >         >>
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >         > --
    > > > > > >     > > >     >     >         > -Regards,
    > > > > > >     > > >     >     >         > Mayuresh R. Gharat
    > > > > > >     > > >     >     >         > (862) 250-7125
    > > > > > >     > > >     >     >         >
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >         --
    > > > > > >     > > >     >     >         -Regards,
    > > > > > >     > > >     >     >         Mayuresh R. Gharat
    > > > > > >     > > >     >     >         (862) 250-7125
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >     The information contained in this email
    > > is
    > > > > > > strictly
    > > > > > >     > > > confidential
    > > > > > >     > > >     > and
    > > > > > >     > > >     >     > for the use of the addressee only, unless
    > > > > otherwise
    > > > > > >     > > indicated.
    > > > > > >     > > > If
    > > > > > >     > > >     > you are
    > > > > > >     > > >     >     > not the intended recipient, please do not
    > > read,
    > > > > > > copy, use
    > > > > > >     > or
    > > > > > >     > > >     > disclose to
    > > > > > >     > > >     >     > others this message or any attachment.
    > Please
    > > > > also
    > > > > > > notify
    > > > > > >     > the
    > > > > > >     > > > sender
    > > > > > >     > > >     > by
    > > > > > >     > > >     >     > replying to this email or by telephone
    > > (+44(020
    > > > > > 7896
    > > > > > > 0011)
    > > > > > >     > > and
    > > > > > >     > > > then
    > > > > > >     > > >     > delete
    > > > > > >     > > >     >     > the email and any copies of it. Opinions,
    > > > > > conclusion
    > > > > > > (etc)
    > > > > > >     > > > that do
    > > > > > >     > > >     > not
    > > > > > >     > > >     >     > relate to the official business of this
    > > company
    > > > > > > shall be
    > > > > > >     > > > understood
    > > > > > >     > > >     > as
    > > > > > >     > > >     >     > neither given nor endorsed by it. IG is a
    > > > trading
    > > > > > > name of
    > > > > > >     > IG
    > > > > > >     > > > Markets
    > > > > > >     > > >     >     > Limited (a company registered in England
    > and
    > > > > Wales,
    > > > > > > company
    > > > > > >     > > > number
    > > > > > >     > > >     >     > 04008957) and IG Index Limited (a company
    > > > > > registered
    > > > > > > in
    > > > > > >     > > > England and
    > > > > > >     > > >     > Wales,
    > > > > > >     > > >     >     > company number 01190902). Registered
    > address
    > > at
    > > > > > > Cannon
    > > > > > >     > Bridge
    > > > > > >     > > > House,
    > > > > > >     > > >     > 25
    > > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG
    > > Markets
    > > > > > > Limited
    > > > > > >     > > > (register
    > > > > > >     > > >     > number
    > > > > > >     > > >     >     > 195355) and IG Index Limited (register
    > number
    > > > > > > 114059) are
    > > > > > >     > > > authorised
    > > > > > >     > > >     > and
    > > > > > >     > > >     >     > regulated by the Financial Conduct
    > Authority.
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >     >
    > > > > > >     > > >     >
    > > > > > >     > > >     >
    > > > > > >     > > >     > The information contained in this email is
    > strictly
    > > > > > > confidential
    > > > > > >     > > and
    > > > > > >     > > > for
    > > > > > >     > > >     > the use of the addressee only, unless otherwise
    > > > > > indicated.
    > > > > > > If you
    > > > > > >     > > > are not
    > > > > > >     > > >     > the intended recipient, please do not read, copy,
    > > use
    > > > > or
    > > > > > > disclose
    > > > > > >     > > to
    > > > > > >     > > > others
    > > > > > >     > > >     > this message or any attachment. Please also
    > notify
    > > > the
    > > > > > > sender by
    > > > > > >     > > > replying
    > > > > > >     > > >     > to this email or by telephone (+44(020 7896 0011
    > > <020%207896%200011>) and
    > > > > > then
    > > > > > > delete
    > > > > > >     > > > the email
    > > > > > >     > > >     > and any copies of it. Opinions, conclusion (etc)
    > > that
    > > > > do
    > > > > > > not
    > > > > > >     > relate
    > > > > > >     > > > to the
    > > > > > >     > > >     > official business of this company shall be
    > > understood
    > > > > as
    > > > > > > neither
    > > > > > >     > > > given nor
    > > > > > >     > > >     > endorsed by it. IG is a trading name of IG
    > Markets
    > > > > > Limited
    > > > > > > (a
    > > > > > >     > > company
    > > > > > >     > > >     > registered in England and Wales, company number
    > > > > 04008957)
    > > > > > > and IG
    > > > > > >     > > > Index
    > > > > > >     > > >     > Limited (a company registered in England and
    > Wales,
    > > > > > company
    > > > > > >     > number
    > > > > > >     > > >     > 01190902). Registered address at Cannon Bridge
    > > House,
    > > > > 25
    > > > > > > Dowgate
    > > > > > >     > > > Hill,
    > > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
    > (register
    > > > > number
    > > > > > > 195355)
    > > > > > >     > > > and IG
    > > > > > >     > > >     > Index Limited (register number 114059) are
    > > authorised
    > > > > and
    > > > > > >     > regulated
    > > > > > >     > > > by the
    > > > > > >     > > >     > Financial Conduct Authority.
    > > > > > >     > > >     >
    > > > > > >     > > >
    > > > > > >     > > >
    > > > > > >     > > > The information contained in this email is strictly
    > > > > > confidential
    > > > > > > and
    > > > > > >     > for
    > > > > > >     > > > the use of the addressee only, unless otherwise
    > > indicated.
    > > > If
    > > > > > > you are
    > > > > > >     > not
    > > > > > >     > > > the intended recipient, please do not read, copy, use
    > or
    > > > > > > disclose to
    > > > > > >     > > others
    > > > > > >     > > > this message or any attachment. Please also notify the
    > > > sender
    > > > > > by
    > > > > > >     > replying
    > > > > > >     > > > to this email or by telephone (+44(020 7896 0011
    > > <020%207896%200011>) and then
    > > > > > > delete the
    > > > > > >     > > email
    > > > > > >     > > > and any copies of it. Opinions, conclusion (etc) that
    > do
    > > > not
    > > > > > > relate to
    > > > > > >     > > the
    > > > > > >     > > > official business of this company shall be understood
    > as
    > > > > > neither
    > > > > > > given
    > > > > > >     > > nor
    > > > > > >     > > > endorsed by it. IG is a trading name of IG Markets
    > > Limited
    > > > (a
    > > > > > > company
    > > > > > >     > > > registered in England and Wales, company number
    > 04008957)
    > > > and
    > > > > > IG
    > > > > > > Index
    > > > > > >     > > > Limited (a company registered in England and Wales,
    > > company
    > > > > > > number
    > > > > > >     > > > 01190902). Registered address at Cannon Bridge House,
    > 25
    > > > > > Dowgate
    > > > > > > Hill,
    > > > > > >     > > > London EC4R 2YA. Both IG Markets Limited (register
    > number
    > > > > > > 195355) and
    > > > > > >     > IG
    > > > > > >     > > > Index Limited (register number 114059) are authorised
    > and
    > > > > > > regulated by
    > > > > > >     > > the
    > > > > > >     > > > Financial Conduct Authority.
    > > > > > >     > > >
    > > > > > >     > > The information contained in this email is strictly
    > > > > confidential
    > > > > > > and for
    > > > > > >     > > the use of the addressee only, unless otherwise
    > indicated.
    > > If
    > > > > you
    > > > > > > are not
    > > > > > >     > > the intended recipient, please do not read, copy, use or
    > > > > disclose
    > > > > > > to
    > > > > > >     > others
    > > > > > >     > > this message or any attachment. Please also notify the
    > > sender
    > > > > by
    > > > > > > replying
    > > > > > >     > > to this email or by telephone (+44(020 7896 0011
    > > <020%207896%200011>) and then
    > > > > delete
    > > > > > > the
    > > > > > >     > email
    > > > > > >     > > and any copies of it. Opinions, conclusion (etc) that do
    > > not
    > > > > > > relate to
    > > > > > >     > the
    > > > > > >     > > official business of this company shall be understood as
    > > > > neither
    > > > > > > given
    > > > > > >     > nor
    > > > > > >     > > endorsed by it. IG is a trading name of IG Markets
    > Limited
    > > (a
    > > > > > > company
    > > > > > >     > > registered in England and Wales, company number 04008957)
    > > and
    > > > > IG
    > > > > > > Index
    > > > > > >     > > Limited (a company registered in England and Wales,
    > company
    > > > > > number
    > > > > > >     > > 01190902). Registered address at Cannon Bridge House, 25
    > > > > Dowgate
    > > > > > > Hill,
    > > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
    > > > > 195355)
    > > > > > > and IG
    > > > > > >     > > Index Limited (register number 114059) are authorised and
    > > > > > > regulated by
    > > > > > >     > the
    > > > > > >     > > Financial Conduct Authority.
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > The information contained in this email is strictly
    > > > confidential
    > > > > > and
    > > > > > > for
    > > > > > >     > the use of the addressee only, unless otherwise indicated.
    > If
    > > > you
    > > > > > > are not
    > > > > > >     > the intended recipient, please do not read, copy, use or
    > > > disclose
    > > > > > to
    > > > > > > others
    > > > > > >     > this message or any attachment. Please also notify the
    > sender
    > > > by
    > > > > > > replying
    > > > > > >     > to this email or by telephone (+44(020 7896 0011
    > > <020%207896%200011>) and then
    > > > delete
    > > > > > > the email
    > > > > > >     > and any copies of it. Opinions, conclusion (etc) that do
    > not
    > > > > relate
    > > > > > > to the
    > > > > > >     > official business of this company shall be understood as
    > > > neither
    > > > > > > given nor
    > > > > > >     > endorsed by it. IG is a trading name of IG Markets Limited
    > (a
    > > > > > company
    > > > > > >     > registered in England and Wales, company number 04008957)
    > and
    > > > IG
    > > > > > > Index
    > > > > > >     > Limited (a company registered in England and Wales, company
    > > > > number
    > > > > > >     > 01190902). Registered address at Cannon Bridge House, 25
    > > > Dowgate
    > > > > > > Hill,
    > > > > > >     > London EC4R 2YA. Both IG Markets Limited (register number
    > > > 195355)
    > > > > > > and IG
    > > > > > >     > Index Limited (register number 114059) are authorised and
    > > > > regulated
    > > > > > > by the
    > > > > > >     > Financial Conduct Authority.
    > > > > > >     >
    > > > > > >     >
    > > > > > >
    > > > > > >
    > > > > > > The information contained in this email is strictly confidential
    > > and
    > > > > for
    > > > > > > the use of the addressee only, unless otherwise indicated. If you
    > > are
    > > > > not
    > > > > > > the intended recipient, please do not read, copy, use or disclose
    > > to
    > > > > > others
    > > > > > > this message or any attachment. Please also notify the sender by
    > > > > replying
    > > > > > > to this email or by telephone (+44(020 7896 0011
    > > <020%207896%200011>) and then delete the
    > > > > > email
    > > > > > > and any copies of it. Opinions, conclusion (etc) that do not
    > relate
    > > > to
    > > > > > the
    > > > > > > official business of this company shall be understood as neither
    > > > given
    > > > > > nor
    > > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
    > > company
    > > > > > > registered in England and Wales, company number 04008957) and IG
    > > > Index
    > > > > > > Limited (a company registered in England and Wales, company
    > number
    > > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > > > Hill,
    > > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
    > > and
    > > > > IG
    > > > > > > Index Limited (register number 114059) are authorised and
    > regulated
    > > > by
    > > > > > the
    > > > > > > Financial Conduct Authority.
    > > > > > >
    > > > > > The information contained in this email is strictly confidential
    > and
    > > > for
    > > > > > the use of the addressee only, unless otherwise indicated. If you
    > are
    > > > not
    > > > > > the intended recipient, please do not read, copy, use or disclose
    > to
    > > > > others
    > > > > > this message or any attachment. Please also notify the sender by
    > > > replying
    > > > > > to this email or by telephone (+44(020 7896 0011
    > <020%207896%200011>)
    > > and then delete the
    > > > > email
    > > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
    > > to
    > > > > the
    > > > > > official business of this company shall be understood as neither
    > > given
    > > > > nor
    > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
    > company
    > > > > > registered in England and Wales, company number 04008957) and IG
    > > Index
    > > > > > Limited (a company registered in England and Wales, company number
    > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > > Hill,
    > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
    > and
    > > > IG
    > > > > > Index Limited (register number 114059) are authorised and regulated
    > > by
    > > > > the
    > > > > > Financial Conduct Authority.
    > > > > >
    > > > > The information contained in this email is strictly confidential and
    > > for
    > > > > the use of the addressee only, unless otherwise indicated. If you are
    > > not
    > > > > the intended recipient, please do not read, copy, use or disclose to
    > > > others
    > > > > this message or any attachment. Please also notify the sender by
    > > replying
    > > > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
    > > and then delete the
    > > > email
    > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
    > to
    > > > the
    > > > > official business of this company shall be understood as neither
    > given
    > > > nor
    > > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > > > registered in England and Wales, company number 04008957) and IG
    > Index
    > > > > Limited (a company registered in England and Wales, company number
    > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > Hill,
    > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    > > IG
    > > > > Index Limited (register number 114059) are authorised and regulated
    > by
    > > > the
    > > > > Financial Conduct Authority.
    > > > >
    > > > The information contained in this email is strictly confidential and
    > for
    > > > the use of the addressee only, unless otherwise indicated. If you are
    > not
    > > > the intended recipient, please do not read, copy, use or disclose to
    > > others
    > > > this message or any attachment. Please also notify the sender by
    > replying
    > > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
    > > and then delete the email
    > > > and any copies of it. Opinions, conclusion (etc) that do not relate to
    > > the
    > > > official business of this company shall be understood as neither given
    > > nor
    > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > > registered in England and Wales, company number 04008957) and IG Index
    > > > Limited (a company registered in England and Wales, company number
    > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    > IG
    > > > Index Limited (register number 114059) are authorised and regulated by
    > > the
    > > > Financial Conduct Authority.
    > > >
    > >
    > The information contained in this email is strictly confidential and for
    > the use of the addressee only, unless otherwise indicated. If you are not
    > the intended recipient, please do not read, copy, use or disclose to others
    > this message or any attachment. Please also notify the sender by replying
    > to this email or by telephone (+44(020 7896 0011) and then delete the email
    > and any copies of it. Opinions, conclusion (etc) that do not relate to the
    > official business of this company shall be understood as neither given nor
    > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > registered in England and Wales, company number 04008957) and IG Index
    > Limited (a company registered in England and Wales, company number
    > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    > Index Limited (register number 114059) are authorised and regulated by the
    > Financial Conduct Authority.
    >


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Ismael Juma <is...@juma.me.uk>]

Hi Michael,

I am not sure about using a new policy. One other option (suggested by
Gwen) is that the behaviour of the producer with regards to tombstones when
you pass a `null` value and don't use the constructor that takes a
tombstone can be influenced by a producer config. Not ideal, but would
limit the complexity of maintaining compatibility while providing a path
for the (what I think) are the right semantics.

Ismael

On Tue, Dec 13, 2016 at 8:32 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Ismael
>
> Did you see our email this morning, what's your thoughts on this approach
> to instead we simply have a brand new policy?
>
> Cheers
> Mike
>
>
> Sent using OWA for iPhone
> ________________________________________
> From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <
> ismael@juma.me.uk>
> Sent: Tuesday, December 13, 2016 11:30:05 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Yes, this is actually tricky to do in a way where we both have the desired
> semantics and maintain compatibility. When someone creates a
> `ProducerRecord` with a `null` value today, the producer doesn't know if
> it's meant to be a tombstone or not. For V3 messages, it's easy when the
> constructor that takes a tombstone is used. However, if any other
> constructor is used, it's not clear. A couple of options I can think of,
> none of them particularly nice:
>
> 1. Have a third state where tombstone = unknown and the broker would set
> the tombstone bit if the value was null and the topic was compacted. People
> that wanted to pass a non-null value for the tombstone would have to use
> the constructor that takes a tombstone. The drawbacks: third state for
> tombstone in message format, message conversion at the broker for a common
> case.
>
> 2. Extend MetadataResponse to optionally include topic configs, which would
> make it possible for the producer to be smarter about setting the
> tombstone. It would only do it if a tombstone was not passed explicitly,
> the value was null and the topic was compacted. The main drawback is that
> the producer would be getting a bit more data for each topic even though it
> probably won't use it most of the time. Extending MetadataResponse to
> return topic configs would be useful for other reasons as well, so that
> part seems OK.
>
> In addition, for both proposals, we could consider adding warnings to the
> documentation that the behaviour of the constructors that don't take a
> tombstone would change in the next major release so that tombstone = false.
> Not sure if this would be worth it though.
>
> Ismael
>
> On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <ewen@confluent.io
> >
> wrote:
>
> > Michael,
> >
> > It kind of depends on how you want to interpret the tombstone flag. If
> it's
> > purely a producer-facing Kafka-level thing that we treat as internal to
> the
> > broker and log cleaner once the record is sent, then your approach makes
> > sense. You're just moving copying the null-indicates-delete behavior of
> the
> > old constructor into the tombstone flag.
> >
> > However, if you want this change to more generally decouple the idea of
> > deletion and null values, then you are sometimes converting what might
> be a
> > completely valid null value that doesn't indicate deletion into a
> > tombstone. Downstream applications could potentially handle these cases
> > differently given the separation of deletion from value.
> >
> > I guess the question is if we want to try to support the latter even for
> > topics where we have older produce requests. An example where this could
> > come up is in something like a CDC Connector. If we try to support the
> > semantic difference, a connector might write changes to Kafka using the
> > tombstone flag to indicate when a row was truly deleted (vs an update
> that
> > sets it to null but still present; this probably makes more sense for CDC
> > from document stores or extracting single columns). There are various
> > reasons we might want to maintain the full log and not turn compaction on
> > (or just use a time-based retention policy), but downstream applications
> > might care to know the difference between a delete and a null value. In
> > fact both versions of the same log (compacted and time-retention) could
> be
> > useful and I don't think it'll be uncommon to maintain both or use KIP-71
> > to maintain a hybrid compacted/retention topic.
> >
> > -Ewen
> >
> > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Why wouldn't that work, the tombstone value is only looked at by the
> > > broker, on a topic configured for compaction as such is benign on non
> > > compacted topics. This is as much as sending a null value currently
> > >
> > >
> > > Regards
> > > Mike
> > >
> > >
> > >
> > > Sent using OWA for iPhone
> > > ________________________________________
> > > From: Jay Kreps <ja...@confluent.io>
> > > Sent: Sunday, December 11, 2016 8:58:53 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > > Hey Michael,
> > >
> > > I'm not quite sure that works as that would translate ALL null values
> to
> > > tombstones, even for non-compacted topics that use null as an
> acceptable
> > > value sent by the producer and expected by the consumer.
> > >
> > > -Jay
> > >
> > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> > > wrote:
> > >
> > > > Hi Ewen,
> > > >
> > > > I think the easiest way to show this is with code.
> > > >
> > > > As you can see we keep the existing behaviour for code/binaries
> calling
> > > > the pre-existing constructors, whereby if the value is null the
> > tombstone
> > > > is set to true.
> > > >
> > > > Regards
> > > > Mike
> > > >
> > > >
> > > >
> > > >     /**
> > > >      * Creates a record with a specified timestamp to be sent to a
> > > > specified topic and partition
> > > >      *
> > > >      * @param topic The topic the record will be appended to
> > > >      * @param partition The partition to which the record should be
> > sent
> > > >      * @param timestamp The timestamp of the record
> > > >      * @param tombstone if the record should be treated as a
> tombstone
> > if
> > > > the topic is compacted
> > > >      * @param key The key that will be included in the record
> > > >      * @param value The record contents
> > > >      */
> > > >     public ProducerRecord(String topic, Integer partition, Boolean
> > > > tombstone, Long timestamp, K key, V value) {
> > > >         if (topic == null)
> > > >             throw new IllegalArgumentException("Topic cannot be
> > null.");
> > > >         if (timestamp != null && timestamp < 0)
> > > >             throw new IllegalArgumentException(
> > > >                     String.format("Invalid timestamp: %d. Timestamp
> > > should
> > > > always be non-negative or null.", timestamp));
> > > >         if (partition != null && partition < 0)
> > > >             throw new IllegalArgumentException(
> > > >                     String.format("Invalid partition: %d. Partition
> > > number
> > > > should always be non-negative or null.", partition));
> > > >         if (!tombstone && value == null){
> > > >             throw new IllegalArgumentException(
> > > >                     String.format("Tombstone must be true if null
> > > value"));
> > > >         }
> > > >         this.topic = topic;
> > > >         this.partition = partition;
> > > >         this.tombstone = tombstone;
> > > >         this.key = key;
> > > >         this.value = value;
> > > >         this.timestamp = timestamp;
> > > >     }
> > > >
> > > >     public ProducerRecord(String topic, Integer partition, Long
> > > timestamp,
> > > > K key, V value) {
> > > >         this(topic, partition, value == null, timestamp, key, value);
> > > >     }
> > > > ________________________________________
> > > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> > > > Sent: Sunday, December 11, 2016 5:45 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >
> > > > It seemed like this was addressed in the migration section, wasn't
> it?
> > > The
> > > > V2 vs V3 requests and the fact that the broker will downgrade the
> > message
> > > > format (losing zero copy) if you issues a V2 request to a broker
> using
> > V3
> > > > format handles compatibility, does it not? The existing consumers
> won't
> > > see
> > > > the extra metadata in the value, but they will get a null instead and
> > > treat
> > > > it as a tombstone. Certainly there is a performance impact, but it
> > seemed
> > > > compatible.
> > > >
> > > > I'm worried about this though:
> > > >
> > > >
> > > >    - *NOTE *: With the new version of producer client using
> > > ProduceRequest
> > > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and
> > null
> > > > value
> > > >    should not be allowed as the older version of consumer using
> > > > FetchRequest
> > > >    V2 will think of this as a tombstone when its actually not.
> > > >
> > > >
> > > > Unless I'm misunderstanding, this ends up breaking binary
> compatibility
> > > for
> > > > the Java API. It sounds like this suggests that passing a null value
> to
> > > the
> > > > existing ProducerRecord constructors would generate an exception
> since
> > > you
> > > > didn't explicitly enable the tombstone (via whatever new constructor
> is
> > > > provided). But that means you can't swap in a newer client jar
> without
> > > > recompiling and get the same behavior if your app deletes keys using
> > the
> > > > current approach because your app will start throwing exceptions.
> Maybe
> > > > this is a tradeoff we're ok with, but we've tried pretty hard to
> avoid
> > > > breaking compatibility like this so far -- it makes picking up bug
> > fixes
> > > in
> > > > the clients harder for users of frameworks that have to pin to
> earlier
> > > > default versions for compatibility.
> > > >
> > > > But then later the KIP says:
> > > >
> > > >
> > > >    - The old constructors for ProducerRecord without this parameter
> > will
> > > be
> > > >    kept but updated so that their default behaviour if setting null
> > value
> > > > will
> > > >    be the tombstone will be set to true to keep existing behaviour.
> > > >
> > > >
> > > > So maybe I am misinterpreting something.
> > > >
> > > > And just a nit re: motivation section, item 6 would be clearer for a
> > > union
> > > > schema with null (or optional schemas in other formats), e.g. [null,
> > > > string], in which case losing the schema truly is losing information
> > > > (whereas null is already the only valid value for a pure null
> schema).
> > > >
> > > > -Ewen
> > > >
> > > >
> > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Jay,
> > > > >
> > > > > Good point this detail is missing in the KIP write up. Ive added
> this
> > > > now.
> > > > >
> > > > > Essentially simply just upgrading the clients we do not expect any
> > > client
> > > > > app code change needed.
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > > ________________________________________
> > > > > From: Jay Kreps <ja...@confluent.io>
> > > > > Sent: Saturday, December 10, 2016 10:51 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > > >
> > > > > Michael,
> > > > >
> > > > > The compatibility section goes through the migration path, but
> isn't
> > > the
> > > > > bigger compatibility issue with existing apps? There are many
> > (probably
> > > > > thousands) of apps in production that use this feature and send
> null
> > to
> > > > > mean delete. It seems like this would break compatibility with
> them,
> > > and
> > > > > they would have to be rewritten to right?
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > 4) On v3 we honour the tombstone. As such we expect it to be set
> > > > > correctly
> > > > > > as per the KIP.
> > > > > >
> > > > > > 4.1) why would we want to produce an error when its v3? This is
> the
> > > > exact
> > > > > > purpose to support non-null tombstone’s
> > > > > > 4.2) again here im unclear on the question on the v3, produce
> > request
> > > > we
> > > > > > expect the tombstone flag to be set correctly.
> > > > > >
> > > > > > 4.4) compaction only occurs on compacted topics, the bit makes no
> > > > > > difference and not looked at on un-compacted (time/size based
> > > > eviction).
> > > > > >
> > > > > >
> > > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> > > > > >
> > > > > >     Hi, Michael,
> > > > > >
> > > > > >     4. Then, I think I misunderstood this point. Could you
> document
> > > the
> > > > > >     following points in the wiki?
> > > > > >     4.1 If producer V3 sets tombstone, but provides a non-null
> > value,
> > > > > does
> > > > > > the
> > > > > >     send() get an error or does the producer automatically set
> the
> > > > value
> > > > > to
> > > > > >     null?
> > > > > >     4.2 If producer V3 doesn't set tombstone, but provides a null
> > > > value,
> > > > > > does
> > > > > >     the send() get an error or does the producer automatically
> sets
> > > the
> > > > > >     tombstone?
> > > > > >     4.3 Does the broker only expect messages that (a) have no
> > > tombstone
> > > > > and
> > > > > >     non-null value; (b) have tombstone and null value and reject
> > the
> > > > > > messages
> > > > > >     with an error code otherwise?
> > > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is
> compacted
> > on
> > > > > not?
> > > > > >
> > > > > >     Thanks,
> > > > > >
> > > > > >     Jun
> > > > > >
> > > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > > > > Michael.Pearce@ig.com
> > > > > > >
> > > > > >     wrote:
> > > > > >
> > > > > >     > Not at all.  This only acts on compacted topics just as
> what
> > > > occurs
> > > > > > today
> > > > > >     >
> > > > > >     > Sent using OWA for iPhone
> > > > > >     > ________________________________________
> > > > > >     > From: Jun Rao <ju...@confluent.io>
> > > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > > > > >     > To: dev@kafka.apache.org
> > > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > > > >     >
> > > > > >     > Hi, Michael,
> > > > > >     >
> > > > > >     > 4. Hmm, does that mean the new client library can never
> send
> > a
> > > > null
> > > > > > message
> > > > > >     > even to a regular topic? This seems like a change of the
> > > existing
> > > > > > behavior.
> > > > > >     >
> > > > > >     > Thanks,
> > > > > >     >
> > > > > >     > Jun
> > > > > >     >
> > > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > >     > wrote:
> > > > > >     >
> > > > > >     > > Hi Jun,
> > > > > >     > >
> > > > > >     > > Re 4) That's because we expect the tombstone value to be
> > set
> > > > > > correctly if
> > > > > >     > > message bit is 2, as such if an older client sends in on
> > old
> > > > > > message the
> > > > > >     > > message is upcast and the bit is set correctly. And such
> no
> > > > > longer
> > > > > > need
> > > > > >     > to
> > > > > >     > > check the value. Mayuresh can you confirm my thinking and
> > > > > > understanding
> > > > > >     > of
> > > > > >     > > what we've discussed?
> > > > > >     > >
> > > > > >     > > The second point I understand what you're getting at now
> my
> > > > > > apologies.
> > > > > >     > Yes
> > > > > >     > > this makes sense to save on touching the message, if
> we're
> > > the
> > > > > > only kip
> > > > > >     > > going in, in this release.
> > > > > >     > >
> > > > > >     > > Cheers
> > > > > >     > > Mike
> > > > > >     > >
> > > > > >     > > Sent using OWA for iPhone
> > > > > >     > > ________________________________________
> > > > > >     > > From: Jun Rao <ju...@confluent.io>
> > > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > > > > >     > > To: dev@kafka.apache.org
> > > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> Flag
> > > > > >     > >
> > > > > >     > > Hi, Michael,
> > > > > >     > >
> > > > > >     > > 4. Is this updated in the wiki? The text "If the magic
> byte
> > > on
> > > > > > message is
> > > > > >     > > 2, the broker should use the tombstone bit for log
> > > compaction."
> > > > > > doesn't
> > > > > >     > > seem to have changed.
> > > > > >     > >
> > > > > >     > > 2. My point is that if we change the message format just
> > for
> > > > this
> > > > > > KIP, we
> > > > > >     > > should consider whether it's worth optimizing the down
> > > > conversion
> > > > > > path
> > > > > >     > > (i.e., decide whether a conversion is needed by just
> > looking
> > > at
> > > > > the
> > > > > >     > > tombstone bit in the wrapper message) since tombstone
> will
> > be
> > > > > used
> > > > > >     > rarely.
> > > > > >     > > However, if the message format change here is combined
> with
> > > > other
> > > > > > KIPs,
> > > > > >     > > then this optimization likely won't be needed. The latter
> > > > > probably
> > > > > > makes
> > > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> > > > > >     > >
> > > > > >     > > Other than those, +1 from me,
> > > > > >     > >
> > > > > >     > > Thanks,
> > > > > >     > >
> > > > > >     > > Jun
> > > > > >     > >
> > > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > >     > > wrote:
> > > > > >     > >
> > > > > >     > > > Hi Jun
> > > > > >     > > >
> > > > > >     > > > do we have your vote on this now?
> > > > > >     > > >
> > > > > >     > > > Any other concerns?
> > > > > >     > > >
> > > > > >     > > > Cheers
> > > > > >     > > > Mike
> > > > > >     > > >
> > > > > >     > > > Sent using OWA for iPhone
> > > > > >     > > > ________________________________________
> > > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> > > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > > > > >     > > > To: dev@kafka.apache.org
> > > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> > Flag
> > > > > >     > > >
> > > > > >     > > > Hi Jun,
> > > > > >     > > >
> > > > > >     > > > Have updated. Thanks again for the feedback.
> > > > > >     > > >
> > > > > >     > > > Agree yes we should align up when it gets to that, I
> > assume
> > > > > > you’ve
> > > > > >     > > flagged
> > > > > >     > > > the same in the other KIP?
> > > > > >     > > >
> > > > > >     > > > I think for now let’s get this KIP approved, then we
> can
> > > > start
> > > > > > the work
> > > > > >     > > to
> > > > > >     > > > get an initial PR, then we can discuss how to align the
> > two
> > > > if
> > > > > > needed.
> > > > > >     > > >
> > > > > >     > > > Cheers,
> > > > > >     > > > Mike
> > > > > >     > > >
> > > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io>
> > wrote:
> > > > > >     > > >
> > > > > >     > > >     Hi, Michael,
> > > > > >     > > >
> > > > > >     > > >     For 2), this is fine. Could you update the KIP wiki
> > to
> > > > make
> > > > > > this
> > > > > >     > and
> > > > > >     > > > other
> > > > > >     > > >     points clearer? Other than that, the KIP looks good
> > to
> > > > me.
> > > > > >     > > >
> > > > > >     > > >     An orthogonal thing is that there are other KIPs
> such
> > > as
> > > > > > KIP-98
> > > > > >     > that
> > > > > >     > > > also
> > > > > >     > > >     intend to change the message format. If they all
> get
> > > > > > approved, we
> > > > > >     > > > should
> > > > > >     > > >     think about whether it's better to just bump up the
> > > magic
> > > > > > byte once
> > > > > >     > > to
> > > > > >     > > >     incorporate multiple format changes like we did in
> > > > > > KIP-31/KIP-32.
> > > > > >     > > >
> > > > > >     > > >     Thanks,
> > > > > >     > > >
> > > > > >     > > >     Jun
> > > > > >     > > >
> > > > > >     > > >
> > > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > > > > >     > > Michael.Pearce@ig.com>
> > > > > >     > > >     wrote:
> > > > > >     > > >
> > > > > >     > > >     > Hi Jao
> > > > > >     > > >     >
> > > > > >     > > >     > Thanks for the response. Sorry for slow reply,
> both
> > > > with
> > > > > > personal
> > > > > >     > > > sickness
> > > > > >     > > >     > and also battling some critical issues
> encountered
> > > > since
> > > > > >     > upgrading
> > > > > >     > > to
> > > > > >     > > >     > 0.10.1.0
> > > > > >     > > >     >
> > > > > >     > > >     > 1) Thans for spotting, Document error where we
> > > branched
> > > > > > this KIP
> > > > > >     > > from
> > > > > >     > > >     > KIP-82, will get that removed.
> > > > > >     > > >     > 2) Intent is to do this just at the record
> message
> > > > level.
> > > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> > > corrected.
> > > > > >     > > >     > 4) As per discussion thread we will support
> > > tombstone +
> > > > > > null
> > > > > >     > value,
> > > > > >     > > >     > tombstone + non null value, no tombstone + null
> > > value.
> > > > > >     > > >     > 5) I believe this was in the discussion thread,
> > > > @Mayuresh
> > > > > > is this
> > > > > >     > > >     > something we’ve overlooked? I thought we would
> down
> > > > > > convert and
> > > > > >     > > > remove the
> > > > > >     > > >     > value so the old consumer had existing behavior,
> or
> > > is
> > > > > > there
> > > > > >     > > > something we
> > > > > >     > > >     > haven’t thought about?
> > > > > >     > > >     >
> > > > > >     > > >     > Cheers
> > > > > >     > > >     > Mike
> > > > > >     > > >     >
> > > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <
> jun@confluent.io>
> > > > > wrote:
> > > > > >     > > >     >
> > > > > >     > > >     >     Hi, Michael,
> > > > > >     > > >     >
> > > > > >     > > >     >     Thanks for the KIP. A few comments below.
> > > > > >     > > >     >
> > > > > >     > > >     >     1. The message format change contains
> > > > "HeadersLength
> > > > > >     > Headers".
> > > > > >     > > > Is that
> > > > > >     > > >     >     intended?
> > > > > >     > > >     >
> > > > > >     > > >     >     2. For compressed messageset, is the
> tombstone
> > > bit
> > > > > > only set
> > > > > >     > at
> > > > > >     > > > the
> > > > > >     > > >     > shallow
> > > > > >     > > >     >     level? Do we always leave that bit in the
> > wrapper
> > > > > > message
> > > > > >     > > unset?
> > > > > >     > > > An
> > > > > >     > > >     >     alternative is to set the tombstone bit in
> the
> > > > > wrapper
> > > > > > if at
> > > > > >     > > > least one
> > > > > >     > > >     >     inner message has the tombstone bit set. This
> > > makes
> > > > > > things a
> > > > > >     > > bit
> > > > > >     > > > more
> > > > > >     > > >     >     complicated, but we could potentially exploit
> > > that
> > > > > for
> > > > > >     > > > optimizing down
> > > > > >     > > >     >     conversion. For example, we only need to
> > convert
> > > > > > messages
> > > > > >     > with
> > > > > >     > > > magic 2
> > > > > >     > > >     > to
> > > > > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > > > > > (conversion is
> > > > > >     > > > always
> > > > > >     > > >     > needed
> > > > > >     > > >     >     from magic 2 to magic 0). Not sure if the
> > > > > optimization
> > > > > > is
> > > > > >     > worth
> > > > > >     > > > the
> > > > > >     > > >     >     complexity though.
> > > > > >     > > >     >
> > > > > >     > > >     >     3. The referencing of the new version of
> > > > > >     > > > ProducerRequest/FetchRequest
> > > > > >     > > >     > is
> > > > > >     > > >     >     inconsistent (v4 vs v3). Since our convention
> > > > starts
> > > > > at
> > > > > >     > version
> > > > > >     > > > at 0, I
> > > > > >     > > >     >     think the new version would be 3.
> > > > > >     > > >     >
> > > > > >     > > >     >     4. "If the magic byte on message is 2, the
> > broker
> > > > > > should use
> > > > > >     > > the
> > > > > >     > > >     > tombstone
> > > > > >     > > >     >     bit for log compaction." What about null
> value?
> > > My
> > > > > >     > > understanding
> > > > > >     > > > is
> > > > > >     > > >     > that
> > > > > >     > > >     >     null value will be treated the same as
> setting
> > > the
> > > > > > tombstone
> > > > > >     > > bit.
> > > > > >     > > >     >
> > > > > >     > > >     >     5. For the migration path, it would be useful
> > to
> > > > > > describe the
> > > > > >     > > > down
> > > > > >     > > >     >     conversion path to consumers (i.e., brokers
> on
> > > > > message
> > > > > > format
> > > > > >     > > > 0.10.2
> > > > > >     > > >     > and
> > > > > >     > > >     >     consumers on older version).
> > > > > >     > > >     >
> > > > > >     > > >     >     Thanks,
> > > > > >     > > >     >
> > > > > >     > > >     >     Jun
> > > > > >     > > >     >
> > > > > >     > > >     >
> > > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael
> > Pearce <
> > > > > >     > > > Michael.Pearce@ig.com
> > > > > >     > > >     > >
> > > > > >     > > >     >     wrote:
> > > > > >     > > >     >
> > > > > >     > > >     >     > Hi All,
> > > > > >     > > >     >     >
> > > > > >     > > >     >     > We have been discussing in the below thread
> > and
> > > > > final
> > > > > >     > changes
> > > > > >     > > > have
> > > > > >     > > >     > been
> > > > > >     > > >     >     > made to the KIP wiki based on these
> > > discussions.
> > > > > >     > > >     >     >
> > > > > >     > > >     >     > We would now like to put to the vote the
> > > > following
> > > > > > KIP:
> > > > > >     > > >     >     > https://cwiki.apache.org/
> > > > > > confluence/display/KAFKA/KIP-
> > > > > >     > 87+-+
> > > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > > > > >     > > >     >     >
> > > > > >     > > >     >     > This kip is for having a distinct
> compaction
> > > > > > attribute
> > > > > >     > > > “tombstone”
> > > > > >     > > >     > flag
> > > > > >     > > >     >     > instead of relying on null value, allowing
> > > > non-null
> > > > > > value
> > > > > >     > > > delete
> > > > > >     > > >     > messages.
> > > > > >     > > >     >     >
> > > > > >     > > >     >     > Many thanks,
> > > > > >     > > >     >     > Michael
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >
> > > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> > > > > >     > > Michael.Pearce@ig.com>
> > > > > >     > > >     > wrote:
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >     Hi Mayuresh,
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >     LGTM. Ive just made one small
> adjustment
> > > > > > updating the
> > > > > >     > > wire
> > > > > >     > > >     > protocol to
> > > > > >     > > >     >     > show the magic byte bump.
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >     Do we think we’re good to put to a
> vote?
> > Is
> > > > > > there any
> > > > > >     > > > other bits
> > > > > >     > > >     >     > needing discussion?
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >     Cheers
> > > > > >     > > >     >     >     Mike
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh
> Gharat" <
> > > > > >     > > >     > gharatmayuresh15@gmail.com>
> > > > > >     > > >     >     > wrote:
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >         Hi Michael,
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >         I have updated the migration
> section
> > of
> > > > the
> > > > > > KIP.
> > > > > >     > Can
> > > > > >     > > > you
> > > > > >     > > >     > please
> > > > > >     > > >     >     > take a look?
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >         Thanks,
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >         Mayuresh
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM,
> > > Mayuresh
> > > > > > Gharat <
> > > > > >     > > >     >     > gharatmayuresh15@gmail.com
> > > > > >     > > >     >     >         > wrote:
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >         > Hi Michael,
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         > That whilst sending tombstone and
> > non
> > > > > null
> > > > > > value,
> > > > > >     > > the
> > > > > >     > > >     > consumer
> > > > > >     > > >     >     > can expect
> > > > > >     > > >     >     >         > only to receive the non-null
> > message
> > > > only
> > > > > > in step
> > > > > >     > > > (3) is
> > > > > >     > > >     > this
> > > > > >     > > >     >     > correct?
> > > > > >     > > >     >     >         > ---> I do agree with you here.
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         > Becket, Ismael : can you guys
> > review
> > > > the
> > > > > >     > migration
> > > > > >     > > > plan
> > > > > >     > > >     > listed
> > > > > >     > > >     >     > above using
> > > > > >     > > >     >     >         > magic byte?
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         > Thanks,
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         > Mayuresh
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
> > > > Michael
> > > > > > Pearce <
> > > > > >     > > >     >     > Michael.Pearce@ig.com>
> > > > > >     > > >     >     >         > wrote:
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         >> Many thanks for this Mayuresh. I
> > > don't
> > > > > > have any
> > > > > >     > > >     > objections.
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> I assume we should state:
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> That whilst sending tombstone
> and
> > > non
> > > > > null
> > > > > >     > value,
> > > > > >     > > > the
> > > > > >     > > >     > consumer
> > > > > >     > > >     >     > can expect
> > > > > >     > > >     >     >         >> only to receive the non-null
> > message
> > > > > only
> > > > > > in
> > > > > >     > step
> > > > > >     > > > (3) is
> > > > > >     > > >     > this
> > > > > >     > > >     >     > correct?
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Cheers
> > > > > >     > > >     >     >         >> Mike
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Sent using OWA for iPhone
> > > > > >     > > >     >     >         >> ______________________________
> > > > > __________
> > > > > >     > > >     >     >         >> From: Mayuresh Gharat <
> > > > > >     > gharatmayuresh15@gmail.com
> > > > > >     > > >
> > > > > >     > > >     >     >         >> Sent: Thursday, November 17,
> 2016
> > > > > 5:18:41
> > > > > > PM
> > > > > >     > > >     >     >         >> To: dev@kafka.apache.org
> > > > > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 -
> > Add
> > > > > > Compaction
> > > > > >     > > > Tombstone
> > > > > >     > > >     > Flag
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Hi Ismael,
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Thanks for the explanation.
> > > > > >     > > >     >     >         >> Specially I like this part where
> > in
> > > > you
> > > > > >     > mentioned
> > > > > >     > > > we can
> > > > > >     > > >     > get
> > > > > >     > > >     >     > rid of the
> > > > > >     > > >     >     >         >> older null value support for log
> > > > > > compaction
> > > > > >     > later
> > > > > >     > > > on,
> > > > > >     > > >     > here :
> > > > > >     > > >     >     >         >> We can't change semantics of the
> > > > message
> > > > > > format
> > > > > >     > > > without
> > > > > >     > > >     > having
> > > > > >     > > >     >     > a long
> > > > > >     > > >     >     >         >> transition period. And we can't
> > rely
> > > > > >     > > >     >     >         >> on people reading documentation
> or
> > > > > acting
> > > > > > on a
> > > > > >     > > > warning for
> > > > > >     > > >     >     > something so
> > > > > >     > > >     >     >         >> fundamental. As such, my take is
> > > that
> > > > we
> > > > > > need to
> > > > > >     > > > bump the
> > > > > >     > > >     > magic
> > > > > >     > > >     >     > byte. The
> > > > > >     > > >     >     >         >> good news is
> > > > > >     > > >     >     >         >> that we don't have to support
> all
> > > > > versions
> > > > > >     > > forever.
> > > > > >     > > > We
> > > > > >     > > >     > have
> > > > > >     > > >     >     > said that we
> > > > > >     > > >     >     >         >> will support direct upgrades
> for 2
> > > > > years.
> > > > > > That
> > > > > >     > > > means that
> > > > > >     > > >     >     > message format
> > > > > >     > > >     >     >         >> version n could, in theory, be
> > > > removed 2
> > > > > > years
> > > > > >     > > > after the
> > > > > >     > > >     > it's
> > > > > >     > > >     >     > introduced.
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Just a heads up, I would like to
> > > > mention
> > > > > > that
> > > > > >     > even
> > > > > >     > > > without
> > > > > >     > > >     >     > bumping magic
> > > > > >     > > >     >     >         >> byte, we will *NOT* loose zero
> > copy
> > > as
> > > > > in
> > > > > > the
> > > > > >     > > > client(x+1)
> > > > > >     > > >     > in my
> > > > > >     > > >     >     >         >> explanation
> > > > > >     > > >     >     >         >> above will convert internally a
> > null
> > > > > > value to
> > > > > >     > > have a
> > > > > >     > > >     > tombstone
> > > > > >     > > >     >     > bit set and
> > > > > >     > > >     >     >         >> a tombstone bit set to have a
> null
> > > > value
> > > > > >     > > > automatically
> > > > > >     > > >     >     > internally and by
> > > > > >     > > >     >     >         >> the time we move to version
> (x+2),
> > > the
> > > > > > clients
> > > > > >     > > > would have
> > > > > >     > > >     >     > upgraded.
> > > > > >     > > >     >     >         >> Obviously if we support a
> request
> > > from
> > > > > >     > > consumer(x),
> > > > > >     > > > we
> > > > > >     > > >     > will
> > > > > >     > > >     >     > loose zero
> > > > > >     > > >     >     >         >> copy
> > > > > >     > > >     >     >         >> but that is the same case with
> > magic
> > > > > byte.
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> But if magic byte bump makes
> life
> > > > easier
> > > > > > for
> > > > > >     > > > transition
> > > > > >     > > >     > for the
> > > > > >     > > >     >     > above
> > > > > >     > > >     >     >         >> reasons that you explained, I am
> > OK
> > > > with
> > > > > > it
> > > > > >     > since
> > > > > >     > > > we are
> > > > > >     > > >     > going
> > > > > >     > > >     >     > to meet the
> > > > > >     > > >     >     >         >> end goal down the road :)
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> On a side note can we update the
> > doc
> > > > > here
> > > > > > on
> > > > > >     > magic
> > > > > >     > > > byte
> > > > > >     > > >     > to say
> > > > > >     > > >     >     > that "*it
> > > > > >     > > >     >     >         >> should be bumped whenever the
> > > message
> > > > > > format is
> > > > > >     > > > changed
> > > > > >     > > >     > or the
> > > > > >     > > >     >     >         >> interpretation of message format
> > > > (usage
> > > > > > of the
> > > > > >     > > > reserved
> > > > > >     > > >     > bits as
> > > > > >     > > >     >     > well) is
> > > > > >     > > >     >     >         >> changed*".
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Hi Michael,
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Here is the update plan that we
> > > > > discussed
> > > > > >     > offline
> > > > > >     > > >     > yesterday :
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Currently the magic-byte which
> > > > > > corresponds to
> > > > > >     > the
> > > > > >     > > >     >     > "message.format.version"
> > > > > >     > > >     >     >         >> is set to 1.
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> 1) On broker it will be set to 1
> > > > > > initially.
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> 2) When a producer client sends
> a
> > > > > message
> > > > > > with
> > > > > >     > > > magic-byte
> > > > > >     > > >     > = 2,
> > > > > >     > > >     >     > since the
> > > > > >     > > >     >     >         >> broker is on magic-byte = 1, we
> > will
> > > > > down
> > > > > >     > convert
> > > > > >     > > > it,
> > > > > >     > > >     > which
> > > > > >     > > >     >     > means if the
> > > > > >     > > >     >     >         >> tombstone bit is set, the value
> > will
> > > > be
> > > > > > set to
> > > > > >     > > > null. A
> > > > > >     > > >     > consumer
> > > > > >     > > >     >     >         >> understanding magic-byte = 1,
> will
> > > > still
> > > > > > work
> > > > > >     > with
> > > > > >     > > > this. A
> > > > > >     > > >     >     > consumer
> > > > > >     > > >     >     >         >> working
> > > > > >     > > >     >     >         >> with magic-byte =2 will also be
> > able
> > > > to
> > > > > >     > understand
> > > > > >     > > > this,
> > > > > >     > > >     > since
> > > > > >     > > >     >     > it
> > > > > >     > > >     >     >         >> understands the tombstone.
> > > > > >     > > >     >     >         >> Now there is still the question
> of
> > > > > > supporting a
> > > > > >     > > >     > non-tombstone
> > > > > >     > > >     >     > and null
> > > > > >     > > >     >     >         >> value from producer client with
> > > > > > magic-byte = 2.*
> > > > > >     > > (I
> > > > > >     > > > am
> > > > > >     > > >     > not sure
> > > > > >     > > >     >     > if we
> > > > > >     > > >     >     >         >> should support this.
> Ismael/Becket
> > > can
> > > > > > comment
> > > > > >     > > > here)*
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> 3) When almost all the clients
> > have
> > > > > > upgraded,
> > > > > >     > the
> > > > > >     > > >     >     > message.format.version
> > > > > >     > > >     >     >         >> on
> > > > > >     > > >     >     >         >> the broker can be changed to 2,
> > > where
> > > > in
> > > > > > the
> > > > > >     > down
> > > > > >     > > >     > conversion in
> > > > > >     > > >     >     > the above
> > > > > >     > > >     >     >         >> step will not happen. If at this
> > > point
> > > > > we
> > > > > > get a
> > > > > >     > > > consumer
> > > > > >     > > >     >     > request from a
> > > > > >     > > >     >     >         >> older consumer, we might have to
> > > down
> > > > > > convert
> > > > > >     > > where
> > > > > >     > > > in we
> > > > > >     > > >     > loose
> > > > > >     > > >     >     > zero copy,
> > > > > >     > > >     >     >         >> but these cases should be rare.
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Becket can you review this plan
> > and
> > > > add
> > > > > > more
> > > > > >     > > > details if I
> > > > > >     > > >     > have
> > > > > >     > > >     >     >         >> missed/wronged something, before
> > we
> > > > put
> > > > > > it on
> > > > > >     > KIP.
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Thanks,
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> Mayuresh
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07
> PM,
> > > > > Michael
> > > > > >     > Pearce <
> > > > > >     > > >     >     > Michael.Pearce@ig.com>
> > > > > >     > > >     >     >         >> wrote:
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> > Thanks guys, for discussing
> this
> > > > > > offline and
> > > > > >     > > > getting
> > > > > >     > > >     > some
> > > > > >     > > >     >     > consensus.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > So its clear for myself and
> > others
> > > > > what
> > > > > > is
> > > > > >     > > > proposed now
> > > > > >     > > >     > (i
> > > > > >     > > >     >     > think i
> > > > > >     > > >     >     >         >> > understand, but want to make
> > sure)
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > Could i ask either directly
> > update
> > > > the
> > > > > > kip to
> > > > > >     > > > detail the
> > > > > >     > > >     >     > migration
> > > > > >     > > >     >     >         >> > strategy, or (re-)state your
> > > offline
> > > > > > discussed
> > > > > >     > > and
> > > > > >     > > >     > agreed
> > > > > >     > > >     >     > migration
> > > > > >     > > >     >     >         >> > strategy based on a magic byte
> > is
> > > in
> > > > > > this
> > > > > >     > > thread.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > The main original driver for
> the
> > > KIP
> > > > > > was to
> > > > > >     > > > support
> > > > > >     > > >     >     > compaction where
> > > > > >     > > >     >     >         >> value
> > > > > >     > > >     >     >         >> > isn't null, based off the
> > > > discussions
> > > > > on
> > > > > >     > KIP-82
> > > > > >     > > > thread.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > We should be able to support
> > > > > > non-tombstone +
> > > > > >     > > null
> > > > > >     > > > value
> > > > > >     > > >     > by the
> > > > > >     > > >     >     >         >> completion
> > > > > >     > > >     >     >         >> > of the KIP, as we noted when
> > > > > discussing
> > > > > > this
> > > > > >     > > kip,
> > > > > >     > > > having
> > > > > >     > > >     >     > logic based on
> > > > > >     > > >     >     >         >> a
> > > > > >     > > >     >     >         >> > null value isn't very clean
> and
> > > also
> > > > > > separates
> > > > > >     > > the
> > > > > >     > > >     > concerns.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > As discussed already though we
> > can
> > > > > > split this
> > > > > >     > > into
> > > > > >     > > >     > KIP-87a
> > > > > >     > > >     >     > and KIP-87b
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > Where we look to deliver
> KIP-87a
> > > on
> > > > a
> > > > > >     > compacted
> > > > > >     > > > topic
> > > > > >     > > >     > (to
> > > > > >     > > >     >     > address the
> > > > > >     > > >     >     >         >> > immediate issues)
> > > > > >     > > >     >     >         >> > * tombstone + null value
> > > > > >     > > >     >     >         >> > * tombstone + non-null value
> > > > > >     > > >     >     >         >> > * non-tombstone + non-null
> value
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > Then we can discuss once
> KIP-87a
> > > is
> > > > > > completed
> > > > > >     > > > options
> > > > > >     > > >     > later
> > > > > >     > > >     >     > and how we
> > > > > >     > > >     >     >         >> > support the second part
> KIP-87b
> > to
> > > > > > deliver:
> > > > > >     > > >     >     >         >> > * non-tombstone + null value
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > Cheers
> > > > > >     > > >     >     >         >> > Mike
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > ______________________________
> > > > > > __________
> > > > > >     > > >     >     >         >> > From: Becket Qin <
> > > > > becket.qin@gmail.com>
> > > > > >     > > >     >     >         >> > Sent: Thursday, November 17,
> > 2016
> > > > 1:43
> > > > > > AM
> > > > > >     > > >     >     >         >> > To: dev@kafka.apache.org
> > > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87
> -
> > > Add
> > > > > > Compaction
> > > > > >     > > >     > Tombstone Flag
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > Renu, Mayuresh and I had an
> > > offline
> > > > > >     > discussion,
> > > > > >     > > > and
> > > > > >     > > >     > following
> > > > > >     > > >     >     > is a brief
> > > > > >     > > >     >     >         >> > summary.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > 1. We agreed that not bumping
> up
> > > > magic
> > > > > > value
> > > > > >     > may
> > > > > >     > > > result
> > > > > >     > > >     > in
> > > > > >     > > >     >     > losing zero
> > > > > >     > > >     >     >         >> copy
> > > > > >     > > >     >     >         >> > during migration.
> > > > > >     > > >     >     >         >> > 2. Given that bumping up magic
> > > value
> > > > > is
> > > > > > almost
> > > > > >     > > > free and
> > > > > >     > > >     > has
> > > > > >     > > >     >     > benefit of
> > > > > >     > > >     >     >         >> > avoiding potential performance
> > > > issue.
> > > > > > It is
> > > > > >     > > > probably
> > > > > >     > > >     > worth
> > > > > >     > > >     >     > doing.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > One issue we still need to
> think
> > > > about
> > > > > > is
> > > > > >     > > whether
> > > > > >     > > > we
> > > > > >     > > >     > want to
> > > > > >     > > >     >     > support a
> > > > > >     > > >     >     >         >> > non-tombstone message with
> null
> > > > value.
> > > > > >     > > >     >     >         >> > Currently it is not supported
> by
> > > > > Kafka.
> > > > > > If we
> > > > > >     > > > allow a
> > > > > >     > > >     >     > non-tombstone null
> > > > > >     > > >     >     >         >> > value message to exist after
> > > KIP-87.
> > > > > The
> > > > > >     > problem
> > > > > >     > > > is
> > > > > >     > > >     > that such
> > > > > >     > > >     >     > message
> > > > > >     > > >     >     >         >> will
> > > > > >     > > >     >     >         >> > not be supported by the
> > consumers
> > > > > prior
> > > > > > to
> > > > > >     > > KIP-87.
> > > > > >     > > >     > Because a
> > > > > >     > > >     >     > null value
> > > > > >     > > >     >     >         >> > will always be interpreted to
> a
> > > > > > tombstone.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > One option is that we keep the
> > > > current
> > > > > > way,
> > > > > >     > i.e.
> > > > > >     > > > do not
> > > > > >     > > >     >     > support such
> > > > > >     > > >     >     >         >> > message. It would be good to
> > know
> > > if
> > > > > > there is
> > > > > >     > a
> > > > > >     > > >     > concrete use
> > > > > >     > > >     >     > case for
> > > > > >     > > >     >     >         >> such
> > > > > >     > > >     >     >         >> > message. If there is not, we
> can
> > > > > > probably just
> > > > > >     > > not
> > > > > >     > > >     > support it.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > Thanks,
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28
> PM,
> > > > > > Mayuresh
> > > > > >     > > Gharat <
> > > > > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> > > > > >     > > >     >     >         >> > > wrote:
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >> > > Hi Ismael,
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > > This is something I can
> think
> > of
> > > > for
> > > > > >     > migration
> > > > > >     > > > plan:
> > > > > >     > > >     >     >         >> > > So the migration plan can
> look
> > > > > > something
> > > > > >     > like
> > > > > >     > > > this,
> > > > > >     > > >     > with up
> > > > > >     > > >     >     >         >> conversion :
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > > 1) Currently lets say we
> have
> > > > Broker
> > > > > > at
> > > > > >     > > version
> > > > > >     > > > x.
> > > > > >     > > >     >     >         >> > > 2) Currently we have clients
> > at
> > > > > > version x.
> > > > > >     > > >     >     >         >> > > 3) a) We move the version to
> > > > > > Broker(x+1) :
> > > > > >     > > > supports
> > > > > >     > > >     > both
> > > > > >     > > >     >     > tombstone and
> > > > > >     > > >     >     >         >> > null
> > > > > >     > > >     >     >         >> > > for log compaction.
> > > > > >     > > >     >     >         >> > >     b) We upgrade the client
> > to
> > > > > > version
> > > > > >     > > > client(x+1) :
> > > > > >     > > >     > if in
> > > > > >     > > >     >     > the
> > > > > >     > > >     >     >         >> producer
> > > > > >     > > >     >     >         >> > > client(x+1) the value is set
> > to
> > > > > null,
> > > > > > we
> > > > > >     > will
> > > > > >     > > >     > automatically
> > > > > >     > > >     >     > set the
> > > > > >     > > >     >     >         >> > > Tombstone bit internally. If
> > the
> > > > > > producer
> > > > > >     > > > client(x+1)
> > > > > >     > > >     > sets
> > > > > >     > > >     >     > the
> > > > > >     > > >     >     >         >> tombstone
> > > > > >     > > >     >     >         >> > > itself, well and good. For
> > > > producer
> > > > > >     > client(x),
> > > > > >     > > > the
> > > > > >     > > >     > broker
> > > > > >     > > >     >     > will up
> > > > > >     > > >     >     >         >> convert
> > > > > >     > > >     >     >         >> > > to have the tombstone bit.
> > > > > > Broker(x+1) is
> > > > > >     > > > supporting
> > > > > >     > > >     > both.
> > > > > >     > > >     >     > Consumer
> > > > > >     > > >     >     >         >> > > client(x+1) will be aware of
> > > this
> > > > > and
> > > > > > should
> > > > > >     > > be
> > > > > >     > > > able
> > > > > >     > > >     > to
> > > > > >     > > >     >     > handle this.
> > > > > >     > > >     >     >         >> For
> > > > > >     > > >     >     >         >> > > consumer client(x) we will
> > down
> > > > > > convert the
> > > > > >     > > > message
> > > > > >     > > >     > on the
> > > > > >     > > >     >     > broker
> > > > > >     > > >     >     >         >> side.
> > > > > >     > > >     >     >         >> > >     c) At this point we will
> > > have
> > > > to
> > > > > >     > specify a
> > > > > >     > > >     > warning or
> > > > > >     > > >     >     > clearly
> > > > > >     > > >     >     >         >> specify
> > > > > >     > > >     >     >         >> > > in docs that this behavior
> is
> > > > about
> > > > > > to be
> > > > > >     > > > changed for
> > > > > >     > > >     > log
> > > > > >     > > >     >     > compaction.
> > > > > >     > > >     >     >         >> > > 4) a) In next release of the
> > > > > > Broker(x+2), we
> > > > > >     > > > say that
> > > > > >     > > >     > only
> > > > > >     > > >     >     > Tombstone
> > > > > >     > > >     >     >         >> is
> > > > > >     > > >     >     >         >> > > used for log compaction on
> the
> > > > > Broker
> > > > > > side.
> > > > > >     > > >     > Clients(x+1)
> > > > > >     > > >     >     > still is
> > > > > >     > > >     >     >         >> > > supported.
> > > > > >     > > >     >     >         >> > >     b) We upgrade the client
> > to
> > > > > > version
> > > > > >     > > > client(x+2) :
> > > > > >     > > >     > if
> > > > > >     > > >     >     > value is set
> > > > > >     > > >     >     >         >> to
> > > > > >     > > >     >     >         >> > > null, tombstone will not be
> > set
> > > > > >     > automatically.
> > > > > >     > > > The
> > > > > >     > > >     > client
> > > > > >     > > >     >     > will have to
> > > > > >     > > >     >     >         >> > call
> > > > > >     > > >     >     >         >> > > setTombstone() to actually
> set
> > > the
> > > > > >     > tombstone.
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > > We should compare this
> > migration
> > > > > plan
> > > > > > with
> > > > > >     > the
> > > > > >     > > >     > migration
> > > > > >     > > >     >     > plan for
> > > > > >     > > >     >     >         >> magic
> > > > > >     > > >     >     >         >> > > byte bump and do whatever
> > looks
> > > > > good.
> > > > > >     > > >     >     >         >> > > I am just worried that if we
> > go
> > > > down
> > > > > > magic
> > > > > >     > > byte
> > > > > >     > > > route,
> > > > > >     > > >     >     > unless I am
> > > > > >     > > >     >     >         >> > missing
> > > > > >     > > >     >     >         >> > > something, it sounds like
> > kafka
> > > > will
> > > > > > be
> > > > > >     > stuck
> > > > > >     > > > with
> > > > > >     > > >     >     > supporting both
> > > > > >     > > >     >     >         >> null
> > > > > >     > > >     >     >         >> > > value and tombstone bit for
> > log
> > > > > > compaction
> > > > > >     > for
> > > > > >     > > > life
> > > > > >     > > >     > long,
> > > > > >     > > >     >     > which does
> > > > > >     > > >     >     >         >> not
> > > > > >     > > >     >     >         >> > > look like a good end state.
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > > Thanks,
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > > Mayuresh
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32
> > AM,
> > > > > > Mayuresh
> > > > > >     > > > Gharat <
> > > > > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> > > > > >     > > >     >     >         >> > > > wrote:
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > > > Hi Ismael,
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > That's a very good point
> > > which I
> > > > > > might
> > > > > >     > have
> > > > > >     > > > not
> > > > > >     > > >     >     > considered earlier.
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > Here is a plan that I can
> > > think
> > > > > of:
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > Stage 1) The broker from
> now
> > > on,
> > > > > up
> > > > > >     > converts
> > > > > >     > > > the
> > > > > >     > > >     > message
> > > > > >     > > >     >     > to have the
> > > > > >     > > >     >     >         >> > > > tombstone marker. The log
> > > > > compaction
> > > > > >     > thread
> > > > > >     > > > does log
> > > > > >     > > >     >     > compaction
> > > > > >     > > >     >     >         >> based
> > > > > >     > > >     >     >         >> > on
> > > > > >     > > >     >     >         >> > > > both null and tombstone
> > > marker.
> > > > > > This is
> > > > > >     > our
> > > > > >     > > >     > transition
> > > > > >     > > >     >     > period.
> > > > > >     > > >     >     >         >> > > > Stage 2) The next release
> we
> > > > only
> > > > > > say that
> > > > > >     > > log
> > > > > >     > > >     > compaction
> > > > > >     > > >     >     > is based
> > > > > >     > > >     >     >         >> on
> > > > > >     > > >     >     >         >> > > > tombstone marker. (Open
> > source
> > > > > > kafka makes
> > > > > >     > > > this as a
> > > > > >     > > >     >     > policy). By
> > > > > >     > > >     >     >         >> this
> > > > > >     > > >     >     >         >> > > time,
> > > > > >     > > >     >     >         >> > > > the organization which is
> > > moving
> > > > > to
> > > > > > this
> > > > > >     > > > release
> > > > > >     > > >     > will be
> > > > > >     > > >     >     > sure that
> > > > > >     > > >     >     >         >> they
> > > > > >     > > >     >     >         >> > > > have gone through the
> entire
> > > > > > transition
> > > > > >     > > > period.
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > My only goal of doing this
> > is
> > > > that
> > > > > > Kafka
> > > > > >     > > > clearly
> > > > > >     > > >     >     > specifies the end
> > > > > >     > > >     >     >         >> > state
> > > > > >     > > >     >     >         >> > > > about what log compaction
> > > means
> > > > > (is
> > > > > > it
> > > > > >     > null
> > > > > >     > > > value
> > > > > >     > > >     > or a
> > > > > >     > > >     >     > tombstone
> > > > > >     > > >     >     >         >> > marker,
> > > > > >     > > >     >     >         >> > > > but not both).
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > What do you think?
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > Thanks,
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > Mayuresh
> > > > > >     > > >     >     >         >> > > > .
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at
> 9:17
> > > AM,
> > > > > > Ismael
> > > > > >     > > Juma <
> > > > > >     > > >     >     > ismael@juma.me.uk>
> > > > > >     > > >     >     >         >> > wrote:
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > >> One comment below.
> > > > > >     > > >     >     >         >> > > >>
> > > > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at
> > 5:08
> > > > PM,
> > > > > > Mayuresh
> > > > > >     > > > Gharat <
> > > > > >     > > >     >     >         >> > > >>
> gharatmayuresh15@gmail.com
> > > > > >     > > >     >     >         >> > > >> > wrote:
> > > > > >     > > >     >     >         >> > > >>
> > > > > >     > > >     >     >         >> > > >> >    - If we don't bump
> up
> > > the
> > > > > > magic
> > > > > >     > byte,
> > > > > >     > > > on the
> > > > > >     > > >     > broker
> > > > > >     > > >     >     > side, the
> > > > > >     > > >     >     >         >> > > broker
> > > > > >     > > >     >     >         >> > > >> >    will always have to
> > look
> > > > at
> > > > > > both
> > > > > >     > > > tombstone
> > > > > >     > > >     > bit and
> > > > > >     > > >     >     > the value
> > > > > >     > > >     >     >         >> when
> > > > > >     > > >     >     >         >> > > do
> > > > > >     > > >     >     >         >> > > >> the
> > > > > >     > > >     >     >         >> > > >> >    compaction. Assuming
> > we
> > > do
> > > > > > not bump
> > > > > >     > up
> > > > > >     > > > the
> > > > > >     > > >     > magic
> > > > > >     > > >     >     > byte,
> > > > > >     > > >     >     >         >> > > >> >    imagine the broker
> > sees
> > > a
> > > > > > message
> > > > > >     > > which
> > > > > >     > > > does
> > > > > >     > > >     > not
> > > > > >     > > >     >     > have a
> > > > > >     > > >     >     >         >> tombstone
> > > > > >     > > >     >     >         >> > > bit
> > > > > >     > > >     >     >         >> > > >> >    set. The broker does
> > not
> > > > > know
> > > > > > when
> > > > > >     > the
> > > > > >     > > >     > message was
> > > > > >     > > >     >     > produced
> > > > > >     > > >     >     >         >> (i.e.
> > > > > >     > > >     >     >         >> > > >> > whether
> > > > > >     > > >     >     >         >> > > >> >    the message has been
> > up
> > > > > > converted or
> > > > > >     > > > not), it
> > > > > >     > > >     > has
> > > > > >     > > >     >     > to take a
> > > > > >     > > >     >     >         >> > further
> > > > > >     > > >     >     >         >> > > >> > look at
> > > > > >     > > >     >     >         >> > > >> >    the value to see if
> it
> > > is
> > > > > > null or
> > > > > >     > not
> > > > > >     > > in
> > > > > >     > > >     > order to
> > > > > >     > > >     >     > determine
> > > > > >     > > >     >     >         >> if it
> > > > > >     > > >     >     >         >> > > is
> > > > > >     > > >     >     >         >> > > >> a
> > > > > >     > > >     >     >         >> > > >> >    tombstone. The same
> > > logic
> > > > > has
> > > > > > to be
> > > > > >     > > put
> > > > > >     > > > on the
> > > > > >     > > >     >     > consumer as
> > > > > >     > > >     >     >         >> well
> > > > > >     > > >     >     >         >> > > >> because
> > > > > >     > > >     >     >         >> > > >> > the
> > > > > >     > > >     >     >         >> > > >> >    consumer does not
> know
> > > if
> > > > > the
> > > > > >     > message
> > > > > >     > > > has
> > > > > >     > > >     > been up
> > > > > >     > > >     >     > converted or
> > > > > >     > > >     >     >         >> > not.
> > > > > >     > > >     >     >         >> > > >> >       - If we upconvert
> > > while
> > > > > >     > appending,
> > > > > >     > > > this is
> > > > > >     > > >     > not
> > > > > >     > > >     >     > the case,
> > > > > >     > > >     >     >         >> > right?
> > > > > >     > > >     >     >         >> > > >>
> > > > > >     > > >     >     >         >> > > >>
> > > > > >     > > >     >     >         >> > > >> If I understand you
> > > correctly,
> > > > > > this is
> > > > > >     > not
> > > > > >     > > >     > sufficient
> > > > > >     > > >     >     > because the
> > > > > >     > > >     >     >         >> log
> > > > > >     > > >     >     >         >> > > may
> > > > > >     > > >     >     >         >> > > >> have messages appended
> > before
> > > > it
> > > > > > was
> > > > > >     > > > upgraded to
> > > > > >     > > >     > include
> > > > > >     > > >     >     > KIP-87.
> > > > > >     > > >     >     >         >> > > >>
> > > > > >     > > >     >     >         >> > > >> Ismael
> > > > > >     > > >     >     >         >> > > >>
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > > > --
> > > > > >     > > >     >     >         >> > > > -Regards,
> > > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> > > > > >     > > >     >     >         >> > > > (862) 250-7125
> > > > > >     > > >     >     >         >> > > >
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > > --
> > > > > >     > > >     >     >         >> > > -Regards,
> > > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > > > > >     > > >     >     >         >> > > (862) 250-7125
> > > > > >     > > >     >     >         >> > >
> > > > > >     > > >     >     >         >> > The information contained in
> > this
> > > > > email
> > > > > > is
> > > > > >     > > > strictly
> > > > > >     > > >     >     > confidential and for
> > > > > >     > > >     >     >         >> > the use of the addressee only,
> > > > unless
> > > > > >     > otherwise
> > > > > >     > > >     > indicated. If
> > > > > >     > > >     >     > you are
> > > > > >     > > >     >     >         >> not
> > > > > >     > > >     >     >         >> > the intended recipient, please
> > do
> > > > not
> > > > > > read,
> > > > > >     > > copy,
> > > > > >     > > > use or
> > > > > >     > > >     >     > disclose to
> > > > > >     > > >     >     >         >> others
> > > > > >     > > >     >     >         >> > this message or any
> attachment.
> > > > Please
> > > > > > also
> > > > > >     > > > notify the
> > > > > >     > > >     > sender
> > > > > >     > > >     >     > by
> > > > > >     > > >     >     >         >> replying
> > > > > >     > > >     >     >         >> > to this email or by telephone
> > > > (+44(020
> > > > > > 7896
> > > > > >     > > 0011)
> > > > > >     > > > and
> > > > > >     > > >     > then
> > > > > >     > > >     >     > delete the
> > > > > >     > > >     >     >         >> email
> > > > > >     > > >     >     >         >> > and any copies of it.
> Opinions,
> > > > > > conclusion
> > > > > >     > (etc)
> > > > > >     > > > that
> > > > > >     > > >     > do not
> > > > > >     > > >     >     > relate to
> > > > > >     > > >     >     >         >> the
> > > > > >     > > >     >     >         >> > official business of this
> > company
> > > > > shall
> > > > > > be
> > > > > >     > > > understood as
> > > > > >     > > >     >     > neither given
> > > > > >     > > >     >     >         >> nor
> > > > > >     > > >     >     >         >> > endorsed by it. IG is a
> trading
> > > name
> > > > > of
> > > > > > IG
> > > > > >     > > Markets
> > > > > >     > > >     > Limited (a
> > > > > >     > > >     >     > company
> > > > > >     > > >     >     >         >> > registered in England and
> Wales,
> > > > > company
> > > > > >     > number
> > > > > >     > > >     > 04008957) and
> > > > > >     > > >     >     > IG Index
> > > > > >     > > >     >     >         >> > Limited (a company registered
> in
> > > > > > England and
> > > > > >     > > > Wales,
> > > > > >     > > >     > company
> > > > > >     > > >     >     > number
> > > > > >     > > >     >     >         >> > 01190902). Registered address
> at
> > > > > Cannon
> > > > > > Bridge
> > > > > >     > > > House, 25
> > > > > >     > > >     >     > Dowgate Hill,
> > > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG
> Markets
> > > > > Limited
> > > > > >     > > (register
> > > > > >     > > >     > number
> > > > > >     > > >     >     > 195355) and IG
> > > > > >     > > >     >     >         >> > Index Limited (register number
> > > > 114059)
> > > > > > are
> > > > > >     > > > authorised
> > > > > >     > > >     > and
> > > > > >     > > >     >     > regulated by
> > > > > >     > > >     >     >         >> the
> > > > > >     > > >     >     >         >> > Financial Conduct Authority.
> > > > > >     > > >     >     >         >> >
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >> --
> > > > > >     > > >     >     >         >> -Regards,
> > > > > >     > > >     >     >         >> Mayuresh R. Gharat
> > > > > >     > > >     >     >         >> (862) 250-7125
> > > > > >     > > >     >     >         >> The information contained in
> this
> > > > email
> > > > > is
> > > > > >     > > strictly
> > > > > >     > > >     >     > confidential and for
> > > > > >     > > >     >     >         >> the use of the addressee only,
> > > unless
> > > > > > otherwise
> > > > > >     > > >     > indicated. If
> > > > > >     > > >     >     > you are not
> > > > > >     > > >     >     >         >> the intended recipient, please
> do
> > > not
> > > > > > read,
> > > > > >     > copy,
> > > > > >     > > > use or
> > > > > >     > > >     >     > disclose to others
> > > > > >     > > >     >     >         >> this message or any attachment.
> > > Please
> > > > > > also
> > > > > >     > notify
> > > > > >     > > > the
> > > > > >     > > >     > sender
> > > > > >     > > >     >     > by replying
> > > > > >     > > >     >     >         >> to this email or by telephone
> > > (+44(020
> > > > > > 7896
> > > > > >     > 0011)
> > > > > >     > > > and then
> > > > > >     > > >     >     > delete the email
> > > > > >     > > >     >     >         >> and any copies of it. Opinions,
> > > > > > conclusion (etc)
> > > > > >     > > > that do
> > > > > >     > > >     > not
> > > > > >     > > >     >     > relate to the
> > > > > >     > > >     >     >         >> official business of this
> company
> > > > shall
> > > > > be
> > > > > >     > > > understood as
> > > > > >     > > >     >     > neither given nor
> > > > > >     > > >     >     >         >> endorsed by it. IG is a trading
> > name
> > > > of
> > > > > IG
> > > > > >     > Markets
> > > > > >     > > >     > Limited (a
> > > > > >     > > >     >     > company
> > > > > >     > > >     >     >         >> registered in England and Wales,
> > > > company
> > > > > > number
> > > > > >     > > > 04008957)
> > > > > >     > > >     > and
> > > > > >     > > >     >     > IG Index
> > > > > >     > > >     >     >         >> Limited (a company registered in
> > > > England
> > > > > > and
> > > > > >     > > Wales,
> > > > > >     > > >     > company
> > > > > >     > > >     >     > number
> > > > > >     > > >     >     >         >> 01190902). Registered address at
> > > > Cannon
> > > > > > Bridge
> > > > > >     > > > House, 25
> > > > > >     > > >     >     > Dowgate Hill,
> > > > > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
> > > > Limited
> > > > > >     > (register
> > > > > >     > > > number
> > > > > >     > > >     >     > 195355) and IG
> > > > > >     > > >     >     >         >> Index Limited (register number
> > > 114059)
> > > > > are
> > > > > >     > > > authorised and
> > > > > >     > > >     >     > regulated by the
> > > > > >     > > >     >     >         >> Financial Conduct Authority.
> > > > > >     > > >     >     >         >>
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >         > --
> > > > > >     > > >     >     >         > -Regards,
> > > > > >     > > >     >     >         > Mayuresh R. Gharat
> > > > > >     > > >     >     >         > (862) 250-7125
> > > > > >     > > >     >     >         >
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >         --
> > > > > >     > > >     >     >         -Regards,
> > > > > >     > > >     >     >         Mayuresh R. Gharat
> > > > > >     > > >     >     >         (862) 250-7125
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >     The information contained in this email
> > is
> > > > > > strictly
> > > > > >     > > > confidential
> > > > > >     > > >     > and
> > > > > >     > > >     >     > for the use of the addressee only, unless
> > > > otherwise
> > > > > >     > > indicated.
> > > > > >     > > > If
> > > > > >     > > >     > you are
> > > > > >     > > >     >     > not the intended recipient, please do not
> > read,
> > > > > > copy, use
> > > > > >     > or
> > > > > >     > > >     > disclose to
> > > > > >     > > >     >     > others this message or any attachment.
> Please
> > > > also
> > > > > > notify
> > > > > >     > the
> > > > > >     > > > sender
> > > > > >     > > >     > by
> > > > > >     > > >     >     > replying to this email or by telephone
> > (+44(020
> > > > > 7896
> > > > > > 0011)
> > > > > >     > > and
> > > > > >     > > > then
> > > > > >     > > >     > delete
> > > > > >     > > >     >     > the email and any copies of it. Opinions,
> > > > > conclusion
> > > > > > (etc)
> > > > > >     > > > that do
> > > > > >     > > >     > not
> > > > > >     > > >     >     > relate to the official business of this
> > company
> > > > > > shall be
> > > > > >     > > > understood
> > > > > >     > > >     > as
> > > > > >     > > >     >     > neither given nor endorsed by it. IG is a
> > > trading
> > > > > > name of
> > > > > >     > IG
> > > > > >     > > > Markets
> > > > > >     > > >     >     > Limited (a company registered in England
> and
> > > > Wales,
> > > > > > company
> > > > > >     > > > number
> > > > > >     > > >     >     > 04008957) and IG Index Limited (a company
> > > > > registered
> > > > > > in
> > > > > >     > > > England and
> > > > > >     > > >     > Wales,
> > > > > >     > > >     >     > company number 01190902). Registered
> address
> > at
> > > > > > Cannon
> > > > > >     > Bridge
> > > > > >     > > > House,
> > > > > >     > > >     > 25
> > > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG
> > Markets
> > > > > > Limited
> > > > > >     > > > (register
> > > > > >     > > >     > number
> > > > > >     > > >     >     > 195355) and IG Index Limited (register
> number
> > > > > > 114059) are
> > > > > >     > > > authorised
> > > > > >     > > >     > and
> > > > > >     > > >     >     > regulated by the Financial Conduct
> Authority.
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >
> > > > > >     > > >     >     >
> > > > > >     > > >     >
> > > > > >     > > >     >
> > > > > >     > > >     > The information contained in this email is
> strictly
> > > > > > confidential
> > > > > >     > > and
> > > > > >     > > > for
> > > > > >     > > >     > the use of the addressee only, unless otherwise
> > > > > indicated.
> > > > > > If you
> > > > > >     > > > are not
> > > > > >     > > >     > the intended recipient, please do not read, copy,
> > use
> > > > or
> > > > > > disclose
> > > > > >     > > to
> > > > > >     > > > others
> > > > > >     > > >     > this message or any attachment. Please also
> notify
> > > the
> > > > > > sender by
> > > > > >     > > > replying
> > > > > >     > > >     > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>) and
> > > > > then
> > > > > > delete
> > > > > >     > > > the email
> > > > > >     > > >     > and any copies of it. Opinions, conclusion (etc)
> > that
> > > > do
> > > > > > not
> > > > > >     > relate
> > > > > >     > > > to the
> > > > > >     > > >     > official business of this company shall be
> > understood
> > > > as
> > > > > > neither
> > > > > >     > > > given nor
> > > > > >     > > >     > endorsed by it. IG is a trading name of IG
> Markets
> > > > > Limited
> > > > > > (a
> > > > > >     > > company
> > > > > >     > > >     > registered in England and Wales, company number
> > > > 04008957)
> > > > > > and IG
> > > > > >     > > > Index
> > > > > >     > > >     > Limited (a company registered in England and
> Wales,
> > > > > company
> > > > > >     > number
> > > > > >     > > >     > 01190902). Registered address at Cannon Bridge
> > House,
> > > > 25
> > > > > > Dowgate
> > > > > >     > > > Hill,
> > > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited
> (register
> > > > number
> > > > > > 195355)
> > > > > >     > > > and IG
> > > > > >     > > >     > Index Limited (register number 114059) are
> > authorised
> > > > and
> > > > > >     > regulated
> > > > > >     > > > by the
> > > > > >     > > >     > Financial Conduct Authority.
> > > > > >     > > >     >
> > > > > >     > > >
> > > > > >     > > >
> > > > > >     > > > The information contained in this email is strictly
> > > > > confidential
> > > > > > and
> > > > > >     > for
> > > > > >     > > > the use of the addressee only, unless otherwise
> > indicated.
> > > If
> > > > > > you are
> > > > > >     > not
> > > > > >     > > > the intended recipient, please do not read, copy, use
> or
> > > > > > disclose to
> > > > > >     > > others
> > > > > >     > > > this message or any attachment. Please also notify the
> > > sender
> > > > > by
> > > > > >     > replying
> > > > > >     > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>) and then
> > > > > > delete the
> > > > > >     > > email
> > > > > >     > > > and any copies of it. Opinions, conclusion (etc) that
> do
> > > not
> > > > > > relate to
> > > > > >     > > the
> > > > > >     > > > official business of this company shall be understood
> as
> > > > > neither
> > > > > > given
> > > > > >     > > nor
> > > > > >     > > > endorsed by it. IG is a trading name of IG Markets
> > Limited
> > > (a
> > > > > > company
> > > > > >     > > > registered in England and Wales, company number
> 04008957)
> > > and
> > > > > IG
> > > > > > Index
> > > > > >     > > > Limited (a company registered in England and Wales,
> > company
> > > > > > number
> > > > > >     > > > 01190902). Registered address at Cannon Bridge House,
> 25
> > > > > Dowgate
> > > > > > Hill,
> > > > > >     > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > > > > 195355) and
> > > > > >     > IG
> > > > > >     > > > Index Limited (register number 114059) are authorised
> and
> > > > > > regulated by
> > > > > >     > > the
> > > > > >     > > > Financial Conduct Authority.
> > > > > >     > > >
> > > > > >     > > The information contained in this email is strictly
> > > > confidential
> > > > > > and for
> > > > > >     > > the use of the addressee only, unless otherwise
> indicated.
> > If
> > > > you
> > > > > > are not
> > > > > >     > > the intended recipient, please do not read, copy, use or
> > > > disclose
> > > > > > to
> > > > > >     > others
> > > > > >     > > this message or any attachment. Please also notify the
> > sender
> > > > by
> > > > > > replying
> > > > > >     > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>) and then
> > > > delete
> > > > > > the
> > > > > >     > email
> > > > > >     > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > > > > relate to
> > > > > >     > the
> > > > > >     > > official business of this company shall be understood as
> > > > neither
> > > > > > given
> > > > > >     > nor
> > > > > >     > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > > > > company
> > > > > >     > > registered in England and Wales, company number 04008957)
> > and
> > > > IG
> > > > > > Index
> > > > > >     > > Limited (a company registered in England and Wales,
> company
> > > > > number
> > > > > >     > > 01190902). Registered address at Cannon Bridge House, 25
> > > > Dowgate
> > > > > > Hill,
> > > > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> > > > 195355)
> > > > > > and IG
> > > > > >     > > Index Limited (register number 114059) are authorised and
> > > > > > regulated by
> > > > > >     > the
> > > > > >     > > Financial Conduct Authority.
> > > > > >     > >
> > > > > >     > >
> > > > > >     > The information contained in this email is strictly
> > > confidential
> > > > > and
> > > > > > for
> > > > > >     > the use of the addressee only, unless otherwise indicated.
> If
> > > you
> > > > > > are not
> > > > > >     > the intended recipient, please do not read, copy, use or
> > > disclose
> > > > > to
> > > > > > others
> > > > > >     > this message or any attachment. Please also notify the
> sender
> > > by
> > > > > > replying
> > > > > >     > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>) and then
> > > delete
> > > > > > the email
> > > > > >     > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > > relate
> > > > > > to the
> > > > > >     > official business of this company shall be understood as
> > > neither
> > > > > > given nor
> > > > > >     > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > > > company
> > > > > >     > registered in England and Wales, company number 04008957)
> and
> > > IG
> > > > > > Index
> > > > > >     > Limited (a company registered in England and Wales, company
> > > > number
> > > > > >     > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > > > > Hill,
> > > > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355)
> > > > > > and IG
> > > > > >     > Index Limited (register number 114059) are authorised and
> > > > regulated
> > > > > > by the
> > > > > >     > Financial Conduct Authority.
> > > > > >     >
> > > > > >     >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is strictly confidential
> > and
> > > > for
> > > > > > the use of the addressee only, unless otherwise indicated. If you
> > are
> > > > not
> > > > > > the intended recipient, please do not read, copy, use or disclose
> > to
> > > > > others
> > > > > > this message or any attachment. Please also notify the sender by
> > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011
> > <020%207896%200011>) and then delete the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > > to
> > > > > the
> > > > > > official business of this company shall be understood as neither
> > > given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > > > > > registered in England and Wales, company number 04008957) and IG
> > > Index
> > > > > > Limited (a company registered in England and Wales, company
> number
> > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and
> > > > IG
> > > > > > Index Limited (register number 114059) are authorised and
> regulated
> > > by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > The information contained in this email is strictly confidential
> and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If you
> are
> > > not
> > > > > the intended recipient, please do not read, copy, use or disclose
> to
> > > > others
> > > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>)
> > and then delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to
> > > > the
> > > > > official business of this company shall be understood as neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > > registered in England and Wales, company number 04008957) and IG
> > Index
> > > > > Limited (a company registered in England and Wales, company number
> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and regulated
> > by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> > and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> > and then delete the email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Ismael

Did you see our email this morning, what's your thoughts on this approach to instead we simply have a brand new policy?

Cheers
Mike


Sent using OWA for iPhone
________________________________________
From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <is...@juma.me.uk>
Sent: Tuesday, December 13, 2016 11:30:05 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Yes, this is actually tricky to do in a way where we both have the desired
semantics and maintain compatibility. When someone creates a
`ProducerRecord` with a `null` value today, the producer doesn't know if
it's meant to be a tombstone or not. For V3 messages, it's easy when the
constructor that takes a tombstone is used. However, if any other
constructor is used, it's not clear. A couple of options I can think of,
none of them particularly nice:

1. Have a third state where tombstone = unknown and the broker would set
the tombstone bit if the value was null and the topic was compacted. People
that wanted to pass a non-null value for the tombstone would have to use
the constructor that takes a tombstone. The drawbacks: third state for
tombstone in message format, message conversion at the broker for a common
case.

2. Extend MetadataResponse to optionally include topic configs, which would
make it possible for the producer to be smarter about setting the
tombstone. It would only do it if a tombstone was not passed explicitly,
the value was null and the topic was compacted. The main drawback is that
the producer would be getting a bit more data for each topic even though it
probably won't use it most of the time. Extending MetadataResponse to
return topic configs would be useful for other reasons as well, so that
part seems OK.

In addition, for both proposals, we could consider adding warnings to the
documentation that the behaviour of the constructors that don't take a
tombstone would change in the next major release so that tombstone = false.
Not sure if this would be worth it though.

Ismael

On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:

> Michael,
>
> It kind of depends on how you want to interpret the tombstone flag. If it's
> purely a producer-facing Kafka-level thing that we treat as internal to the
> broker and log cleaner once the record is sent, then your approach makes
> sense. You're just moving copying the null-indicates-delete behavior of the
> old constructor into the tombstone flag.
>
> However, if you want this change to more generally decouple the idea of
> deletion and null values, then you are sometimes converting what might be a
> completely valid null value that doesn't indicate deletion into a
> tombstone. Downstream applications could potentially handle these cases
> differently given the separation of deletion from value.
>
> I guess the question is if we want to try to support the latter even for
> topics where we have older produce requests. An example where this could
> come up is in something like a CDC Connector. If we try to support the
> semantic difference, a connector might write changes to Kafka using the
> tombstone flag to indicate when a row was truly deleted (vs an update that
> sets it to null but still present; this probably makes more sense for CDC
> from document stores or extracting single columns). There are various
> reasons we might want to maintain the full log and not turn compaction on
> (or just use a time-based retention policy), but downstream applications
> might care to know the difference between a delete and a null value. In
> fact both versions of the same log (compacted and time-retention) could be
> useful and I don't think it'll be uncommon to maintain both or use KIP-71
> to maintain a hybrid compacted/retention topic.
>
> -Ewen
>
> On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jay,
> >
> > Why wouldn't that work, the tombstone value is only looked at by the
> > broker, on a topic configured for compaction as such is benign on non
> > compacted topics. This is as much as sending a null value currently
> >
> >
> > Regards
> > Mike
> >
> >
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jay Kreps <ja...@confluent.io>
> > Sent: Sunday, December 11, 2016 8:58:53 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hey Michael,
> >
> > I'm not quite sure that works as that would translate ALL null values to
> > tombstones, even for non-compacted topics that use null as an acceptable
> > value sent by the producer and expected by the consumer.
> >
> > -Jay
> >
> > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Ewen,
> > >
> > > I think the easiest way to show this is with code.
> > >
> > > As you can see we keep the existing behaviour for code/binaries calling
> > > the pre-existing constructors, whereby if the value is null the
> tombstone
> > > is set to true.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > >
> > >     /**
> > >      * Creates a record with a specified timestamp to be sent to a
> > > specified topic and partition
> > >      *
> > >      * @param topic The topic the record will be appended to
> > >      * @param partition The partition to which the record should be
> sent
> > >      * @param timestamp The timestamp of the record
> > >      * @param tombstone if the record should be treated as a tombstone
> if
> > > the topic is compacted
> > >      * @param key The key that will be included in the record
> > >      * @param value The record contents
> > >      */
> > >     public ProducerRecord(String topic, Integer partition, Boolean
> > > tombstone, Long timestamp, K key, V value) {
> > >         if (topic == null)
> > >             throw new IllegalArgumentException("Topic cannot be
> null.");
> > >         if (timestamp != null && timestamp < 0)
> > >             throw new IllegalArgumentException(
> > >                     String.format("Invalid timestamp: %d. Timestamp
> > should
> > > always be non-negative or null.", timestamp));
> > >         if (partition != null && partition < 0)
> > >             throw new IllegalArgumentException(
> > >                     String.format("Invalid partition: %d. Partition
> > number
> > > should always be non-negative or null.", partition));
> > >         if (!tombstone && value == null){
> > >             throw new IllegalArgumentException(
> > >                     String.format("Tombstone must be true if null
> > value"));
> > >         }
> > >         this.topic = topic;
> > >         this.partition = partition;
> > >         this.tombstone = tombstone;
> > >         this.key = key;
> > >         this.value = value;
> > >         this.timestamp = timestamp;
> > >     }
> > >
> > >     public ProducerRecord(String topic, Integer partition, Long
> > timestamp,
> > > K key, V value) {
> > >         this(topic, partition, value == null, timestamp, key, value);
> > >     }
> > > ________________________________________
> > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> > > Sent: Sunday, December 11, 2016 5:45 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > > It seemed like this was addressed in the migration section, wasn't it?
> > The
> > > V2 vs V3 requests and the fact that the broker will downgrade the
> message
> > > format (losing zero copy) if you issues a V2 request to a broker using
> V3
> > > format handles compatibility, does it not? The existing consumers won't
> > see
> > > the extra metadata in the value, but they will get a null instead and
> > treat
> > > it as a tombstone. Certainly there is a performance impact, but it
> seemed
> > > compatible.
> > >
> > > I'm worried about this though:
> > >
> > >
> > >    - *NOTE *: With the new version of producer client using
> > ProduceRequest
> > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and
> null
> > > value
> > >    should not be allowed as the older version of consumer using
> > > FetchRequest
> > >    V2 will think of this as a tombstone when its actually not.
> > >
> > >
> > > Unless I'm misunderstanding, this ends up breaking binary compatibility
> > for
> > > the Java API. It sounds like this suggests that passing a null value to
> > the
> > > existing ProducerRecord constructors would generate an exception since
> > you
> > > didn't explicitly enable the tombstone (via whatever new constructor is
> > > provided). But that means you can't swap in a newer client jar without
> > > recompiling and get the same behavior if your app deletes keys using
> the
> > > current approach because your app will start throwing exceptions. Maybe
> > > this is a tradeoff we're ok with, but we've tried pretty hard to avoid
> > > breaking compatibility like this so far -- it makes picking up bug
> fixes
> > in
> > > the clients harder for users of frameworks that have to pin to earlier
> > > default versions for compatibility.
> > >
> > > But then later the KIP says:
> > >
> > >
> > >    - The old constructors for ProducerRecord without this parameter
> will
> > be
> > >    kept but updated so that their default behaviour if setting null
> value
> > > will
> > >    be the tombstone will be set to true to keep existing behaviour.
> > >
> > >
> > > So maybe I am misinterpreting something.
> > >
> > > And just a nit re: motivation section, item 6 would be clearer for a
> > union
> > > schema with null (or optional schemas in other formats), e.g. [null,
> > > string], in which case losing the schema truly is losing information
> > > (whereas null is already the only valid value for a pure null schema).
> > >
> > > -Ewen
> > >
> > >
> > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> > > wrote:
> > >
> > > > Hi Jay,
> > > >
> > > > Good point this detail is missing in the KIP write up. Ive added this
> > > now.
> > > >
> > > > Essentially simply just upgrading the clients we do not expect any
> > client
> > > > app code change needed.
> > > >
> > > > Cheers
> > > > Mike
> > > > ________________________________________
> > > > From: Jay Kreps <ja...@confluent.io>
> > > > Sent: Saturday, December 10, 2016 10:51 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >
> > > > Michael,
> > > >
> > > > The compatibility section goes through the migration path, but isn't
> > the
> > > > bigger compatibility issue with existing apps? There are many
> (probably
> > > > thousands) of apps in production that use this feature and send null
> to
> > > > mean delete. It seems like this would break compatibility with them,
> > and
> > > > they would have to be rewritten to right?
> > > >
> > > > -Jay
> > > >
> > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > 4) On v3 we honour the tombstone. As such we expect it to be set
> > > > correctly
> > > > > as per the KIP.
> > > > >
> > > > > 4.1) why would we want to produce an error when its v3? This is the
> > > exact
> > > > > purpose to support non-null tombstone’s
> > > > > 4.2) again here im unclear on the question on the v3, produce
> request
> > > we
> > > > > expect the tombstone flag to be set correctly.
> > > > >
> > > > > 4.4) compaction only occurs on compacted topics, the bit makes no
> > > > > difference and not looked at on un-compacted (time/size based
> > > eviction).
> > > > >
> > > > >
> > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> > > > >
> > > > >     Hi, Michael,
> > > > >
> > > > >     4. Then, I think I misunderstood this point. Could you document
> > the
> > > > >     following points in the wiki?
> > > > >     4.1 If producer V3 sets tombstone, but provides a non-null
> value,
> > > > does
> > > > > the
> > > > >     send() get an error or does the producer automatically set the
> > > value
> > > > to
> > > > >     null?
> > > > >     4.2 If producer V3 doesn't set tombstone, but provides a null
> > > value,
> > > > > does
> > > > >     the send() get an error or does the producer automatically sets
> > the
> > > > >     tombstone?
> > > > >     4.3 Does the broker only expect messages that (a) have no
> > tombstone
> > > > and
> > > > >     non-null value; (b) have tombstone and null value and reject
> the
> > > > > messages
> > > > >     with an error code otherwise?
> > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted
> on
> > > > not?
> > > > >
> > > > >     Thanks,
> > > > >
> > > > >     Jun
> > > > >
> > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > >     wrote:
> > > > >
> > > > >     > Not at all.  This only acts on compacted topics just as what
> > > occurs
> > > > > today
> > > > >     >
> > > > >     > Sent using OWA for iPhone
> > > > >     > ________________________________________
> > > > >     > From: Jun Rao <ju...@confluent.io>
> > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > > > >     > To: dev@kafka.apache.org
> > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > > >     >
> > > > >     > Hi, Michael,
> > > > >     >
> > > > >     > 4. Hmm, does that mean the new client library can never send
> a
> > > null
> > > > > message
> > > > >     > even to a regular topic? This seems like a change of the
> > existing
> > > > > behavior.
> > > > >     >
> > > > >     > Thanks,
> > > > >     >
> > > > >     > Jun
> > > > >     >
> > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > >     > wrote:
> > > > >     >
> > > > >     > > Hi Jun,
> > > > >     > >
> > > > >     > > Re 4) That's because we expect the tombstone value to be
> set
> > > > > correctly if
> > > > >     > > message bit is 2, as such if an older client sends in on
> old
> > > > > message the
> > > > >     > > message is upcast and the bit is set correctly. And such no
> > > > longer
> > > > > need
> > > > >     > to
> > > > >     > > check the value. Mayuresh can you confirm my thinking and
> > > > > understanding
> > > > >     > of
> > > > >     > > what we've discussed?
> > > > >     > >
> > > > >     > > The second point I understand what you're getting at now my
> > > > > apologies.
> > > > >     > Yes
> > > > >     > > this makes sense to save on touching the message, if we're
> > the
> > > > > only kip
> > > > >     > > going in, in this release.
> > > > >     > >
> > > > >     > > Cheers
> > > > >     > > Mike
> > > > >     > >
> > > > >     > > Sent using OWA for iPhone
> > > > >     > > ________________________________________
> > > > >     > > From: Jun Rao <ju...@confluent.io>
> > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > > > >     > > To: dev@kafka.apache.org
> > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > > >     > >
> > > > >     > > Hi, Michael,
> > > > >     > >
> > > > >     > > 4. Is this updated in the wiki? The text "If the magic byte
> > on
> > > > > message is
> > > > >     > > 2, the broker should use the tombstone bit for log
> > compaction."
> > > > > doesn't
> > > > >     > > seem to have changed.
> > > > >     > >
> > > > >     > > 2. My point is that if we change the message format just
> for
> > > this
> > > > > KIP, we
> > > > >     > > should consider whether it's worth optimizing the down
> > > conversion
> > > > > path
> > > > >     > > (i.e., decide whether a conversion is needed by just
> looking
> > at
> > > > the
> > > > >     > > tombstone bit in the wrapper message) since tombstone will
> be
> > > > used
> > > > >     > rarely.
> > > > >     > > However, if the message format change here is combined with
> > > other
> > > > > KIPs,
> > > > >     > > then this optimization likely won't be needed. The latter
> > > > probably
> > > > > makes
> > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> > > > >     > >
> > > > >     > > Other than those, +1 from me,
> > > > >     > >
> > > > >     > > Thanks,
> > > > >     > >
> > > > >     > > Jun
> > > > >     > >
> > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > >     > > wrote:
> > > > >     > >
> > > > >     > > > Hi Jun
> > > > >     > > >
> > > > >     > > > do we have your vote on this now?
> > > > >     > > >
> > > > >     > > > Any other concerns?
> > > > >     > > >
> > > > >     > > > Cheers
> > > > >     > > > Mike
> > > > >     > > >
> > > > >     > > > Sent using OWA for iPhone
> > > > >     > > > ________________________________________
> > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > > > >     > > > To: dev@kafka.apache.org
> > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> Flag
> > > > >     > > >
> > > > >     > > > Hi Jun,
> > > > >     > > >
> > > > >     > > > Have updated. Thanks again for the feedback.
> > > > >     > > >
> > > > >     > > > Agree yes we should align up when it gets to that, I
> assume
> > > > > you’ve
> > > > >     > > flagged
> > > > >     > > > the same in the other KIP?
> > > > >     > > >
> > > > >     > > > I think for now let’s get this KIP approved, then we can
> > > start
> > > > > the work
> > > > >     > > to
> > > > >     > > > get an initial PR, then we can discuss how to align the
> two
> > > if
> > > > > needed.
> > > > >     > > >
> > > > >     > > > Cheers,
> > > > >     > > > Mike
> > > > >     > > >
> > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io>
> wrote:
> > > > >     > > >
> > > > >     > > >     Hi, Michael,
> > > > >     > > >
> > > > >     > > >     For 2), this is fine. Could you update the KIP wiki
> to
> > > make
> > > > > this
> > > > >     > and
> > > > >     > > > other
> > > > >     > > >     points clearer? Other than that, the KIP looks good
> to
> > > me.
> > > > >     > > >
> > > > >     > > >     An orthogonal thing is that there are other KIPs such
> > as
> > > > > KIP-98
> > > > >     > that
> > > > >     > > > also
> > > > >     > > >     intend to change the message format. If they all get
> > > > > approved, we
> > > > >     > > > should
> > > > >     > > >     think about whether it's better to just bump up the
> > magic
> > > > > byte once
> > > > >     > > to
> > > > >     > > >     incorporate multiple format changes like we did in
> > > > > KIP-31/KIP-32.
> > > > >     > > >
> > > > >     > > >     Thanks,
> > > > >     > > >
> > > > >     > > >     Jun
> > > > >     > > >
> > > > >     > > >
> > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > > > >     > > Michael.Pearce@ig.com>
> > > > >     > > >     wrote:
> > > > >     > > >
> > > > >     > > >     > Hi Jao
> > > > >     > > >     >
> > > > >     > > >     > Thanks for the response. Sorry for slow reply, both
> > > with
> > > > > personal
> > > > >     > > > sickness
> > > > >     > > >     > and also battling some critical issues encountered
> > > since
> > > > >     > upgrading
> > > > >     > > to
> > > > >     > > >     > 0.10.1.0
> > > > >     > > >     >
> > > > >     > > >     > 1) Thans for spotting, Document error where we
> > branched
> > > > > this KIP
> > > > >     > > from
> > > > >     > > >     > KIP-82, will get that removed.
> > > > >     > > >     > 2) Intent is to do this just at the record message
> > > level.
> > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> > corrected.
> > > > >     > > >     > 4) As per discussion thread we will support
> > tombstone +
> > > > > null
> > > > >     > value,
> > > > >     > > >     > tombstone + non null value, no tombstone + null
> > value.
> > > > >     > > >     > 5) I believe this was in the discussion thread,
> > > @Mayuresh
> > > > > is this
> > > > >     > > >     > something we’ve overlooked? I thought we would down
> > > > > convert and
> > > > >     > > > remove the
> > > > >     > > >     > value so the old consumer had existing behavior, or
> > is
> > > > > there
> > > > >     > > > something we
> > > > >     > > >     > haven’t thought about?
> > > > >     > > >     >
> > > > >     > > >     > Cheers
> > > > >     > > >     > Mike
> > > > >     > > >     >
> > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
> > > > wrote:
> > > > >     > > >     >
> > > > >     > > >     >     Hi, Michael,
> > > > >     > > >     >
> > > > >     > > >     >     Thanks for the KIP. A few comments below.
> > > > >     > > >     >
> > > > >     > > >     >     1. The message format change contains
> > > "HeadersLength
> > > > >     > Headers".
> > > > >     > > > Is that
> > > > >     > > >     >     intended?
> > > > >     > > >     >
> > > > >     > > >     >     2. For compressed messageset, is the tombstone
> > bit
> > > > > only set
> > > > >     > at
> > > > >     > > > the
> > > > >     > > >     > shallow
> > > > >     > > >     >     level? Do we always leave that bit in the
> wrapper
> > > > > message
> > > > >     > > unset?
> > > > >     > > > An
> > > > >     > > >     >     alternative is to set the tombstone bit in the
> > > > wrapper
> > > > > if at
> > > > >     > > > least one
> > > > >     > > >     >     inner message has the tombstone bit set. This
> > makes
> > > > > things a
> > > > >     > > bit
> > > > >     > > > more
> > > > >     > > >     >     complicated, but we could potentially exploit
> > that
> > > > for
> > > > >     > > > optimizing down
> > > > >     > > >     >     conversion. For example, we only need to
> convert
> > > > > messages
> > > > >     > with
> > > > >     > > > magic 2
> > > > >     > > >     > to
> > > > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > > > > (conversion is
> > > > >     > > > always
> > > > >     > > >     > needed
> > > > >     > > >     >     from magic 2 to magic 0). Not sure if the
> > > > optimization
> > > > > is
> > > > >     > worth
> > > > >     > > > the
> > > > >     > > >     >     complexity though.
> > > > >     > > >     >
> > > > >     > > >     >     3. The referencing of the new version of
> > > > >     > > > ProducerRequest/FetchRequest
> > > > >     > > >     > is
> > > > >     > > >     >     inconsistent (v4 vs v3). Since our convention
> > > starts
> > > > at
> > > > >     > version
> > > > >     > > > at 0, I
> > > > >     > > >     >     think the new version would be 3.
> > > > >     > > >     >
> > > > >     > > >     >     4. "If the magic byte on message is 2, the
> broker
> > > > > should use
> > > > >     > > the
> > > > >     > > >     > tombstone
> > > > >     > > >     >     bit for log compaction." What about null value?
> > My
> > > > >     > > understanding
> > > > >     > > > is
> > > > >     > > >     > that
> > > > >     > > >     >     null value will be treated the same as setting
> > the
> > > > > tombstone
> > > > >     > > bit.
> > > > >     > > >     >
> > > > >     > > >     >     5. For the migration path, it would be useful
> to
> > > > > describe the
> > > > >     > > > down
> > > > >     > > >     >     conversion path to consumers (i.e., brokers on
> > > > message
> > > > > format
> > > > >     > > > 0.10.2
> > > > >     > > >     > and
> > > > >     > > >     >     consumers on older version).
> > > > >     > > >     >
> > > > >     > > >     >     Thanks,
> > > > >     > > >     >
> > > > >     > > >     >     Jun
> > > > >     > > >     >
> > > > >     > > >     >
> > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael
> Pearce <
> > > > >     > > > Michael.Pearce@ig.com
> > > > >     > > >     > >
> > > > >     > > >     >     wrote:
> > > > >     > > >     >
> > > > >     > > >     >     > Hi All,
> > > > >     > > >     >     >
> > > > >     > > >     >     > We have been discussing in the below thread
> and
> > > > final
> > > > >     > changes
> > > > >     > > > have
> > > > >     > > >     > been
> > > > >     > > >     >     > made to the KIP wiki based on these
> > discussions.
> > > > >     > > >     >     >
> > > > >     > > >     >     > We would now like to put to the vote the
> > > following
> > > > > KIP:
> > > > >     > > >     >     > https://cwiki.apache.org/
> > > > > confluence/display/KAFKA/KIP-
> > > > >     > 87+-+
> > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > > > >     > > >     >     >
> > > > >     > > >     >     > This kip is for having a distinct compaction
> > > > > attribute
> > > > >     > > > “tombstone”
> > > > >     > > >     > flag
> > > > >     > > >     >     > instead of relying on null value, allowing
> > > non-null
> > > > > value
> > > > >     > > > delete
> > > > >     > > >     > messages.
> > > > >     > > >     >     >
> > > > >     > > >     >     > Many thanks,
> > > > >     > > >     >     > Michael
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> > > > >     > > Michael.Pearce@ig.com>
> > > > >     > > >     > wrote:
> > > > >     > > >     >     >
> > > > >     > > >     >     >     Hi Mayuresh,
> > > > >     > > >     >     >
> > > > >     > > >     >     >     LGTM. Ive just made one small adjustment
> > > > > updating the
> > > > >     > > wire
> > > > >     > > >     > protocol to
> > > > >     > > >     >     > show the magic byte bump.
> > > > >     > > >     >     >
> > > > >     > > >     >     >     Do we think we’re good to put to a vote?
> Is
> > > > > there any
> > > > >     > > > other bits
> > > > >     > > >     >     > needing discussion?
> > > > >     > > >     >     >
> > > > >     > > >     >     >     Cheers
> > > > >     > > >     >     >     Mike
> > > > >     > > >     >     >
> > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> > > > >     > > >     > gharatmayuresh15@gmail.com>
> > > > >     > > >     >     > wrote:
> > > > >     > > >     >     >
> > > > >     > > >     >     >         Hi Michael,
> > > > >     > > >     >     >
> > > > >     > > >     >     >         I have updated the migration section
> of
> > > the
> > > > > KIP.
> > > > >     > Can
> > > > >     > > > you
> > > > >     > > >     > please
> > > > >     > > >     >     > take a look?
> > > > >     > > >     >     >
> > > > >     > > >     >     >         Thanks,
> > > > >     > > >     >     >
> > > > >     > > >     >     >         Mayuresh
> > > > >     > > >     >     >
> > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM,
> > Mayuresh
> > > > > Gharat <
> > > > >     > > >     >     > gharatmayuresh15@gmail.com
> > > > >     > > >     >     >         > wrote:
> > > > >     > > >     >     >
> > > > >     > > >     >     >         > Hi Michael,
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > That whilst sending tombstone and
> non
> > > > null
> > > > > value,
> > > > >     > > the
> > > > >     > > >     > consumer
> > > > >     > > >     >     > can expect
> > > > >     > > >     >     >         > only to receive the non-null
> message
> > > only
> > > > > in step
> > > > >     > > > (3) is
> > > > >     > > >     > this
> > > > >     > > >     >     > correct?
> > > > >     > > >     >     >         > ---> I do agree with you here.
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > Becket, Ismael : can you guys
> review
> > > the
> > > > >     > migration
> > > > >     > > > plan
> > > > >     > > >     > listed
> > > > >     > > >     >     > above using
> > > > >     > > >     >     >         > magic byte?
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > Thanks,
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > Mayuresh
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
> > > Michael
> > > > > Pearce <
> > > > >     > > >     >     > Michael.Pearce@ig.com>
> > > > >     > > >     >     >         > wrote:
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         >> Many thanks for this Mayuresh. I
> > don't
> > > > > have any
> > > > >     > > >     > objections.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> I assume we should state:
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> That whilst sending tombstone and
> > non
> > > > null
> > > > >     > value,
> > > > >     > > > the
> > > > >     > > >     > consumer
> > > > >     > > >     >     > can expect
> > > > >     > > >     >     >         >> only to receive the non-null
> message
> > > > only
> > > > > in
> > > > >     > step
> > > > >     > > > (3) is
> > > > >     > > >     > this
> > > > >     > > >     >     > correct?
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Cheers
> > > > >     > > >     >     >         >> Mike
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Sent using OWA for iPhone
> > > > >     > > >     >     >         >> ______________________________
> > > > __________
> > > > >     > > >     >     >         >> From: Mayuresh Gharat <
> > > > >     > gharatmayuresh15@gmail.com
> > > > >     > > >
> > > > >     > > >     >     >         >> Sent: Thursday, November 17, 2016
> > > > 5:18:41
> > > > > PM
> > > > >     > > >     >     >         >> To: dev@kafka.apache.org
> > > > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 -
> Add
> > > > > Compaction
> > > > >     > > > Tombstone
> > > > >     > > >     > Flag
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Hi Ismael,
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Thanks for the explanation.
> > > > >     > > >     >     >         >> Specially I like this part where
> in
> > > you
> > > > >     > mentioned
> > > > >     > > > we can
> > > > >     > > >     > get
> > > > >     > > >     >     > rid of the
> > > > >     > > >     >     >         >> older null value support for log
> > > > > compaction
> > > > >     > later
> > > > >     > > > on,
> > > > >     > > >     > here :
> > > > >     > > >     >     >         >> We can't change semantics of the
> > > message
> > > > > format
> > > > >     > > > without
> > > > >     > > >     > having
> > > > >     > > >     >     > a long
> > > > >     > > >     >     >         >> transition period. And we can't
> rely
> > > > >     > > >     >     >         >> on people reading documentation or
> > > > acting
> > > > > on a
> > > > >     > > > warning for
> > > > >     > > >     >     > something so
> > > > >     > > >     >     >         >> fundamental. As such, my take is
> > that
> > > we
> > > > > need to
> > > > >     > > > bump the
> > > > >     > > >     > magic
> > > > >     > > >     >     > byte. The
> > > > >     > > >     >     >         >> good news is
> > > > >     > > >     >     >         >> that we don't have to support all
> > > > versions
> > > > >     > > forever.
> > > > >     > > > We
> > > > >     > > >     > have
> > > > >     > > >     >     > said that we
> > > > >     > > >     >     >         >> will support direct upgrades for 2
> > > > years.
> > > > > That
> > > > >     > > > means that
> > > > >     > > >     >     > message format
> > > > >     > > >     >     >         >> version n could, in theory, be
> > > removed 2
> > > > > years
> > > > >     > > > after the
> > > > >     > > >     > it's
> > > > >     > > >     >     > introduced.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Just a heads up, I would like to
> > > mention
> > > > > that
> > > > >     > even
> > > > >     > > > without
> > > > >     > > >     >     > bumping magic
> > > > >     > > >     >     >         >> byte, we will *NOT* loose zero
> copy
> > as
> > > > in
> > > > > the
> > > > >     > > > client(x+1)
> > > > >     > > >     > in my
> > > > >     > > >     >     >         >> explanation
> > > > >     > > >     >     >         >> above will convert internally a
> null
> > > > > value to
> > > > >     > > have a
> > > > >     > > >     > tombstone
> > > > >     > > >     >     > bit set and
> > > > >     > > >     >     >         >> a tombstone bit set to have a null
> > > value
> > > > >     > > > automatically
> > > > >     > > >     >     > internally and by
> > > > >     > > >     >     >         >> the time we move to version (x+2),
> > the
> > > > > clients
> > > > >     > > > would have
> > > > >     > > >     >     > upgraded.
> > > > >     > > >     >     >         >> Obviously if we support a request
> > from
> > > > >     > > consumer(x),
> > > > >     > > > we
> > > > >     > > >     > will
> > > > >     > > >     >     > loose zero
> > > > >     > > >     >     >         >> copy
> > > > >     > > >     >     >         >> but that is the same case with
> magic
> > > > byte.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> But if magic byte bump makes life
> > > easier
> > > > > for
> > > > >     > > > transition
> > > > >     > > >     > for the
> > > > >     > > >     >     > above
> > > > >     > > >     >     >         >> reasons that you explained, I am
> OK
> > > with
> > > > > it
> > > > >     > since
> > > > >     > > > we are
> > > > >     > > >     > going
> > > > >     > > >     >     > to meet the
> > > > >     > > >     >     >         >> end goal down the road :)
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> On a side note can we update the
> doc
> > > > here
> > > > > on
> > > > >     > magic
> > > > >     > > > byte
> > > > >     > > >     > to say
> > > > >     > > >     >     > that "*it
> > > > >     > > >     >     >         >> should be bumped whenever the
> > message
> > > > > format is
> > > > >     > > > changed
> > > > >     > > >     > or the
> > > > >     > > >     >     >         >> interpretation of message format
> > > (usage
> > > > > of the
> > > > >     > > > reserved
> > > > >     > > >     > bits as
> > > > >     > > >     >     > well) is
> > > > >     > > >     >     >         >> changed*".
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Hi Michael,
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Here is the update plan that we
> > > > discussed
> > > > >     > offline
> > > > >     > > >     > yesterday :
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Currently the magic-byte which
> > > > > corresponds to
> > > > >     > the
> > > > >     > > >     >     > "message.format.version"
> > > > >     > > >     >     >         >> is set to 1.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> 1) On broker it will be set to 1
> > > > > initially.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> 2) When a producer client sends a
> > > > message
> > > > > with
> > > > >     > > > magic-byte
> > > > >     > > >     > = 2,
> > > > >     > > >     >     > since the
> > > > >     > > >     >     >         >> broker is on magic-byte = 1, we
> will
> > > > down
> > > > >     > convert
> > > > >     > > > it,
> > > > >     > > >     > which
> > > > >     > > >     >     > means if the
> > > > >     > > >     >     >         >> tombstone bit is set, the value
> will
> > > be
> > > > > set to
> > > > >     > > > null. A
> > > > >     > > >     > consumer
> > > > >     > > >     >     >         >> understanding magic-byte = 1, will
> > > still
> > > > > work
> > > > >     > with
> > > > >     > > > this. A
> > > > >     > > >     >     > consumer
> > > > >     > > >     >     >         >> working
> > > > >     > > >     >     >         >> with magic-byte =2 will also be
> able
> > > to
> > > > >     > understand
> > > > >     > > > this,
> > > > >     > > >     > since
> > > > >     > > >     >     > it
> > > > >     > > >     >     >         >> understands the tombstone.
> > > > >     > > >     >     >         >> Now there is still the question of
> > > > > supporting a
> > > > >     > > >     > non-tombstone
> > > > >     > > >     >     > and null
> > > > >     > > >     >     >         >> value from producer client with
> > > > > magic-byte = 2.*
> > > > >     > > (I
> > > > >     > > > am
> > > > >     > > >     > not sure
> > > > >     > > >     >     > if we
> > > > >     > > >     >     >         >> should support this. Ismael/Becket
> > can
> > > > > comment
> > > > >     > > > here)*
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> 3) When almost all the clients
> have
> > > > > upgraded,
> > > > >     > the
> > > > >     > > >     >     > message.format.version
> > > > >     > > >     >     >         >> on
> > > > >     > > >     >     >         >> the broker can be changed to 2,
> > where
> > > in
> > > > > the
> > > > >     > down
> > > > >     > > >     > conversion in
> > > > >     > > >     >     > the above
> > > > >     > > >     >     >         >> step will not happen. If at this
> > point
> > > > we
> > > > > get a
> > > > >     > > > consumer
> > > > >     > > >     >     > request from a
> > > > >     > > >     >     >         >> older consumer, we might have to
> > down
> > > > > convert
> > > > >     > > where
> > > > >     > > > in we
> > > > >     > > >     > loose
> > > > >     > > >     >     > zero copy,
> > > > >     > > >     >     >         >> but these cases should be rare.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Becket can you review this plan
> and
> > > add
> > > > > more
> > > > >     > > > details if I
> > > > >     > > >     > have
> > > > >     > > >     >     >         >> missed/wronged something, before
> we
> > > put
> > > > > it on
> > > > >     > KIP.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Thanks,
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Mayuresh
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
> > > > Michael
> > > > >     > Pearce <
> > > > >     > > >     >     > Michael.Pearce@ig.com>
> > > > >     > > >     >     >         >> wrote:
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> > Thanks guys, for discussing this
> > > > > offline and
> > > > >     > > > getting
> > > > >     > > >     > some
> > > > >     > > >     >     > consensus.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > So its clear for myself and
> others
> > > > what
> > > > > is
> > > > >     > > > proposed now
> > > > >     > > >     > (i
> > > > >     > > >     >     > think i
> > > > >     > > >     >     >         >> > understand, but want to make
> sure)
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Could i ask either directly
> update
> > > the
> > > > > kip to
> > > > >     > > > detail the
> > > > >     > > >     >     > migration
> > > > >     > > >     >     >         >> > strategy, or (re-)state your
> > offline
> > > > > discussed
> > > > >     > > and
> > > > >     > > >     > agreed
> > > > >     > > >     >     > migration
> > > > >     > > >     >     >         >> > strategy based on a magic byte
> is
> > in
> > > > > this
> > > > >     > > thread.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > The main original driver for the
> > KIP
> > > > > was to
> > > > >     > > > support
> > > > >     > > >     >     > compaction where
> > > > >     > > >     >     >         >> value
> > > > >     > > >     >     >         >> > isn't null, based off the
> > > discussions
> > > > on
> > > > >     > KIP-82
> > > > >     > > > thread.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > We should be able to support
> > > > > non-tombstone +
> > > > >     > > null
> > > > >     > > > value
> > > > >     > > >     > by the
> > > > >     > > >     >     >         >> completion
> > > > >     > > >     >     >         >> > of the KIP, as we noted when
> > > > discussing
> > > > > this
> > > > >     > > kip,
> > > > >     > > > having
> > > > >     > > >     >     > logic based on
> > > > >     > > >     >     >         >> a
> > > > >     > > >     >     >         >> > null value isn't very clean and
> > also
> > > > > separates
> > > > >     > > the
> > > > >     > > >     > concerns.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > As discussed already though we
> can
> > > > > split this
> > > > >     > > into
> > > > >     > > >     > KIP-87a
> > > > >     > > >     >     > and KIP-87b
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Where we look to deliver KIP-87a
> > on
> > > a
> > > > >     > compacted
> > > > >     > > > topic
> > > > >     > > >     > (to
> > > > >     > > >     >     > address the
> > > > >     > > >     >     >         >> > immediate issues)
> > > > >     > > >     >     >         >> > * tombstone + null value
> > > > >     > > >     >     >         >> > * tombstone + non-null value
> > > > >     > > >     >     >         >> > * non-tombstone + non-null value
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Then we can discuss once KIP-87a
> > is
> > > > > completed
> > > > >     > > > options
> > > > >     > > >     > later
> > > > >     > > >     >     > and how we
> > > > >     > > >     >     >         >> > support the second part KIP-87b
> to
> > > > > deliver:
> > > > >     > > >     >     >         >> > * non-tombstone + null value
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Cheers
> > > > >     > > >     >     >         >> > Mike
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > ______________________________
> > > > > __________
> > > > >     > > >     >     >         >> > From: Becket Qin <
> > > > becket.qin@gmail.com>
> > > > >     > > >     >     >         >> > Sent: Thursday, November 17,
> 2016
> > > 1:43
> > > > > AM
> > > > >     > > >     >     >         >> > To: dev@kafka.apache.org
> > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 -
> > Add
> > > > > Compaction
> > > > >     > > >     > Tombstone Flag
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Renu, Mayuresh and I had an
> > offline
> > > > >     > discussion,
> > > > >     > > > and
> > > > >     > > >     > following
> > > > >     > > >     >     > is a brief
> > > > >     > > >     >     >         >> > summary.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > 1. We agreed that not bumping up
> > > magic
> > > > > value
> > > > >     > may
> > > > >     > > > result
> > > > >     > > >     > in
> > > > >     > > >     >     > losing zero
> > > > >     > > >     >     >         >> copy
> > > > >     > > >     >     >         >> > during migration.
> > > > >     > > >     >     >         >> > 2. Given that bumping up magic
> > value
> > > > is
> > > > > almost
> > > > >     > > > free and
> > > > >     > > >     > has
> > > > >     > > >     >     > benefit of
> > > > >     > > >     >     >         >> > avoiding potential performance
> > > issue.
> > > > > It is
> > > > >     > > > probably
> > > > >     > > >     > worth
> > > > >     > > >     >     > doing.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > One issue we still need to think
> > > about
> > > > > is
> > > > >     > > whether
> > > > >     > > > we
> > > > >     > > >     > want to
> > > > >     > > >     >     > support a
> > > > >     > > >     >     >         >> > non-tombstone message with null
> > > value.
> > > > >     > > >     >     >         >> > Currently it is not supported by
> > > > Kafka.
> > > > > If we
> > > > >     > > > allow a
> > > > >     > > >     >     > non-tombstone null
> > > > >     > > >     >     >         >> > value message to exist after
> > KIP-87.
> > > > The
> > > > >     > problem
> > > > >     > > > is
> > > > >     > > >     > that such
> > > > >     > > >     >     > message
> > > > >     > > >     >     >         >> will
> > > > >     > > >     >     >         >> > not be supported by the
> consumers
> > > > prior
> > > > > to
> > > > >     > > KIP-87.
> > > > >     > > >     > Because a
> > > > >     > > >     >     > null value
> > > > >     > > >     >     >         >> > will always be interpreted to a
> > > > > tombstone.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > One option is that we keep the
> > > current
> > > > > way,
> > > > >     > i.e.
> > > > >     > > > do not
> > > > >     > > >     >     > support such
> > > > >     > > >     >     >         >> > message. It would be good to
> know
> > if
> > > > > there is
> > > > >     > a
> > > > >     > > >     > concrete use
> > > > >     > > >     >     > case for
> > > > >     > > >     >     >         >> such
> > > > >     > > >     >     >         >> > message. If there is not, we can
> > > > > probably just
> > > > >     > > not
> > > > >     > > >     > support it.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Thanks,
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> > > > > Mayuresh
> > > > >     > > Gharat <
> > > > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> > > > >     > > >     >     >         >> > > wrote:
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > > Hi Ismael,
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > This is something I can think
> of
> > > for
> > > > >     > migration
> > > > >     > > > plan:
> > > > >     > > >     >     >         >> > > So the migration plan can look
> > > > > something
> > > > >     > like
> > > > >     > > > this,
> > > > >     > > >     > with up
> > > > >     > > >     >     >         >> conversion :
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > 1) Currently lets say we have
> > > Broker
> > > > > at
> > > > >     > > version
> > > > >     > > > x.
> > > > >     > > >     >     >         >> > > 2) Currently we have clients
> at
> > > > > version x.
> > > > >     > > >     >     >         >> > > 3) a) We move the version to
> > > > > Broker(x+1) :
> > > > >     > > > supports
> > > > >     > > >     > both
> > > > >     > > >     >     > tombstone and
> > > > >     > > >     >     >         >> > null
> > > > >     > > >     >     >         >> > > for log compaction.
> > > > >     > > >     >     >         >> > >     b) We upgrade the client
> to
> > > > > version
> > > > >     > > > client(x+1) :
> > > > >     > > >     > if in
> > > > >     > > >     >     > the
> > > > >     > > >     >     >         >> producer
> > > > >     > > >     >     >         >> > > client(x+1) the value is set
> to
> > > > null,
> > > > > we
> > > > >     > will
> > > > >     > > >     > automatically
> > > > >     > > >     >     > set the
> > > > >     > > >     >     >         >> > > Tombstone bit internally. If
> the
> > > > > producer
> > > > >     > > > client(x+1)
> > > > >     > > >     > sets
> > > > >     > > >     >     > the
> > > > >     > > >     >     >         >> tombstone
> > > > >     > > >     >     >         >> > > itself, well and good. For
> > > producer
> > > > >     > client(x),
> > > > >     > > > the
> > > > >     > > >     > broker
> > > > >     > > >     >     > will up
> > > > >     > > >     >     >         >> convert
> > > > >     > > >     >     >         >> > > to have the tombstone bit.
> > > > > Broker(x+1) is
> > > > >     > > > supporting
> > > > >     > > >     > both.
> > > > >     > > >     >     > Consumer
> > > > >     > > >     >     >         >> > > client(x+1) will be aware of
> > this
> > > > and
> > > > > should
> > > > >     > > be
> > > > >     > > > able
> > > > >     > > >     > to
> > > > >     > > >     >     > handle this.
> > > > >     > > >     >     >         >> For
> > > > >     > > >     >     >         >> > > consumer client(x) we will
> down
> > > > > convert the
> > > > >     > > > message
> > > > >     > > >     > on the
> > > > >     > > >     >     > broker
> > > > >     > > >     >     >         >> side.
> > > > >     > > >     >     >         >> > >     c) At this point we will
> > have
> > > to
> > > > >     > specify a
> > > > >     > > >     > warning or
> > > > >     > > >     >     > clearly
> > > > >     > > >     >     >         >> specify
> > > > >     > > >     >     >         >> > > in docs that this behavior is
> > > about
> > > > > to be
> > > > >     > > > changed for
> > > > >     > > >     > log
> > > > >     > > >     >     > compaction.
> > > > >     > > >     >     >         >> > > 4) a) In next release of the
> > > > > Broker(x+2), we
> > > > >     > > > say that
> > > > >     > > >     > only
> > > > >     > > >     >     > Tombstone
> > > > >     > > >     >     >         >> is
> > > > >     > > >     >     >         >> > > used for log compaction on the
> > > > Broker
> > > > > side.
> > > > >     > > >     > Clients(x+1)
> > > > >     > > >     >     > still is
> > > > >     > > >     >     >         >> > > supported.
> > > > >     > > >     >     >         >> > >     b) We upgrade the client
> to
> > > > > version
> > > > >     > > > client(x+2) :
> > > > >     > > >     > if
> > > > >     > > >     >     > value is set
> > > > >     > > >     >     >         >> to
> > > > >     > > >     >     >         >> > > null, tombstone will not be
> set
> > > > >     > automatically.
> > > > >     > > > The
> > > > >     > > >     > client
> > > > >     > > >     >     > will have to
> > > > >     > > >     >     >         >> > call
> > > > >     > > >     >     >         >> > > setTombstone() to actually set
> > the
> > > > >     > tombstone.
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > We should compare this
> migration
> > > > plan
> > > > > with
> > > > >     > the
> > > > >     > > >     > migration
> > > > >     > > >     >     > plan for
> > > > >     > > >     >     >         >> magic
> > > > >     > > >     >     >         >> > > byte bump and do whatever
> looks
> > > > good.
> > > > >     > > >     >     >         >> > > I am just worried that if we
> go
> > > down
> > > > > magic
> > > > >     > > byte
> > > > >     > > > route,
> > > > >     > > >     >     > unless I am
> > > > >     > > >     >     >         >> > missing
> > > > >     > > >     >     >         >> > > something, it sounds like
> kafka
> > > will
> > > > > be
> > > > >     > stuck
> > > > >     > > > with
> > > > >     > > >     >     > supporting both
> > > > >     > > >     >     >         >> null
> > > > >     > > >     >     >         >> > > value and tombstone bit for
> log
> > > > > compaction
> > > > >     > for
> > > > >     > > > life
> > > > >     > > >     > long,
> > > > >     > > >     >     > which does
> > > > >     > > >     >     >         >> not
> > > > >     > > >     >     >         >> > > look like a good end state.
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > Thanks,
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > Mayuresh
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32
> AM,
> > > > > Mayuresh
> > > > >     > > > Gharat <
> > > > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> > > > >     > > >     >     >         >> > > > wrote:
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > > Hi Ismael,
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > That's a very good point
> > which I
> > > > > might
> > > > >     > have
> > > > >     > > > not
> > > > >     > > >     >     > considered earlier.
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > Here is a plan that I can
> > think
> > > > of:
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > Stage 1) The broker from now
> > on,
> > > > up
> > > > >     > converts
> > > > >     > > > the
> > > > >     > > >     > message
> > > > >     > > >     >     > to have the
> > > > >     > > >     >     >         >> > > > tombstone marker. The log
> > > > compaction
> > > > >     > thread
> > > > >     > > > does log
> > > > >     > > >     >     > compaction
> > > > >     > > >     >     >         >> based
> > > > >     > > >     >     >         >> > on
> > > > >     > > >     >     >         >> > > > both null and tombstone
> > marker.
> > > > > This is
> > > > >     > our
> > > > >     > > >     > transition
> > > > >     > > >     >     > period.
> > > > >     > > >     >     >         >> > > > Stage 2) The next release we
> > > only
> > > > > say that
> > > > >     > > log
> > > > >     > > >     > compaction
> > > > >     > > >     >     > is based
> > > > >     > > >     >     >         >> on
> > > > >     > > >     >     >         >> > > > tombstone marker. (Open
> source
> > > > > kafka makes
> > > > >     > > > this as a
> > > > >     > > >     >     > policy). By
> > > > >     > > >     >     >         >> this
> > > > >     > > >     >     >         >> > > time,
> > > > >     > > >     >     >         >> > > > the organization which is
> > moving
> > > > to
> > > > > this
> > > > >     > > > release
> > > > >     > > >     > will be
> > > > >     > > >     >     > sure that
> > > > >     > > >     >     >         >> they
> > > > >     > > >     >     >         >> > > > have gone through the entire
> > > > > transition
> > > > >     > > > period.
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > My only goal of doing this
> is
> > > that
> > > > > Kafka
> > > > >     > > > clearly
> > > > >     > > >     >     > specifies the end
> > > > >     > > >     >     >         >> > state
> > > > >     > > >     >     >         >> > > > about what log compaction
> > means
> > > > (is
> > > > > it
> > > > >     > null
> > > > >     > > > value
> > > > >     > > >     > or a
> > > > >     > > >     >     > tombstone
> > > > >     > > >     >     >         >> > marker,
> > > > >     > > >     >     >         >> > > > but not both).
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > What do you think?
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > Thanks,
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > Mayuresh
> > > > >     > > >     >     >         >> > > > .
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17
> > AM,
> > > > > Ismael
> > > > >     > > Juma <
> > > > >     > > >     >     > ismael@juma.me.uk>
> > > > >     > > >     >     >         >> > wrote:
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > >> One comment below.
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at
> 5:08
> > > PM,
> > > > > Mayuresh
> > > > >     > > > Gharat <
> > > > >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> > > > >     > > >     >     >         >> > > >> > wrote:
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >> >    - If we don't bump up
> > the
> > > > > magic
> > > > >     > byte,
> > > > >     > > > on the
> > > > >     > > >     > broker
> > > > >     > > >     >     > side, the
> > > > >     > > >     >     >         >> > > broker
> > > > >     > > >     >     >         >> > > >> >    will always have to
> look
> > > at
> > > > > both
> > > > >     > > > tombstone
> > > > >     > > >     > bit and
> > > > >     > > >     >     > the value
> > > > >     > > >     >     >         >> when
> > > > >     > > >     >     >         >> > > do
> > > > >     > > >     >     >         >> > > >> the
> > > > >     > > >     >     >         >> > > >> >    compaction. Assuming
> we
> > do
> > > > > not bump
> > > > >     > up
> > > > >     > > > the
> > > > >     > > >     > magic
> > > > >     > > >     >     > byte,
> > > > >     > > >     >     >         >> > > >> >    imagine the broker
> sees
> > a
> > > > > message
> > > > >     > > which
> > > > >     > > > does
> > > > >     > > >     > not
> > > > >     > > >     >     > have a
> > > > >     > > >     >     >         >> tombstone
> > > > >     > > >     >     >         >> > > bit
> > > > >     > > >     >     >         >> > > >> >    set. The broker does
> not
> > > > know
> > > > > when
> > > > >     > the
> > > > >     > > >     > message was
> > > > >     > > >     >     > produced
> > > > >     > > >     >     >         >> (i.e.
> > > > >     > > >     >     >         >> > > >> > whether
> > > > >     > > >     >     >         >> > > >> >    the message has been
> up
> > > > > converted or
> > > > >     > > > not), it
> > > > >     > > >     > has
> > > > >     > > >     >     > to take a
> > > > >     > > >     >     >         >> > further
> > > > >     > > >     >     >         >> > > >> > look at
> > > > >     > > >     >     >         >> > > >> >    the value to see if it
> > is
> > > > > null or
> > > > >     > not
> > > > >     > > in
> > > > >     > > >     > order to
> > > > >     > > >     >     > determine
> > > > >     > > >     >     >         >> if it
> > > > >     > > >     >     >         >> > > is
> > > > >     > > >     >     >         >> > > >> a
> > > > >     > > >     >     >         >> > > >> >    tombstone. The same
> > logic
> > > > has
> > > > > to be
> > > > >     > > put
> > > > >     > > > on the
> > > > >     > > >     >     > consumer as
> > > > >     > > >     >     >         >> well
> > > > >     > > >     >     >         >> > > >> because
> > > > >     > > >     >     >         >> > > >> > the
> > > > >     > > >     >     >         >> > > >> >    consumer does not know
> > if
> > > > the
> > > > >     > message
> > > > >     > > > has
> > > > >     > > >     > been up
> > > > >     > > >     >     > converted or
> > > > >     > > >     >     >         >> > not.
> > > > >     > > >     >     >         >> > > >> >       - If we upconvert
> > while
> > > > >     > appending,
> > > > >     > > > this is
> > > > >     > > >     > not
> > > > >     > > >     >     > the case,
> > > > >     > > >     >     >         >> > right?
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >> If I understand you
> > correctly,
> > > > > this is
> > > > >     > not
> > > > >     > > >     > sufficient
> > > > >     > > >     >     > because the
> > > > >     > > >     >     >         >> log
> > > > >     > > >     >     >         >> > > may
> > > > >     > > >     >     >         >> > > >> have messages appended
> before
> > > it
> > > > > was
> > > > >     > > > upgraded to
> > > > >     > > >     > include
> > > > >     > > >     >     > KIP-87.
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >> Ismael
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > --
> > > > >     > > >     >     >         >> > > > -Regards,
> > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> > > > >     > > >     >     >         >> > > > (862) 250-7125
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > --
> > > > >     > > >     >     >         >> > > -Regards,
> > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > > > >     > > >     >     >         >> > > (862) 250-7125
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > The information contained in
> this
> > > > email
> > > > > is
> > > > >     > > > strictly
> > > > >     > > >     >     > confidential and for
> > > > >     > > >     >     >         >> > the use of the addressee only,
> > > unless
> > > > >     > otherwise
> > > > >     > > >     > indicated. If
> > > > >     > > >     >     > you are
> > > > >     > > >     >     >         >> not
> > > > >     > > >     >     >         >> > the intended recipient, please
> do
> > > not
> > > > > read,
> > > > >     > > copy,
> > > > >     > > > use or
> > > > >     > > >     >     > disclose to
> > > > >     > > >     >     >         >> others
> > > > >     > > >     >     >         >> > this message or any attachment.
> > > Please
> > > > > also
> > > > >     > > > notify the
> > > > >     > > >     > sender
> > > > >     > > >     >     > by
> > > > >     > > >     >     >         >> replying
> > > > >     > > >     >     >         >> > to this email or by telephone
> > > (+44(020
> > > > > 7896
> > > > >     > > 0011)
> > > > >     > > > and
> > > > >     > > >     > then
> > > > >     > > >     >     > delete the
> > > > >     > > >     >     >         >> email
> > > > >     > > >     >     >         >> > and any copies of it. Opinions,
> > > > > conclusion
> > > > >     > (etc)
> > > > >     > > > that
> > > > >     > > >     > do not
> > > > >     > > >     >     > relate to
> > > > >     > > >     >     >         >> the
> > > > >     > > >     >     >         >> > official business of this
> company
> > > > shall
> > > > > be
> > > > >     > > > understood as
> > > > >     > > >     >     > neither given
> > > > >     > > >     >     >         >> nor
> > > > >     > > >     >     >         >> > endorsed by it. IG is a trading
> > name
> > > > of
> > > > > IG
> > > > >     > > Markets
> > > > >     > > >     > Limited (a
> > > > >     > > >     >     > company
> > > > >     > > >     >     >         >> > registered in England and Wales,
> > > > company
> > > > >     > number
> > > > >     > > >     > 04008957) and
> > > > >     > > >     >     > IG Index
> > > > >     > > >     >     >         >> > Limited (a company registered in
> > > > > England and
> > > > >     > > > Wales,
> > > > >     > > >     > company
> > > > >     > > >     >     > number
> > > > >     > > >     >     >         >> > 01190902). Registered address at
> > > > Cannon
> > > > > Bridge
> > > > >     > > > House, 25
> > > > >     > > >     >     > Dowgate Hill,
> > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
> > > > Limited
> > > > >     > > (register
> > > > >     > > >     > number
> > > > >     > > >     >     > 195355) and IG
> > > > >     > > >     >     >         >> > Index Limited (register number
> > > 114059)
> > > > > are
> > > > >     > > > authorised
> > > > >     > > >     > and
> > > > >     > > >     >     > regulated by
> > > > >     > > >     >     >         >> the
> > > > >     > > >     >     >         >> > Financial Conduct Authority.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> --
> > > > >     > > >     >     >         >> -Regards,
> > > > >     > > >     >     >         >> Mayuresh R. Gharat
> > > > >     > > >     >     >         >> (862) 250-7125
> > > > >     > > >     >     >         >> The information contained in this
> > > email
> > > > is
> > > > >     > > strictly
> > > > >     > > >     >     > confidential and for
> > > > >     > > >     >     >         >> the use of the addressee only,
> > unless
> > > > > otherwise
> > > > >     > > >     > indicated. If
> > > > >     > > >     >     > you are not
> > > > >     > > >     >     >         >> the intended recipient, please do
> > not
> > > > > read,
> > > > >     > copy,
> > > > >     > > > use or
> > > > >     > > >     >     > disclose to others
> > > > >     > > >     >     >         >> this message or any attachment.
> > Please
> > > > > also
> > > > >     > notify
> > > > >     > > > the
> > > > >     > > >     > sender
> > > > >     > > >     >     > by replying
> > > > >     > > >     >     >         >> to this email or by telephone
> > (+44(020
> > > > > 7896
> > > > >     > 0011)
> > > > >     > > > and then
> > > > >     > > >     >     > delete the email
> > > > >     > > >     >     >         >> and any copies of it. Opinions,
> > > > > conclusion (etc)
> > > > >     > > > that do
> > > > >     > > >     > not
> > > > >     > > >     >     > relate to the
> > > > >     > > >     >     >         >> official business of this company
> > > shall
> > > > be
> > > > >     > > > understood as
> > > > >     > > >     >     > neither given nor
> > > > >     > > >     >     >         >> endorsed by it. IG is a trading
> name
> > > of
> > > > IG
> > > > >     > Markets
> > > > >     > > >     > Limited (a
> > > > >     > > >     >     > company
> > > > >     > > >     >     >         >> registered in England and Wales,
> > > company
> > > > > number
> > > > >     > > > 04008957)
> > > > >     > > >     > and
> > > > >     > > >     >     > IG Index
> > > > >     > > >     >     >         >> Limited (a company registered in
> > > England
> > > > > and
> > > > >     > > Wales,
> > > > >     > > >     > company
> > > > >     > > >     >     > number
> > > > >     > > >     >     >         >> 01190902). Registered address at
> > > Cannon
> > > > > Bridge
> > > > >     > > > House, 25
> > > > >     > > >     >     > Dowgate Hill,
> > > > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
> > > Limited
> > > > >     > (register
> > > > >     > > > number
> > > > >     > > >     >     > 195355) and IG
> > > > >     > > >     >     >         >> Index Limited (register number
> > 114059)
> > > > are
> > > > >     > > > authorised and
> > > > >     > > >     >     > regulated by the
> > > > >     > > >     >     >         >> Financial Conduct Authority.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > --
> > > > >     > > >     >     >         > -Regards,
> > > > >     > > >     >     >         > Mayuresh R. Gharat
> > > > >     > > >     >     >         > (862) 250-7125
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >         --
> > > > >     > > >     >     >         -Regards,
> > > > >     > > >     >     >         Mayuresh R. Gharat
> > > > >     > > >     >     >         (862) 250-7125
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >     The information contained in this email
> is
> > > > > strictly
> > > > >     > > > confidential
> > > > >     > > >     > and
> > > > >     > > >     >     > for the use of the addressee only, unless
> > > otherwise
> > > > >     > > indicated.
> > > > >     > > > If
> > > > >     > > >     > you are
> > > > >     > > >     >     > not the intended recipient, please do not
> read,
> > > > > copy, use
> > > > >     > or
> > > > >     > > >     > disclose to
> > > > >     > > >     >     > others this message or any attachment. Please
> > > also
> > > > > notify
> > > > >     > the
> > > > >     > > > sender
> > > > >     > > >     > by
> > > > >     > > >     >     > replying to this email or by telephone
> (+44(020
> > > > 7896
> > > > > 0011)
> > > > >     > > and
> > > > >     > > > then
> > > > >     > > >     > delete
> > > > >     > > >     >     > the email and any copies of it. Opinions,
> > > > conclusion
> > > > > (etc)
> > > > >     > > > that do
> > > > >     > > >     > not
> > > > >     > > >     >     > relate to the official business of this
> company
> > > > > shall be
> > > > >     > > > understood
> > > > >     > > >     > as
> > > > >     > > >     >     > neither given nor endorsed by it. IG is a
> > trading
> > > > > name of
> > > > >     > IG
> > > > >     > > > Markets
> > > > >     > > >     >     > Limited (a company registered in England and
> > > Wales,
> > > > > company
> > > > >     > > > number
> > > > >     > > >     >     > 04008957) and IG Index Limited (a company
> > > > registered
> > > > > in
> > > > >     > > > England and
> > > > >     > > >     > Wales,
> > > > >     > > >     >     > company number 01190902). Registered address
> at
> > > > > Cannon
> > > > >     > Bridge
> > > > >     > > > House,
> > > > >     > > >     > 25
> > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG
> Markets
> > > > > Limited
> > > > >     > > > (register
> > > > >     > > >     > number
> > > > >     > > >     >     > 195355) and IG Index Limited (register number
> > > > > 114059) are
> > > > >     > > > authorised
> > > > >     > > >     > and
> > > > >     > > >     >     > regulated by the Financial Conduct Authority.
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >
> > > > >     > > >     >
> > > > >     > > >     > The information contained in this email is strictly
> > > > > confidential
> > > > >     > > and
> > > > >     > > > for
> > > > >     > > >     > the use of the addressee only, unless otherwise
> > > > indicated.
> > > > > If you
> > > > >     > > > are not
> > > > >     > > >     > the intended recipient, please do not read, copy,
> use
> > > or
> > > > > disclose
> > > > >     > > to
> > > > >     > > > others
> > > > >     > > >     > this message or any attachment. Please also notify
> > the
> > > > > sender by
> > > > >     > > > replying
> > > > >     > > >     > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and
> > > > then
> > > > > delete
> > > > >     > > > the email
> > > > >     > > >     > and any copies of it. Opinions, conclusion (etc)
> that
> > > do
> > > > > not
> > > > >     > relate
> > > > >     > > > to the
> > > > >     > > >     > official business of this company shall be
> understood
> > > as
> > > > > neither
> > > > >     > > > given nor
> > > > >     > > >     > endorsed by it. IG is a trading name of IG Markets
> > > > Limited
> > > > > (a
> > > > >     > > company
> > > > >     > > >     > registered in England and Wales, company number
> > > 04008957)
> > > > > and IG
> > > > >     > > > Index
> > > > >     > > >     > Limited (a company registered in England and Wales,
> > > > company
> > > > >     > number
> > > > >     > > >     > 01190902). Registered address at Cannon Bridge
> House,
> > > 25
> > > > > Dowgate
> > > > >     > > > Hill,
> > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > > number
> > > > > 195355)
> > > > >     > > > and IG
> > > > >     > > >     > Index Limited (register number 114059) are
> authorised
> > > and
> > > > >     > regulated
> > > > >     > > > by the
> > > > >     > > >     > Financial Conduct Authority.
> > > > >     > > >     >
> > > > >     > > >
> > > > >     > > >
> > > > >     > > > The information contained in this email is strictly
> > > > confidential
> > > > > and
> > > > >     > for
> > > > >     > > > the use of the addressee only, unless otherwise
> indicated.
> > If
> > > > > you are
> > > > >     > not
> > > > >     > > > the intended recipient, please do not read, copy, use or
> > > > > disclose to
> > > > >     > > others
> > > > >     > > > this message or any attachment. Please also notify the
> > sender
> > > > by
> > > > >     > replying
> > > > >     > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > > > > delete the
> > > > >     > > email
> > > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > > > relate to
> > > > >     > > the
> > > > >     > > > official business of this company shall be understood as
> > > > neither
> > > > > given
> > > > >     > > nor
> > > > >     > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > > > company
> > > > >     > > > registered in England and Wales, company number 04008957)
> > and
> > > > IG
> > > > > Index
> > > > >     > > > Limited (a company registered in England and Wales,
> company
> > > > > number
> > > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
> > > > Dowgate
> > > > > Hill,
> > > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > > > 195355) and
> > > > >     > IG
> > > > >     > > > Index Limited (register number 114059) are authorised and
> > > > > regulated by
> > > > >     > > the
> > > > >     > > > Financial Conduct Authority.
> > > > >     > > >
> > > > >     > > The information contained in this email is strictly
> > > confidential
> > > > > and for
> > > > >     > > the use of the addressee only, unless otherwise indicated.
> If
> > > you
> > > > > are not
> > > > >     > > the intended recipient, please do not read, copy, use or
> > > disclose
> > > > > to
> > > > >     > others
> > > > >     > > this message or any attachment. Please also notify the
> sender
> > > by
> > > > > replying
> > > > >     > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > > delete
> > > > > the
> > > > >     > email
> > > > >     > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > > > relate to
> > > > >     > the
> > > > >     > > official business of this company shall be understood as
> > > neither
> > > > > given
> > > > >     > nor
> > > > >     > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > > > company
> > > > >     > > registered in England and Wales, company number 04008957)
> and
> > > IG
> > > > > Index
> > > > >     > > Limited (a company registered in England and Wales, company
> > > > number
> > > > >     > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > > > Hill,
> > > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355)
> > > > > and IG
> > > > >     > > Index Limited (register number 114059) are authorised and
> > > > > regulated by
> > > > >     > the
> > > > >     > > Financial Conduct Authority.
> > > > >     > >
> > > > >     > >
> > > > >     > The information contained in this email is strictly
> > confidential
> > > > and
> > > > > for
> > > > >     > the use of the addressee only, unless otherwise indicated. If
> > you
> > > > > are not
> > > > >     > the intended recipient, please do not read, copy, use or
> > disclose
> > > > to
> > > > > others
> > > > >     > this message or any attachment. Please also notify the sender
> > by
> > > > > replying
> > > > >     > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > delete
> > > > > the email
> > > > >     > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate
> > > > > to the
> > > > >     > official business of this company shall be understood as
> > neither
> > > > > given nor
> > > > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > > company
> > > > >     > registered in England and Wales, company number 04008957) and
> > IG
> > > > > Index
> > > > >     > Limited (a company registered in England and Wales, company
> > > number
> > > > >     > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > > Hill,
> > > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > > and IG
> > > > >     > Index Limited (register number 114059) are authorised and
> > > regulated
> > > > > by the
> > > > >     > Financial Conduct Authority.
> > > > >     >
> > > > >     >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly confidential
> and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If you
> are
> > > not
> > > > > the intended recipient, please do not read, copy, use or disclose
> to
> > > > others
> > > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to
> > > > the
> > > > > official business of this company shall be understood as neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > > registered in England and Wales, company number 04008957) and IG
> > Index
> > > > > Limited (a company registered in England and Wales, company number
> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and regulated
> > by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> and then delete the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Ismael Juma <is...@juma.me.uk>]

Yes, this is actually tricky to do in a way where we both have the desired
semantics and maintain compatibility. When someone creates a
`ProducerRecord` with a `null` value today, the producer doesn't know if
it's meant to be a tombstone or not. For V3 messages, it's easy when the
constructor that takes a tombstone is used. However, if any other
constructor is used, it's not clear. A couple of options I can think of,
none of them particularly nice:

1. Have a third state where tombstone = unknown and the broker would set
the tombstone bit if the value was null and the topic was compacted. People
that wanted to pass a non-null value for the tombstone would have to use
the constructor that takes a tombstone. The drawbacks: third state for
tombstone in message format, message conversion at the broker for a common
case.

2. Extend MetadataResponse to optionally include topic configs, which would
make it possible for the producer to be smarter about setting the
tombstone. It would only do it if a tombstone was not passed explicitly,
the value was null and the topic was compacted. The main drawback is that
the producer would be getting a bit more data for each topic even though it
probably won't use it most of the time. Extending MetadataResponse to
return topic configs would be useful for other reasons as well, so that
part seems OK.

In addition, for both proposals, we could consider adding warnings to the
documentation that the behaviour of the constructors that don't take a
tombstone would change in the next major release so that tombstone = false.
Not sure if this would be worth it though.

Ismael

On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:

> Michael,
>
> It kind of depends on how you want to interpret the tombstone flag. If it's
> purely a producer-facing Kafka-level thing that we treat as internal to the
> broker and log cleaner once the record is sent, then your approach makes
> sense. You're just moving copying the null-indicates-delete behavior of the
> old constructor into the tombstone flag.
>
> However, if you want this change to more generally decouple the idea of
> deletion and null values, then you are sometimes converting what might be a
> completely valid null value that doesn't indicate deletion into a
> tombstone. Downstream applications could potentially handle these cases
> differently given the separation of deletion from value.
>
> I guess the question is if we want to try to support the latter even for
> topics where we have older produce requests. An example where this could
> come up is in something like a CDC Connector. If we try to support the
> semantic difference, a connector might write changes to Kafka using the
> tombstone flag to indicate when a row was truly deleted (vs an update that
> sets it to null but still present; this probably makes more sense for CDC
> from document stores or extracting single columns). There are various
> reasons we might want to maintain the full log and not turn compaction on
> (or just use a time-based retention policy), but downstream applications
> might care to know the difference between a delete and a null value. In
> fact both versions of the same log (compacted and time-retention) could be
> useful and I don't think it'll be uncommon to maintain both or use KIP-71
> to maintain a hybrid compacted/retention topic.
>
> -Ewen
>
> On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jay,
> >
> > Why wouldn't that work, the tombstone value is only looked at by the
> > broker, on a topic configured for compaction as such is benign on non
> > compacted topics. This is as much as sending a null value currently
> >
> >
> > Regards
> > Mike
> >
> >
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jay Kreps <ja...@confluent.io>
> > Sent: Sunday, December 11, 2016 8:58:53 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hey Michael,
> >
> > I'm not quite sure that works as that would translate ALL null values to
> > tombstones, even for non-compacted topics that use null as an acceptable
> > value sent by the producer and expected by the consumer.
> >
> > -Jay
> >
> > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Ewen,
> > >
> > > I think the easiest way to show this is with code.
> > >
> > > As you can see we keep the existing behaviour for code/binaries calling
> > > the pre-existing constructors, whereby if the value is null the
> tombstone
> > > is set to true.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > >
> > >     /**
> > >      * Creates a record with a specified timestamp to be sent to a
> > > specified topic and partition
> > >      *
> > >      * @param topic The topic the record will be appended to
> > >      * @param partition The partition to which the record should be
> sent
> > >      * @param timestamp The timestamp of the record
> > >      * @param tombstone if the record should be treated as a tombstone
> if
> > > the topic is compacted
> > >      * @param key The key that will be included in the record
> > >      * @param value The record contents
> > >      */
> > >     public ProducerRecord(String topic, Integer partition, Boolean
> > > tombstone, Long timestamp, K key, V value) {
> > >         if (topic == null)
> > >             throw new IllegalArgumentException("Topic cannot be
> null.");
> > >         if (timestamp != null && timestamp < 0)
> > >             throw new IllegalArgumentException(
> > >                     String.format("Invalid timestamp: %d. Timestamp
> > should
> > > always be non-negative or null.", timestamp));
> > >         if (partition != null && partition < 0)
> > >             throw new IllegalArgumentException(
> > >                     String.format("Invalid partition: %d. Partition
> > number
> > > should always be non-negative or null.", partition));
> > >         if (!tombstone && value == null){
> > >             throw new IllegalArgumentException(
> > >                     String.format("Tombstone must be true if null
> > value"));
> > >         }
> > >         this.topic = topic;
> > >         this.partition = partition;
> > >         this.tombstone = tombstone;
> > >         this.key = key;
> > >         this.value = value;
> > >         this.timestamp = timestamp;
> > >     }
> > >
> > >     public ProducerRecord(String topic, Integer partition, Long
> > timestamp,
> > > K key, V value) {
> > >         this(topic, partition, value == null, timestamp, key, value);
> > >     }
> > > ________________________________________
> > > From: Ewen Cheslack-Postava <ew...@confluent.io>
> > > Sent: Sunday, December 11, 2016 5:45 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > > It seemed like this was addressed in the migration section, wasn't it?
> > The
> > > V2 vs V3 requests and the fact that the broker will downgrade the
> message
> > > format (losing zero copy) if you issues a V2 request to a broker using
> V3
> > > format handles compatibility, does it not? The existing consumers won't
> > see
> > > the extra metadata in the value, but they will get a null instead and
> > treat
> > > it as a tombstone. Certainly there is a performance impact, but it
> seemed
> > > compatible.
> > >
> > > I'm worried about this though:
> > >
> > >
> > >    - *NOTE *: With the new version of producer client using
> > ProduceRequest
> > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and
> null
> > > value
> > >    should not be allowed as the older version of consumer using
> > > FetchRequest
> > >    V2 will think of this as a tombstone when its actually not.
> > >
> > >
> > > Unless I'm misunderstanding, this ends up breaking binary compatibility
> > for
> > > the Java API. It sounds like this suggests that passing a null value to
> > the
> > > existing ProducerRecord constructors would generate an exception since
> > you
> > > didn't explicitly enable the tombstone (via whatever new constructor is
> > > provided). But that means you can't swap in a newer client jar without
> > > recompiling and get the same behavior if your app deletes keys using
> the
> > > current approach because your app will start throwing exceptions. Maybe
> > > this is a tradeoff we're ok with, but we've tried pretty hard to avoid
> > > breaking compatibility like this so far -- it makes picking up bug
> fixes
> > in
> > > the clients harder for users of frameworks that have to pin to earlier
> > > default versions for compatibility.
> > >
> > > But then later the KIP says:
> > >
> > >
> > >    - The old constructors for ProducerRecord without this parameter
> will
> > be
> > >    kept but updated so that their default behaviour if setting null
> value
> > > will
> > >    be the tombstone will be set to true to keep existing behaviour.
> > >
> > >
> > > So maybe I am misinterpreting something.
> > >
> > > And just a nit re: motivation section, item 6 would be clearer for a
> > union
> > > schema with null (or optional schemas in other formats), e.g. [null,
> > > string], in which case losing the schema truly is losing information
> > > (whereas null is already the only valid value for a pure null schema).
> > >
> > > -Ewen
> > >
> > >
> > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> > > wrote:
> > >
> > > > Hi Jay,
> > > >
> > > > Good point this detail is missing in the KIP write up. Ive added this
> > > now.
> > > >
> > > > Essentially simply just upgrading the clients we do not expect any
> > client
> > > > app code change needed.
> > > >
> > > > Cheers
> > > > Mike
> > > > ________________________________________
> > > > From: Jay Kreps <ja...@confluent.io>
> > > > Sent: Saturday, December 10, 2016 10:51 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >
> > > > Michael,
> > > >
> > > > The compatibility section goes through the migration path, but isn't
> > the
> > > > bigger compatibility issue with existing apps? There are many
> (probably
> > > > thousands) of apps in production that use this feature and send null
> to
> > > > mean delete. It seems like this would break compatibility with them,
> > and
> > > > they would have to be rewritten to right?
> > > >
> > > > -Jay
> > > >
> > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > 4) On v3 we honour the tombstone. As such we expect it to be set
> > > > correctly
> > > > > as per the KIP.
> > > > >
> > > > > 4.1) why would we want to produce an error when its v3? This is the
> > > exact
> > > > > purpose to support non-null tombstone’s
> > > > > 4.2) again here im unclear on the question on the v3, produce
> request
> > > we
> > > > > expect the tombstone flag to be set correctly.
> > > > >
> > > > > 4.4) compaction only occurs on compacted topics, the bit makes no
> > > > > difference and not looked at on un-compacted (time/size based
> > > eviction).
> > > > >
> > > > >
> > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> > > > >
> > > > >     Hi, Michael,
> > > > >
> > > > >     4. Then, I think I misunderstood this point. Could you document
> > the
> > > > >     following points in the wiki?
> > > > >     4.1 If producer V3 sets tombstone, but provides a non-null
> value,
> > > > does
> > > > > the
> > > > >     send() get an error or does the producer automatically set the
> > > value
> > > > to
> > > > >     null?
> > > > >     4.2 If producer V3 doesn't set tombstone, but provides a null
> > > value,
> > > > > does
> > > > >     the send() get an error or does the producer automatically sets
> > the
> > > > >     tombstone?
> > > > >     4.3 Does the broker only expect messages that (a) have no
> > tombstone
> > > > and
> > > > >     non-null value; (b) have tombstone and null value and reject
> the
> > > > > messages
> > > > >     with an error code otherwise?
> > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted
> on
> > > > not?
> > > > >
> > > > >     Thanks,
> > > > >
> > > > >     Jun
> > > > >
> > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > >     wrote:
> > > > >
> > > > >     > Not at all.  This only acts on compacted topics just as what
> > > occurs
> > > > > today
> > > > >     >
> > > > >     > Sent using OWA for iPhone
> > > > >     > ________________________________________
> > > > >     > From: Jun Rao <ju...@confluent.io>
> > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > > > >     > To: dev@kafka.apache.org
> > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > > >     >
> > > > >     > Hi, Michael,
> > > > >     >
> > > > >     > 4. Hmm, does that mean the new client library can never send
> a
> > > null
> > > > > message
> > > > >     > even to a regular topic? This seems like a change of the
> > existing
> > > > > behavior.
> > > > >     >
> > > > >     > Thanks,
> > > > >     >
> > > > >     > Jun
> > > > >     >
> > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > >     > wrote:
> > > > >     >
> > > > >     > > Hi Jun,
> > > > >     > >
> > > > >     > > Re 4) That's because we expect the tombstone value to be
> set
> > > > > correctly if
> > > > >     > > message bit is 2, as such if an older client sends in on
> old
> > > > > message the
> > > > >     > > message is upcast and the bit is set correctly. And such no
> > > > longer
> > > > > need
> > > > >     > to
> > > > >     > > check the value. Mayuresh can you confirm my thinking and
> > > > > understanding
> > > > >     > of
> > > > >     > > what we've discussed?
> > > > >     > >
> > > > >     > > The second point I understand what you're getting at now my
> > > > > apologies.
> > > > >     > Yes
> > > > >     > > this makes sense to save on touching the message, if we're
> > the
> > > > > only kip
> > > > >     > > going in, in this release.
> > > > >     > >
> > > > >     > > Cheers
> > > > >     > > Mike
> > > > >     > >
> > > > >     > > Sent using OWA for iPhone
> > > > >     > > ________________________________________
> > > > >     > > From: Jun Rao <ju...@confluent.io>
> > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > > > >     > > To: dev@kafka.apache.org
> > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > > >     > >
> > > > >     > > Hi, Michael,
> > > > >     > >
> > > > >     > > 4. Is this updated in the wiki? The text "If the magic byte
> > on
> > > > > message is
> > > > >     > > 2, the broker should use the tombstone bit for log
> > compaction."
> > > > > doesn't
> > > > >     > > seem to have changed.
> > > > >     > >
> > > > >     > > 2. My point is that if we change the message format just
> for
> > > this
> > > > > KIP, we
> > > > >     > > should consider whether it's worth optimizing the down
> > > conversion
> > > > > path
> > > > >     > > (i.e., decide whether a conversion is needed by just
> looking
> > at
> > > > the
> > > > >     > > tombstone bit in the wrapper message) since tombstone will
> be
> > > > used
> > > > >     > rarely.
> > > > >     > > However, if the message format change here is combined with
> > > other
> > > > > KIPs,
> > > > >     > > then this optimization likely won't be needed. The latter
> > > > probably
> > > > > makes
> > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> > > > >     > >
> > > > >     > > Other than those, +1 from me,
> > > > >     > >
> > > > >     > > Thanks,
> > > > >     > >
> > > > >     > > Jun
> > > > >     > >
> > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > >     > > wrote:
> > > > >     > >
> > > > >     > > > Hi Jun
> > > > >     > > >
> > > > >     > > > do we have your vote on this now?
> > > > >     > > >
> > > > >     > > > Any other concerns?
> > > > >     > > >
> > > > >     > > > Cheers
> > > > >     > > > Mike
> > > > >     > > >
> > > > >     > > > Sent using OWA for iPhone
> > > > >     > > > ________________________________________
> > > > >     > > > From: Michael Pearce <Mi...@ig.com>
> > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > > > >     > > > To: dev@kafka.apache.org
> > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> Flag
> > > > >     > > >
> > > > >     > > > Hi Jun,
> > > > >     > > >
> > > > >     > > > Have updated. Thanks again for the feedback.
> > > > >     > > >
> > > > >     > > > Agree yes we should align up when it gets to that, I
> assume
> > > > > you’ve
> > > > >     > > flagged
> > > > >     > > > the same in the other KIP?
> > > > >     > > >
> > > > >     > > > I think for now let’s get this KIP approved, then we can
> > > start
> > > > > the work
> > > > >     > > to
> > > > >     > > > get an initial PR, then we can discuss how to align the
> two
> > > if
> > > > > needed.
> > > > >     > > >
> > > > >     > > > Cheers,
> > > > >     > > > Mike
> > > > >     > > >
> > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io>
> wrote:
> > > > >     > > >
> > > > >     > > >     Hi, Michael,
> > > > >     > > >
> > > > >     > > >     For 2), this is fine. Could you update the KIP wiki
> to
> > > make
> > > > > this
> > > > >     > and
> > > > >     > > > other
> > > > >     > > >     points clearer? Other than that, the KIP looks good
> to
> > > me.
> > > > >     > > >
> > > > >     > > >     An orthogonal thing is that there are other KIPs such
> > as
> > > > > KIP-98
> > > > >     > that
> > > > >     > > > also
> > > > >     > > >     intend to change the message format. If they all get
> > > > > approved, we
> > > > >     > > > should
> > > > >     > > >     think about whether it's better to just bump up the
> > magic
> > > > > byte once
> > > > >     > > to
> > > > >     > > >     incorporate multiple format changes like we did in
> > > > > KIP-31/KIP-32.
> > > > >     > > >
> > > > >     > > >     Thanks,
> > > > >     > > >
> > > > >     > > >     Jun
> > > > >     > > >
> > > > >     > > >
> > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > > > >     > > Michael.Pearce@ig.com>
> > > > >     > > >     wrote:
> > > > >     > > >
> > > > >     > > >     > Hi Jao
> > > > >     > > >     >
> > > > >     > > >     > Thanks for the response. Sorry for slow reply, both
> > > with
> > > > > personal
> > > > >     > > > sickness
> > > > >     > > >     > and also battling some critical issues encountered
> > > since
> > > > >     > upgrading
> > > > >     > > to
> > > > >     > > >     > 0.10.1.0
> > > > >     > > >     >
> > > > >     > > >     > 1) Thans for spotting, Document error where we
> > branched
> > > > > this KIP
> > > > >     > > from
> > > > >     > > >     > KIP-82, will get that removed.
> > > > >     > > >     > 2) Intent is to do this just at the record message
> > > level.
> > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> > corrected.
> > > > >     > > >     > 4) As per discussion thread we will support
> > tombstone +
> > > > > null
> > > > >     > value,
> > > > >     > > >     > tombstone + non null value, no tombstone + null
> > value.
> > > > >     > > >     > 5) I believe this was in the discussion thread,
> > > @Mayuresh
> > > > > is this
> > > > >     > > >     > something we’ve overlooked? I thought we would down
> > > > > convert and
> > > > >     > > > remove the
> > > > >     > > >     > value so the old consumer had existing behavior, or
> > is
> > > > > there
> > > > >     > > > something we
> > > > >     > > >     > haven’t thought about?
> > > > >     > > >     >
> > > > >     > > >     > Cheers
> > > > >     > > >     > Mike
> > > > >     > > >     >
> > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
> > > > wrote:
> > > > >     > > >     >
> > > > >     > > >     >     Hi, Michael,
> > > > >     > > >     >
> > > > >     > > >     >     Thanks for the KIP. A few comments below.
> > > > >     > > >     >
> > > > >     > > >     >     1. The message format change contains
> > > "HeadersLength
> > > > >     > Headers".
> > > > >     > > > Is that
> > > > >     > > >     >     intended?
> > > > >     > > >     >
> > > > >     > > >     >     2. For compressed messageset, is the tombstone
> > bit
> > > > > only set
> > > > >     > at
> > > > >     > > > the
> > > > >     > > >     > shallow
> > > > >     > > >     >     level? Do we always leave that bit in the
> wrapper
> > > > > message
> > > > >     > > unset?
> > > > >     > > > An
> > > > >     > > >     >     alternative is to set the tombstone bit in the
> > > > wrapper
> > > > > if at
> > > > >     > > > least one
> > > > >     > > >     >     inner message has the tombstone bit set. This
> > makes
> > > > > things a
> > > > >     > > bit
> > > > >     > > > more
> > > > >     > > >     >     complicated, but we could potentially exploit
> > that
> > > > for
> > > > >     > > > optimizing down
> > > > >     > > >     >     conversion. For example, we only need to
> convert
> > > > > messages
> > > > >     > with
> > > > >     > > > magic 2
> > > > >     > > >     > to
> > > > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > > > > (conversion is
> > > > >     > > > always
> > > > >     > > >     > needed
> > > > >     > > >     >     from magic 2 to magic 0). Not sure if the
> > > > optimization
> > > > > is
> > > > >     > worth
> > > > >     > > > the
> > > > >     > > >     >     complexity though.
> > > > >     > > >     >
> > > > >     > > >     >     3. The referencing of the new version of
> > > > >     > > > ProducerRequest/FetchRequest
> > > > >     > > >     > is
> > > > >     > > >     >     inconsistent (v4 vs v3). Since our convention
> > > starts
> > > > at
> > > > >     > version
> > > > >     > > > at 0, I
> > > > >     > > >     >     think the new version would be 3.
> > > > >     > > >     >
> > > > >     > > >     >     4. "If the magic byte on message is 2, the
> broker
> > > > > should use
> > > > >     > > the
> > > > >     > > >     > tombstone
> > > > >     > > >     >     bit for log compaction." What about null value?
> > My
> > > > >     > > understanding
> > > > >     > > > is
> > > > >     > > >     > that
> > > > >     > > >     >     null value will be treated the same as setting
> > the
> > > > > tombstone
> > > > >     > > bit.
> > > > >     > > >     >
> > > > >     > > >     >     5. For the migration path, it would be useful
> to
> > > > > describe the
> > > > >     > > > down
> > > > >     > > >     >     conversion path to consumers (i.e., brokers on
> > > > message
> > > > > format
> > > > >     > > > 0.10.2
> > > > >     > > >     > and
> > > > >     > > >     >     consumers on older version).
> > > > >     > > >     >
> > > > >     > > >     >     Thanks,
> > > > >     > > >     >
> > > > >     > > >     >     Jun
> > > > >     > > >     >
> > > > >     > > >     >
> > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael
> Pearce <
> > > > >     > > > Michael.Pearce@ig.com
> > > > >     > > >     > >
> > > > >     > > >     >     wrote:
> > > > >     > > >     >
> > > > >     > > >     >     > Hi All,
> > > > >     > > >     >     >
> > > > >     > > >     >     > We have been discussing in the below thread
> and
> > > > final
> > > > >     > changes
> > > > >     > > > have
> > > > >     > > >     > been
> > > > >     > > >     >     > made to the KIP wiki based on these
> > discussions.
> > > > >     > > >     >     >
> > > > >     > > >     >     > We would now like to put to the vote the
> > > following
> > > > > KIP:
> > > > >     > > >     >     > https://cwiki.apache.org/
> > > > > confluence/display/KAFKA/KIP-
> > > > >     > 87+-+
> > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > > > >     > > >     >     >
> > > > >     > > >     >     > This kip is for having a distinct compaction
> > > > > attribute
> > > > >     > > > “tombstone”
> > > > >     > > >     > flag
> > > > >     > > >     >     > instead of relying on null value, allowing
> > > non-null
> > > > > value
> > > > >     > > > delete
> > > > >     > > >     > messages.
> > > > >     > > >     >     >
> > > > >     > > >     >     > Many thanks,
> > > > >     > > >     >     > Michael
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> > > > >     > > Michael.Pearce@ig.com>
> > > > >     > > >     > wrote:
> > > > >     > > >     >     >
> > > > >     > > >     >     >     Hi Mayuresh,
> > > > >     > > >     >     >
> > > > >     > > >     >     >     LGTM. Ive just made one small adjustment
> > > > > updating the
> > > > >     > > wire
> > > > >     > > >     > protocol to
> > > > >     > > >     >     > show the magic byte bump.
> > > > >     > > >     >     >
> > > > >     > > >     >     >     Do we think we’re good to put to a vote?
> Is
> > > > > there any
> > > > >     > > > other bits
> > > > >     > > >     >     > needing discussion?
> > > > >     > > >     >     >
> > > > >     > > >     >     >     Cheers
> > > > >     > > >     >     >     Mike
> > > > >     > > >     >     >
> > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> > > > >     > > >     > gharatmayuresh15@gmail.com>
> > > > >     > > >     >     > wrote:
> > > > >     > > >     >     >
> > > > >     > > >     >     >         Hi Michael,
> > > > >     > > >     >     >
> > > > >     > > >     >     >         I have updated the migration section
> of
> > > the
> > > > > KIP.
> > > > >     > Can
> > > > >     > > > you
> > > > >     > > >     > please
> > > > >     > > >     >     > take a look?
> > > > >     > > >     >     >
> > > > >     > > >     >     >         Thanks,
> > > > >     > > >     >     >
> > > > >     > > >     >     >         Mayuresh
> > > > >     > > >     >     >
> > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM,
> > Mayuresh
> > > > > Gharat <
> > > > >     > > >     >     > gharatmayuresh15@gmail.com
> > > > >     > > >     >     >         > wrote:
> > > > >     > > >     >     >
> > > > >     > > >     >     >         > Hi Michael,
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > That whilst sending tombstone and
> non
> > > > null
> > > > > value,
> > > > >     > > the
> > > > >     > > >     > consumer
> > > > >     > > >     >     > can expect
> > > > >     > > >     >     >         > only to receive the non-null
> message
> > > only
> > > > > in step
> > > > >     > > > (3) is
> > > > >     > > >     > this
> > > > >     > > >     >     > correct?
> > > > >     > > >     >     >         > ---> I do agree with you here.
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > Becket, Ismael : can you guys
> review
> > > the
> > > > >     > migration
> > > > >     > > > plan
> > > > >     > > >     > listed
> > > > >     > > >     >     > above using
> > > > >     > > >     >     >         > magic byte?
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > Thanks,
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > Mayuresh
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
> > > Michael
> > > > > Pearce <
> > > > >     > > >     >     > Michael.Pearce@ig.com>
> > > > >     > > >     >     >         > wrote:
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         >> Many thanks for this Mayuresh. I
> > don't
> > > > > have any
> > > > >     > > >     > objections.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> I assume we should state:
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> That whilst sending tombstone and
> > non
> > > > null
> > > > >     > value,
> > > > >     > > > the
> > > > >     > > >     > consumer
> > > > >     > > >     >     > can expect
> > > > >     > > >     >     >         >> only to receive the non-null
> message
> > > > only
> > > > > in
> > > > >     > step
> > > > >     > > > (3) is
> > > > >     > > >     > this
> > > > >     > > >     >     > correct?
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Cheers
> > > > >     > > >     >     >         >> Mike
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Sent using OWA for iPhone
> > > > >     > > >     >     >         >> ______________________________
> > > > __________
> > > > >     > > >     >     >         >> From: Mayuresh Gharat <
> > > > >     > gharatmayuresh15@gmail.com
> > > > >     > > >
> > > > >     > > >     >     >         >> Sent: Thursday, November 17, 2016
> > > > 5:18:41
> > > > > PM
> > > > >     > > >     >     >         >> To: dev@kafka.apache.org
> > > > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 -
> Add
> > > > > Compaction
> > > > >     > > > Tombstone
> > > > >     > > >     > Flag
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Hi Ismael,
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Thanks for the explanation.
> > > > >     > > >     >     >         >> Specially I like this part where
> in
> > > you
> > > > >     > mentioned
> > > > >     > > > we can
> > > > >     > > >     > get
> > > > >     > > >     >     > rid of the
> > > > >     > > >     >     >         >> older null value support for log
> > > > > compaction
> > > > >     > later
> > > > >     > > > on,
> > > > >     > > >     > here :
> > > > >     > > >     >     >         >> We can't change semantics of the
> > > message
> > > > > format
> > > > >     > > > without
> > > > >     > > >     > having
> > > > >     > > >     >     > a long
> > > > >     > > >     >     >         >> transition period. And we can't
> rely
> > > > >     > > >     >     >         >> on people reading documentation or
> > > > acting
> > > > > on a
> > > > >     > > > warning for
> > > > >     > > >     >     > something so
> > > > >     > > >     >     >         >> fundamental. As such, my take is
> > that
> > > we
> > > > > need to
> > > > >     > > > bump the
> > > > >     > > >     > magic
> > > > >     > > >     >     > byte. The
> > > > >     > > >     >     >         >> good news is
> > > > >     > > >     >     >         >> that we don't have to support all
> > > > versions
> > > > >     > > forever.
> > > > >     > > > We
> > > > >     > > >     > have
> > > > >     > > >     >     > said that we
> > > > >     > > >     >     >         >> will support direct upgrades for 2
> > > > years.
> > > > > That
> > > > >     > > > means that
> > > > >     > > >     >     > message format
> > > > >     > > >     >     >         >> version n could, in theory, be
> > > removed 2
> > > > > years
> > > > >     > > > after the
> > > > >     > > >     > it's
> > > > >     > > >     >     > introduced.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Just a heads up, I would like to
> > > mention
> > > > > that
> > > > >     > even
> > > > >     > > > without
> > > > >     > > >     >     > bumping magic
> > > > >     > > >     >     >         >> byte, we will *NOT* loose zero
> copy
> > as
> > > > in
> > > > > the
> > > > >     > > > client(x+1)
> > > > >     > > >     > in my
> > > > >     > > >     >     >         >> explanation
> > > > >     > > >     >     >         >> above will convert internally a
> null
> > > > > value to
> > > > >     > > have a
> > > > >     > > >     > tombstone
> > > > >     > > >     >     > bit set and
> > > > >     > > >     >     >         >> a tombstone bit set to have a null
> > > value
> > > > >     > > > automatically
> > > > >     > > >     >     > internally and by
> > > > >     > > >     >     >         >> the time we move to version (x+2),
> > the
> > > > > clients
> > > > >     > > > would have
> > > > >     > > >     >     > upgraded.
> > > > >     > > >     >     >         >> Obviously if we support a request
> > from
> > > > >     > > consumer(x),
> > > > >     > > > we
> > > > >     > > >     > will
> > > > >     > > >     >     > loose zero
> > > > >     > > >     >     >         >> copy
> > > > >     > > >     >     >         >> but that is the same case with
> magic
> > > > byte.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> But if magic byte bump makes life
> > > easier
> > > > > for
> > > > >     > > > transition
> > > > >     > > >     > for the
> > > > >     > > >     >     > above
> > > > >     > > >     >     >         >> reasons that you explained, I am
> OK
> > > with
> > > > > it
> > > > >     > since
> > > > >     > > > we are
> > > > >     > > >     > going
> > > > >     > > >     >     > to meet the
> > > > >     > > >     >     >         >> end goal down the road :)
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> On a side note can we update the
> doc
> > > > here
> > > > > on
> > > > >     > magic
> > > > >     > > > byte
> > > > >     > > >     > to say
> > > > >     > > >     >     > that "*it
> > > > >     > > >     >     >         >> should be bumped whenever the
> > message
> > > > > format is
> > > > >     > > > changed
> > > > >     > > >     > or the
> > > > >     > > >     >     >         >> interpretation of message format
> > > (usage
> > > > > of the
> > > > >     > > > reserved
> > > > >     > > >     > bits as
> > > > >     > > >     >     > well) is
> > > > >     > > >     >     >         >> changed*".
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Hi Michael,
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Here is the update plan that we
> > > > discussed
> > > > >     > offline
> > > > >     > > >     > yesterday :
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Currently the magic-byte which
> > > > > corresponds to
> > > > >     > the
> > > > >     > > >     >     > "message.format.version"
> > > > >     > > >     >     >         >> is set to 1.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> 1) On broker it will be set to 1
> > > > > initially.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> 2) When a producer client sends a
> > > > message
> > > > > with
> > > > >     > > > magic-byte
> > > > >     > > >     > = 2,
> > > > >     > > >     >     > since the
> > > > >     > > >     >     >         >> broker is on magic-byte = 1, we
> will
> > > > down
> > > > >     > convert
> > > > >     > > > it,
> > > > >     > > >     > which
> > > > >     > > >     >     > means if the
> > > > >     > > >     >     >         >> tombstone bit is set, the value
> will
> > > be
> > > > > set to
> > > > >     > > > null. A
> > > > >     > > >     > consumer
> > > > >     > > >     >     >         >> understanding magic-byte = 1, will
> > > still
> > > > > work
> > > > >     > with
> > > > >     > > > this. A
> > > > >     > > >     >     > consumer
> > > > >     > > >     >     >         >> working
> > > > >     > > >     >     >         >> with magic-byte =2 will also be
> able
> > > to
> > > > >     > understand
> > > > >     > > > this,
> > > > >     > > >     > since
> > > > >     > > >     >     > it
> > > > >     > > >     >     >         >> understands the tombstone.
> > > > >     > > >     >     >         >> Now there is still the question of
> > > > > supporting a
> > > > >     > > >     > non-tombstone
> > > > >     > > >     >     > and null
> > > > >     > > >     >     >         >> value from producer client with
> > > > > magic-byte = 2.*
> > > > >     > > (I
> > > > >     > > > am
> > > > >     > > >     > not sure
> > > > >     > > >     >     > if we
> > > > >     > > >     >     >         >> should support this. Ismael/Becket
> > can
> > > > > comment
> > > > >     > > > here)*
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> 3) When almost all the clients
> have
> > > > > upgraded,
> > > > >     > the
> > > > >     > > >     >     > message.format.version
> > > > >     > > >     >     >         >> on
> > > > >     > > >     >     >         >> the broker can be changed to 2,
> > where
> > > in
> > > > > the
> > > > >     > down
> > > > >     > > >     > conversion in
> > > > >     > > >     >     > the above
> > > > >     > > >     >     >         >> step will not happen. If at this
> > point
> > > > we
> > > > > get a
> > > > >     > > > consumer
> > > > >     > > >     >     > request from a
> > > > >     > > >     >     >         >> older consumer, we might have to
> > down
> > > > > convert
> > > > >     > > where
> > > > >     > > > in we
> > > > >     > > >     > loose
> > > > >     > > >     >     > zero copy,
> > > > >     > > >     >     >         >> but these cases should be rare.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Becket can you review this plan
> and
> > > add
> > > > > more
> > > > >     > > > details if I
> > > > >     > > >     > have
> > > > >     > > >     >     >         >> missed/wronged something, before
> we
> > > put
> > > > > it on
> > > > >     > KIP.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Thanks,
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> Mayuresh
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
> > > > Michael
> > > > >     > Pearce <
> > > > >     > > >     >     > Michael.Pearce@ig.com>
> > > > >     > > >     >     >         >> wrote:
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> > Thanks guys, for discussing this
> > > > > offline and
> > > > >     > > > getting
> > > > >     > > >     > some
> > > > >     > > >     >     > consensus.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > So its clear for myself and
> others
> > > > what
> > > > > is
> > > > >     > > > proposed now
> > > > >     > > >     > (i
> > > > >     > > >     >     > think i
> > > > >     > > >     >     >         >> > understand, but want to make
> sure)
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Could i ask either directly
> update
> > > the
> > > > > kip to
> > > > >     > > > detail the
> > > > >     > > >     >     > migration
> > > > >     > > >     >     >         >> > strategy, or (re-)state your
> > offline
> > > > > discussed
> > > > >     > > and
> > > > >     > > >     > agreed
> > > > >     > > >     >     > migration
> > > > >     > > >     >     >         >> > strategy based on a magic byte
> is
> > in
> > > > > this
> > > > >     > > thread.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > The main original driver for the
> > KIP
> > > > > was to
> > > > >     > > > support
> > > > >     > > >     >     > compaction where
> > > > >     > > >     >     >         >> value
> > > > >     > > >     >     >         >> > isn't null, based off the
> > > discussions
> > > > on
> > > > >     > KIP-82
> > > > >     > > > thread.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > We should be able to support
> > > > > non-tombstone +
> > > > >     > > null
> > > > >     > > > value
> > > > >     > > >     > by the
> > > > >     > > >     >     >         >> completion
> > > > >     > > >     >     >         >> > of the KIP, as we noted when
> > > > discussing
> > > > > this
> > > > >     > > kip,
> > > > >     > > > having
> > > > >     > > >     >     > logic based on
> > > > >     > > >     >     >         >> a
> > > > >     > > >     >     >         >> > null value isn't very clean and
> > also
> > > > > separates
> > > > >     > > the
> > > > >     > > >     > concerns.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > As discussed already though we
> can
> > > > > split this
> > > > >     > > into
> > > > >     > > >     > KIP-87a
> > > > >     > > >     >     > and KIP-87b
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Where we look to deliver KIP-87a
> > on
> > > a
> > > > >     > compacted
> > > > >     > > > topic
> > > > >     > > >     > (to
> > > > >     > > >     >     > address the
> > > > >     > > >     >     >         >> > immediate issues)
> > > > >     > > >     >     >         >> > * tombstone + null value
> > > > >     > > >     >     >         >> > * tombstone + non-null value
> > > > >     > > >     >     >         >> > * non-tombstone + non-null value
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Then we can discuss once KIP-87a
> > is
> > > > > completed
> > > > >     > > > options
> > > > >     > > >     > later
> > > > >     > > >     >     > and how we
> > > > >     > > >     >     >         >> > support the second part KIP-87b
> to
> > > > > deliver:
> > > > >     > > >     >     >         >> > * non-tombstone + null value
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Cheers
> > > > >     > > >     >     >         >> > Mike
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > ______________________________
> > > > > __________
> > > > >     > > >     >     >         >> > From: Becket Qin <
> > > > becket.qin@gmail.com>
> > > > >     > > >     >     >         >> > Sent: Thursday, November 17,
> 2016
> > > 1:43
> > > > > AM
> > > > >     > > >     >     >         >> > To: dev@kafka.apache.org
> > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 -
> > Add
> > > > > Compaction
> > > > >     > > >     > Tombstone Flag
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Renu, Mayuresh and I had an
> > offline
> > > > >     > discussion,
> > > > >     > > > and
> > > > >     > > >     > following
> > > > >     > > >     >     > is a brief
> > > > >     > > >     >     >         >> > summary.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > 1. We agreed that not bumping up
> > > magic
> > > > > value
> > > > >     > may
> > > > >     > > > result
> > > > >     > > >     > in
> > > > >     > > >     >     > losing zero
> > > > >     > > >     >     >         >> copy
> > > > >     > > >     >     >         >> > during migration.
> > > > >     > > >     >     >         >> > 2. Given that bumping up magic
> > value
> > > > is
> > > > > almost
> > > > >     > > > free and
> > > > >     > > >     > has
> > > > >     > > >     >     > benefit of
> > > > >     > > >     >     >         >> > avoiding potential performance
> > > issue.
> > > > > It is
> > > > >     > > > probably
> > > > >     > > >     > worth
> > > > >     > > >     >     > doing.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > One issue we still need to think
> > > about
> > > > > is
> > > > >     > > whether
> > > > >     > > > we
> > > > >     > > >     > want to
> > > > >     > > >     >     > support a
> > > > >     > > >     >     >         >> > non-tombstone message with null
> > > value.
> > > > >     > > >     >     >         >> > Currently it is not supported by
> > > > Kafka.
> > > > > If we
> > > > >     > > > allow a
> > > > >     > > >     >     > non-tombstone null
> > > > >     > > >     >     >         >> > value message to exist after
> > KIP-87.
> > > > The
> > > > >     > problem
> > > > >     > > > is
> > > > >     > > >     > that such
> > > > >     > > >     >     > message
> > > > >     > > >     >     >         >> will
> > > > >     > > >     >     >         >> > not be supported by the
> consumers
> > > > prior
> > > > > to
> > > > >     > > KIP-87.
> > > > >     > > >     > Because a
> > > > >     > > >     >     > null value
> > > > >     > > >     >     >         >> > will always be interpreted to a
> > > > > tombstone.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > One option is that we keep the
> > > current
> > > > > way,
> > > > >     > i.e.
> > > > >     > > > do not
> > > > >     > > >     >     > support such
> > > > >     > > >     >     >         >> > message. It would be good to
> know
> > if
> > > > > there is
> > > > >     > a
> > > > >     > > >     > concrete use
> > > > >     > > >     >     > case for
> > > > >     > > >     >     >         >> such
> > > > >     > > >     >     >         >> > message. If there is not, we can
> > > > > probably just
> > > > >     > > not
> > > > >     > > >     > support it.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > Thanks,
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> > > > > Mayuresh
> > > > >     > > Gharat <
> > > > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> > > > >     > > >     >     >         >> > > wrote:
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >> > > Hi Ismael,
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > This is something I can think
> of
> > > for
> > > > >     > migration
> > > > >     > > > plan:
> > > > >     > > >     >     >         >> > > So the migration plan can look
> > > > > something
> > > > >     > like
> > > > >     > > > this,
> > > > >     > > >     > with up
> > > > >     > > >     >     >         >> conversion :
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > 1) Currently lets say we have
> > > Broker
> > > > > at
> > > > >     > > version
> > > > >     > > > x.
> > > > >     > > >     >     >         >> > > 2) Currently we have clients
> at
> > > > > version x.
> > > > >     > > >     >     >         >> > > 3) a) We move the version to
> > > > > Broker(x+1) :
> > > > >     > > > supports
> > > > >     > > >     > both
> > > > >     > > >     >     > tombstone and
> > > > >     > > >     >     >         >> > null
> > > > >     > > >     >     >         >> > > for log compaction.
> > > > >     > > >     >     >         >> > >     b) We upgrade the client
> to
> > > > > version
> > > > >     > > > client(x+1) :
> > > > >     > > >     > if in
> > > > >     > > >     >     > the
> > > > >     > > >     >     >         >> producer
> > > > >     > > >     >     >         >> > > client(x+1) the value is set
> to
> > > > null,
> > > > > we
> > > > >     > will
> > > > >     > > >     > automatically
> > > > >     > > >     >     > set the
> > > > >     > > >     >     >         >> > > Tombstone bit internally. If
> the
> > > > > producer
> > > > >     > > > client(x+1)
> > > > >     > > >     > sets
> > > > >     > > >     >     > the
> > > > >     > > >     >     >         >> tombstone
> > > > >     > > >     >     >         >> > > itself, well and good. For
> > > producer
> > > > >     > client(x),
> > > > >     > > > the
> > > > >     > > >     > broker
> > > > >     > > >     >     > will up
> > > > >     > > >     >     >         >> convert
> > > > >     > > >     >     >         >> > > to have the tombstone bit.
> > > > > Broker(x+1) is
> > > > >     > > > supporting
> > > > >     > > >     > both.
> > > > >     > > >     >     > Consumer
> > > > >     > > >     >     >         >> > > client(x+1) will be aware of
> > this
> > > > and
> > > > > should
> > > > >     > > be
> > > > >     > > > able
> > > > >     > > >     > to
> > > > >     > > >     >     > handle this.
> > > > >     > > >     >     >         >> For
> > > > >     > > >     >     >         >> > > consumer client(x) we will
> down
> > > > > convert the
> > > > >     > > > message
> > > > >     > > >     > on the
> > > > >     > > >     >     > broker
> > > > >     > > >     >     >         >> side.
> > > > >     > > >     >     >         >> > >     c) At this point we will
> > have
> > > to
> > > > >     > specify a
> > > > >     > > >     > warning or
> > > > >     > > >     >     > clearly
> > > > >     > > >     >     >         >> specify
> > > > >     > > >     >     >         >> > > in docs that this behavior is
> > > about
> > > > > to be
> > > > >     > > > changed for
> > > > >     > > >     > log
> > > > >     > > >     >     > compaction.
> > > > >     > > >     >     >         >> > > 4) a) In next release of the
> > > > > Broker(x+2), we
> > > > >     > > > say that
> > > > >     > > >     > only
> > > > >     > > >     >     > Tombstone
> > > > >     > > >     >     >         >> is
> > > > >     > > >     >     >         >> > > used for log compaction on the
> > > > Broker
> > > > > side.
> > > > >     > > >     > Clients(x+1)
> > > > >     > > >     >     > still is
> > > > >     > > >     >     >         >> > > supported.
> > > > >     > > >     >     >         >> > >     b) We upgrade the client
> to
> > > > > version
> > > > >     > > > client(x+2) :
> > > > >     > > >     > if
> > > > >     > > >     >     > value is set
> > > > >     > > >     >     >         >> to
> > > > >     > > >     >     >         >> > > null, tombstone will not be
> set
> > > > >     > automatically.
> > > > >     > > > The
> > > > >     > > >     > client
> > > > >     > > >     >     > will have to
> > > > >     > > >     >     >         >> > call
> > > > >     > > >     >     >         >> > > setTombstone() to actually set
> > the
> > > > >     > tombstone.
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > We should compare this
> migration
> > > > plan
> > > > > with
> > > > >     > the
> > > > >     > > >     > migration
> > > > >     > > >     >     > plan for
> > > > >     > > >     >     >         >> magic
> > > > >     > > >     >     >         >> > > byte bump and do whatever
> looks
> > > > good.
> > > > >     > > >     >     >         >> > > I am just worried that if we
> go
> > > down
> > > > > magic
> > > > >     > > byte
> > > > >     > > > route,
> > > > >     > > >     >     > unless I am
> > > > >     > > >     >     >         >> > missing
> > > > >     > > >     >     >         >> > > something, it sounds like
> kafka
> > > will
> > > > > be
> > > > >     > stuck
> > > > >     > > > with
> > > > >     > > >     >     > supporting both
> > > > >     > > >     >     >         >> null
> > > > >     > > >     >     >         >> > > value and tombstone bit for
> log
> > > > > compaction
> > > > >     > for
> > > > >     > > > life
> > > > >     > > >     > long,
> > > > >     > > >     >     > which does
> > > > >     > > >     >     >         >> not
> > > > >     > > >     >     >         >> > > look like a good end state.
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > Thanks,
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > Mayuresh
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32
> AM,
> > > > > Mayuresh
> > > > >     > > > Gharat <
> > > > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> > > > >     > > >     >     >         >> > > > wrote:
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > > Hi Ismael,
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > That's a very good point
> > which I
> > > > > might
> > > > >     > have
> > > > >     > > > not
> > > > >     > > >     >     > considered earlier.
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > Here is a plan that I can
> > think
> > > > of:
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > Stage 1) The broker from now
> > on,
> > > > up
> > > > >     > converts
> > > > >     > > > the
> > > > >     > > >     > message
> > > > >     > > >     >     > to have the
> > > > >     > > >     >     >         >> > > > tombstone marker. The log
> > > > compaction
> > > > >     > thread
> > > > >     > > > does log
> > > > >     > > >     >     > compaction
> > > > >     > > >     >     >         >> based
> > > > >     > > >     >     >         >> > on
> > > > >     > > >     >     >         >> > > > both null and tombstone
> > marker.
> > > > > This is
> > > > >     > our
> > > > >     > > >     > transition
> > > > >     > > >     >     > period.
> > > > >     > > >     >     >         >> > > > Stage 2) The next release we
> > > only
> > > > > say that
> > > > >     > > log
> > > > >     > > >     > compaction
> > > > >     > > >     >     > is based
> > > > >     > > >     >     >         >> on
> > > > >     > > >     >     >         >> > > > tombstone marker. (Open
> source
> > > > > kafka makes
> > > > >     > > > this as a
> > > > >     > > >     >     > policy). By
> > > > >     > > >     >     >         >> this
> > > > >     > > >     >     >         >> > > time,
> > > > >     > > >     >     >         >> > > > the organization which is
> > moving
> > > > to
> > > > > this
> > > > >     > > > release
> > > > >     > > >     > will be
> > > > >     > > >     >     > sure that
> > > > >     > > >     >     >         >> they
> > > > >     > > >     >     >         >> > > > have gone through the entire
> > > > > transition
> > > > >     > > > period.
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > My only goal of doing this
> is
> > > that
> > > > > Kafka
> > > > >     > > > clearly
> > > > >     > > >     >     > specifies the end
> > > > >     > > >     >     >         >> > state
> > > > >     > > >     >     >         >> > > > about what log compaction
> > means
> > > > (is
> > > > > it
> > > > >     > null
> > > > >     > > > value
> > > > >     > > >     > or a
> > > > >     > > >     >     > tombstone
> > > > >     > > >     >     >         >> > marker,
> > > > >     > > >     >     >         >> > > > but not both).
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > What do you think?
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > Thanks,
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > Mayuresh
> > > > >     > > >     >     >         >> > > > .
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17
> > AM,
> > > > > Ismael
> > > > >     > > Juma <
> > > > >     > > >     >     > ismael@juma.me.uk>
> > > > >     > > >     >     >         >> > wrote:
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > >> One comment below.
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at
> 5:08
> > > PM,
> > > > > Mayuresh
> > > > >     > > > Gharat <
> > > > >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> > > > >     > > >     >     >         >> > > >> > wrote:
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >> >    - If we don't bump up
> > the
> > > > > magic
> > > > >     > byte,
> > > > >     > > > on the
> > > > >     > > >     > broker
> > > > >     > > >     >     > side, the
> > > > >     > > >     >     >         >> > > broker
> > > > >     > > >     >     >         >> > > >> >    will always have to
> look
> > > at
> > > > > both
> > > > >     > > > tombstone
> > > > >     > > >     > bit and
> > > > >     > > >     >     > the value
> > > > >     > > >     >     >         >> when
> > > > >     > > >     >     >         >> > > do
> > > > >     > > >     >     >         >> > > >> the
> > > > >     > > >     >     >         >> > > >> >    compaction. Assuming
> we
> > do
> > > > > not bump
> > > > >     > up
> > > > >     > > > the
> > > > >     > > >     > magic
> > > > >     > > >     >     > byte,
> > > > >     > > >     >     >         >> > > >> >    imagine the broker
> sees
> > a
> > > > > message
> > > > >     > > which
> > > > >     > > > does
> > > > >     > > >     > not
> > > > >     > > >     >     > have a
> > > > >     > > >     >     >         >> tombstone
> > > > >     > > >     >     >         >> > > bit
> > > > >     > > >     >     >         >> > > >> >    set. The broker does
> not
> > > > know
> > > > > when
> > > > >     > the
> > > > >     > > >     > message was
> > > > >     > > >     >     > produced
> > > > >     > > >     >     >         >> (i.e.
> > > > >     > > >     >     >         >> > > >> > whether
> > > > >     > > >     >     >         >> > > >> >    the message has been
> up
> > > > > converted or
> > > > >     > > > not), it
> > > > >     > > >     > has
> > > > >     > > >     >     > to take a
> > > > >     > > >     >     >         >> > further
> > > > >     > > >     >     >         >> > > >> > look at
> > > > >     > > >     >     >         >> > > >> >    the value to see if it
> > is
> > > > > null or
> > > > >     > not
> > > > >     > > in
> > > > >     > > >     > order to
> > > > >     > > >     >     > determine
> > > > >     > > >     >     >         >> if it
> > > > >     > > >     >     >         >> > > is
> > > > >     > > >     >     >         >> > > >> a
> > > > >     > > >     >     >         >> > > >> >    tombstone. The same
> > logic
> > > > has
> > > > > to be
> > > > >     > > put
> > > > >     > > > on the
> > > > >     > > >     >     > consumer as
> > > > >     > > >     >     >         >> well
> > > > >     > > >     >     >         >> > > >> because
> > > > >     > > >     >     >         >> > > >> > the
> > > > >     > > >     >     >         >> > > >> >    consumer does not know
> > if
> > > > the
> > > > >     > message
> > > > >     > > > has
> > > > >     > > >     > been up
> > > > >     > > >     >     > converted or
> > > > >     > > >     >     >         >> > not.
> > > > >     > > >     >     >         >> > > >> >       - If we upconvert
> > while
> > > > >     > appending,
> > > > >     > > > this is
> > > > >     > > >     > not
> > > > >     > > >     >     > the case,
> > > > >     > > >     >     >         >> > right?
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >> If I understand you
> > correctly,
> > > > > this is
> > > > >     > not
> > > > >     > > >     > sufficient
> > > > >     > > >     >     > because the
> > > > >     > > >     >     >         >> log
> > > > >     > > >     >     >         >> > > may
> > > > >     > > >     >     >         >> > > >> have messages appended
> before
> > > it
> > > > > was
> > > > >     > > > upgraded to
> > > > >     > > >     > include
> > > > >     > > >     >     > KIP-87.
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >> Ismael
> > > > >     > > >     >     >         >> > > >>
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > > > --
> > > > >     > > >     >     >         >> > > > -Regards,
> > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> > > > >     > > >     >     >         >> > > > (862) 250-7125
> > > > >     > > >     >     >         >> > > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > > --
> > > > >     > > >     >     >         >> > > -Regards,
> > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > > > >     > > >     >     >         >> > > (862) 250-7125
> > > > >     > > >     >     >         >> > >
> > > > >     > > >     >     >         >> > The information contained in
> this
> > > > email
> > > > > is
> > > > >     > > > strictly
> > > > >     > > >     >     > confidential and for
> > > > >     > > >     >     >         >> > the use of the addressee only,
> > > unless
> > > > >     > otherwise
> > > > >     > > >     > indicated. If
> > > > >     > > >     >     > you are
> > > > >     > > >     >     >         >> not
> > > > >     > > >     >     >         >> > the intended recipient, please
> do
> > > not
> > > > > read,
> > > > >     > > copy,
> > > > >     > > > use or
> > > > >     > > >     >     > disclose to
> > > > >     > > >     >     >         >> others
> > > > >     > > >     >     >         >> > this message or any attachment.
> > > Please
> > > > > also
> > > > >     > > > notify the
> > > > >     > > >     > sender
> > > > >     > > >     >     > by
> > > > >     > > >     >     >         >> replying
> > > > >     > > >     >     >         >> > to this email or by telephone
> > > (+44(020
> > > > > 7896
> > > > >     > > 0011)
> > > > >     > > > and
> > > > >     > > >     > then
> > > > >     > > >     >     > delete the
> > > > >     > > >     >     >         >> email
> > > > >     > > >     >     >         >> > and any copies of it. Opinions,
> > > > > conclusion
> > > > >     > (etc)
> > > > >     > > > that
> > > > >     > > >     > do not
> > > > >     > > >     >     > relate to
> > > > >     > > >     >     >         >> the
> > > > >     > > >     >     >         >> > official business of this
> company
> > > > shall
> > > > > be
> > > > >     > > > understood as
> > > > >     > > >     >     > neither given
> > > > >     > > >     >     >         >> nor
> > > > >     > > >     >     >         >> > endorsed by it. IG is a trading
> > name
> > > > of
> > > > > IG
> > > > >     > > Markets
> > > > >     > > >     > Limited (a
> > > > >     > > >     >     > company
> > > > >     > > >     >     >         >> > registered in England and Wales,
> > > > company
> > > > >     > number
> > > > >     > > >     > 04008957) and
> > > > >     > > >     >     > IG Index
> > > > >     > > >     >     >         >> > Limited (a company registered in
> > > > > England and
> > > > >     > > > Wales,
> > > > >     > > >     > company
> > > > >     > > >     >     > number
> > > > >     > > >     >     >         >> > 01190902). Registered address at
> > > > Cannon
> > > > > Bridge
> > > > >     > > > House, 25
> > > > >     > > >     >     > Dowgate Hill,
> > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
> > > > Limited
> > > > >     > > (register
> > > > >     > > >     > number
> > > > >     > > >     >     > 195355) and IG
> > > > >     > > >     >     >         >> > Index Limited (register number
> > > 114059)
> > > > > are
> > > > >     > > > authorised
> > > > >     > > >     > and
> > > > >     > > >     >     > regulated by
> > > > >     > > >     >     >         >> the
> > > > >     > > >     >     >         >> > Financial Conduct Authority.
> > > > >     > > >     >     >         >> >
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >> --
> > > > >     > > >     >     >         >> -Regards,
> > > > >     > > >     >     >         >> Mayuresh R. Gharat
> > > > >     > > >     >     >         >> (862) 250-7125
> > > > >     > > >     >     >         >> The information contained in this
> > > email
> > > > is
> > > > >     > > strictly
> > > > >     > > >     >     > confidential and for
> > > > >     > > >     >     >         >> the use of the addressee only,
> > unless
> > > > > otherwise
> > > > >     > > >     > indicated. If
> > > > >     > > >     >     > you are not
> > > > >     > > >     >     >         >> the intended recipient, please do
> > not
> > > > > read,
> > > > >     > copy,
> > > > >     > > > use or
> > > > >     > > >     >     > disclose to others
> > > > >     > > >     >     >         >> this message or any attachment.
> > Please
> > > > > also
> > > > >     > notify
> > > > >     > > > the
> > > > >     > > >     > sender
> > > > >     > > >     >     > by replying
> > > > >     > > >     >     >         >> to this email or by telephone
> > (+44(020
> > > > > 7896
> > > > >     > 0011)
> > > > >     > > > and then
> > > > >     > > >     >     > delete the email
> > > > >     > > >     >     >         >> and any copies of it. Opinions,
> > > > > conclusion (etc)
> > > > >     > > > that do
> > > > >     > > >     > not
> > > > >     > > >     >     > relate to the
> > > > >     > > >     >     >         >> official business of this company
> > > shall
> > > > be
> > > > >     > > > understood as
> > > > >     > > >     >     > neither given nor
> > > > >     > > >     >     >         >> endorsed by it. IG is a trading
> name
> > > of
> > > > IG
> > > > >     > Markets
> > > > >     > > >     > Limited (a
> > > > >     > > >     >     > company
> > > > >     > > >     >     >         >> registered in England and Wales,
> > > company
> > > > > number
> > > > >     > > > 04008957)
> > > > >     > > >     > and
> > > > >     > > >     >     > IG Index
> > > > >     > > >     >     >         >> Limited (a company registered in
> > > England
> > > > > and
> > > > >     > > Wales,
> > > > >     > > >     > company
> > > > >     > > >     >     > number
> > > > >     > > >     >     >         >> 01190902). Registered address at
> > > Cannon
> > > > > Bridge
> > > > >     > > > House, 25
> > > > >     > > >     >     > Dowgate Hill,
> > > > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
> > > Limited
> > > > >     > (register
> > > > >     > > > number
> > > > >     > > >     >     > 195355) and IG
> > > > >     > > >     >     >         >> Index Limited (register number
> > 114059)
> > > > are
> > > > >     > > > authorised and
> > > > >     > > >     >     > regulated by the
> > > > >     > > >     >     >         >> Financial Conduct Authority.
> > > > >     > > >     >     >         >>
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >         > --
> > > > >     > > >     >     >         > -Regards,
> > > > >     > > >     >     >         > Mayuresh R. Gharat
> > > > >     > > >     >     >         > (862) 250-7125
> > > > >     > > >     >     >         >
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >         --
> > > > >     > > >     >     >         -Regards,
> > > > >     > > >     >     >         Mayuresh R. Gharat
> > > > >     > > >     >     >         (862) 250-7125
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >     The information contained in this email
> is
> > > > > strictly
> > > > >     > > > confidential
> > > > >     > > >     > and
> > > > >     > > >     >     > for the use of the addressee only, unless
> > > otherwise
> > > > >     > > indicated.
> > > > >     > > > If
> > > > >     > > >     > you are
> > > > >     > > >     >     > not the intended recipient, please do not
> read,
> > > > > copy, use
> > > > >     > or
> > > > >     > > >     > disclose to
> > > > >     > > >     >     > others this message or any attachment. Please
> > > also
> > > > > notify
> > > > >     > the
> > > > >     > > > sender
> > > > >     > > >     > by
> > > > >     > > >     >     > replying to this email or by telephone
> (+44(020
> > > > 7896
> > > > > 0011)
> > > > >     > > and
> > > > >     > > > then
> > > > >     > > >     > delete
> > > > >     > > >     >     > the email and any copies of it. Opinions,
> > > > conclusion
> > > > > (etc)
> > > > >     > > > that do
> > > > >     > > >     > not
> > > > >     > > >     >     > relate to the official business of this
> company
> > > > > shall be
> > > > >     > > > understood
> > > > >     > > >     > as
> > > > >     > > >     >     > neither given nor endorsed by it. IG is a
> > trading
> > > > > name of
> > > > >     > IG
> > > > >     > > > Markets
> > > > >     > > >     >     > Limited (a company registered in England and
> > > Wales,
> > > > > company
> > > > >     > > > number
> > > > >     > > >     >     > 04008957) and IG Index Limited (a company
> > > > registered
> > > > > in
> > > > >     > > > England and
> > > > >     > > >     > Wales,
> > > > >     > > >     >     > company number 01190902). Registered address
> at
> > > > > Cannon
> > > > >     > Bridge
> > > > >     > > > House,
> > > > >     > > >     > 25
> > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG
> Markets
> > > > > Limited
> > > > >     > > > (register
> > > > >     > > >     > number
> > > > >     > > >     >     > 195355) and IG Index Limited (register number
> > > > > 114059) are
> > > > >     > > > authorised
> > > > >     > > >     > and
> > > > >     > > >     >     > regulated by the Financial Conduct Authority.
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >     >
> > > > >     > > >     >
> > > > >     > > >     >
> > > > >     > > >     > The information contained in this email is strictly
> > > > > confidential
> > > > >     > > and
> > > > >     > > > for
> > > > >     > > >     > the use of the addressee only, unless otherwise
> > > > indicated.
> > > > > If you
> > > > >     > > > are not
> > > > >     > > >     > the intended recipient, please do not read, copy,
> use
> > > or
> > > > > disclose
> > > > >     > > to
> > > > >     > > > others
> > > > >     > > >     > this message or any attachment. Please also notify
> > the
> > > > > sender by
> > > > >     > > > replying
> > > > >     > > >     > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and
> > > > then
> > > > > delete
> > > > >     > > > the email
> > > > >     > > >     > and any copies of it. Opinions, conclusion (etc)
> that
> > > do
> > > > > not
> > > > >     > relate
> > > > >     > > > to the
> > > > >     > > >     > official business of this company shall be
> understood
> > > as
> > > > > neither
> > > > >     > > > given nor
> > > > >     > > >     > endorsed by it. IG is a trading name of IG Markets
> > > > Limited
> > > > > (a
> > > > >     > > company
> > > > >     > > >     > registered in England and Wales, company number
> > > 04008957)
> > > > > and IG
> > > > >     > > > Index
> > > > >     > > >     > Limited (a company registered in England and Wales,
> > > > company
> > > > >     > number
> > > > >     > > >     > 01190902). Registered address at Cannon Bridge
> House,
> > > 25
> > > > > Dowgate
> > > > >     > > > Hill,
> > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > > number
> > > > > 195355)
> > > > >     > > > and IG
> > > > >     > > >     > Index Limited (register number 114059) are
> authorised
> > > and
> > > > >     > regulated
> > > > >     > > > by the
> > > > >     > > >     > Financial Conduct Authority.
> > > > >     > > >     >
> > > > >     > > >
> > > > >     > > >
> > > > >     > > > The information contained in this email is strictly
> > > > confidential
> > > > > and
> > > > >     > for
> > > > >     > > > the use of the addressee only, unless otherwise
> indicated.
> > If
> > > > > you are
> > > > >     > not
> > > > >     > > > the intended recipient, please do not read, copy, use or
> > > > > disclose to
> > > > >     > > others
> > > > >     > > > this message or any attachment. Please also notify the
> > sender
> > > > by
> > > > >     > replying
> > > > >     > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > > > > delete the
> > > > >     > > email
> > > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > > > relate to
> > > > >     > > the
> > > > >     > > > official business of this company shall be understood as
> > > > neither
> > > > > given
> > > > >     > > nor
> > > > >     > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > > > company
> > > > >     > > > registered in England and Wales, company number 04008957)
> > and
> > > > IG
> > > > > Index
> > > > >     > > > Limited (a company registered in England and Wales,
> company
> > > > > number
> > > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
> > > > Dowgate
> > > > > Hill,
> > > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > > > 195355) and
> > > > >     > IG
> > > > >     > > > Index Limited (register number 114059) are authorised and
> > > > > regulated by
> > > > >     > > the
> > > > >     > > > Financial Conduct Authority.
> > > > >     > > >
> > > > >     > > The information contained in this email is strictly
> > > confidential
> > > > > and for
> > > > >     > > the use of the addressee only, unless otherwise indicated.
> If
> > > you
> > > > > are not
> > > > >     > > the intended recipient, please do not read, copy, use or
> > > disclose
> > > > > to
> > > > >     > others
> > > > >     > > this message or any attachment. Please also notify the
> sender
> > > by
> > > > > replying
> > > > >     > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > > delete
> > > > > the
> > > > >     > email
> > > > >     > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > > > relate to
> > > > >     > the
> > > > >     > > official business of this company shall be understood as
> > > neither
> > > > > given
> > > > >     > nor
> > > > >     > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > > > company
> > > > >     > > registered in England and Wales, company number 04008957)
> and
> > > IG
> > > > > Index
> > > > >     > > Limited (a company registered in England and Wales, company
> > > > number
> > > > >     > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > > > Hill,
> > > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355)
> > > > > and IG
> > > > >     > > Index Limited (register number 114059) are authorised and
> > > > > regulated by
> > > > >     > the
> > > > >     > > Financial Conduct Authority.
> > > > >     > >
> > > > >     > >
> > > > >     > The information contained in this email is strictly
> > confidential
> > > > and
> > > > > for
> > > > >     > the use of the addressee only, unless otherwise indicated. If
> > you
> > > > > are not
> > > > >     > the intended recipient, please do not read, copy, use or
> > disclose
> > > > to
> > > > > others
> > > > >     > this message or any attachment. Please also notify the sender
> > by
> > > > > replying
> > > > >     > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > delete
> > > > > the email
> > > > >     > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate
> > > > > to the
> > > > >     > official business of this company shall be understood as
> > neither
> > > > > given nor
> > > > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > > company
> > > > >     > registered in England and Wales, company number 04008957) and
> > IG
> > > > > Index
> > > > >     > Limited (a company registered in England and Wales, company
> > > number
> > > > >     > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > > Hill,
> > > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > > and IG
> > > > >     > Index Limited (register number 114059) are authorised and
> > > regulated
> > > > > by the
> > > > >     > Financial Conduct Authority.
> > > > >     >
> > > > >     >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly confidential
> and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If you
> are
> > > not
> > > > > the intended recipient, please do not read, copy, use or disclose
> to
> > > > others
> > > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to
> > > > the
> > > > > official business of this company shall be understood as neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > > registered in England and Wales, company number 04008957) and IG
> > Index
> > > > > Limited (a company registered in England and Wales, company number
> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and regulated
> > by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> and then delete the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Ewen, Jay,

We discussed internally. If this is the reason for not giving a +1 to the below, we assume you want a solution where we can provide all 4 on day one.

To do this our idea is to simply leave the compaction policy alone, it continues to work based on null values, but instead we make a new policy “tombstone-compaction” want for a better name.

As such we only need to deal with new clients, and flows, to which then we can essentially do away with any need for a migration, or compromised due to the historic compaction.

Whilst this does deviate from the VOTE its unfortunate these concerns not raised during the lengthy DISCUSS but it’s worth discussing to get the votes needed, as we’d like to make this for the Jan release.

Thoughts? Would having a new policy and no migration but supporting all four combinations be preferred over updating the compaction policy and mechanics?

@Mayuresh, Radai, Becket, Jun as you’ve all +1 voted on the existing solution to do this by upgrading the existing compaction policy would any of you have concerns with the alternative?

Regards,
Mike

On 11/12/2016, 23:33, "Michael Pearce" <Mi...@ig.com> wrote:

    Hi Ewen
    
    So this is what got discussed in the discussion thread. We wanted to go for supporting all four combinations
    
    Tombstone + non null value
    Tombstone + null value
    Non tombstone + non null value
    Non tombstone + null value
    
    The issues were mostly around compatibility with old consumers.
    
    In the end the consensus was for reasons of making an upgrade/compatibility we would get to the position of handling in this KIP:
    
    Tombstone + null value
    Tombstone + non null value
    Non tombstone + non null value
    
    Only later once either we can break client compatibility or time expires long enough we can say consumers should no longer be inferring from null value instead the isTombstone flag can we support:
    
    Non tombstone + null value.
    
    Saying that this puts in a lot of the work to enable the last piece.
    
    Also re having the change events of that makes the state aka compacted topic by doing what we're doing would give this to the supported combinations above mentioned.
    
    Ideally I'd love to have ability to support non tombstone + null value, unfortunately this is some of the tech debt in the original compaction design we can't easily undo, without a period of time.
    
    Regards
    Mike
    
    Sent using OWA for iPhone
    ________________________________________
    From: Ewen Cheslack-Postava <ew...@confluent.io>
    Sent: Sunday, December 11, 2016 11:15:42 PM
    To: dev@kafka.apache.org
    Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    
    Michael,
    
    It kind of depends on how you want to interpret the tombstone flag. If it's
    purely a producer-facing Kafka-level thing that we treat as internal to the
    broker and log cleaner once the record is sent, then your approach makes
    sense. You're just moving copying the null-indicates-delete behavior of the
    old constructor into the tombstone flag.
    
    However, if you want this change to more generally decouple the idea of
    deletion and null values, then you are sometimes converting what might be a
    completely valid null value that doesn't indicate deletion into a
    tombstone. Downstream applications could potentially handle these cases
    differently given the separation of deletion from value.
    
    I guess the question is if we want to try to support the latter even for
    topics where we have older produce requests. An example where this could
    come up is in something like a CDC Connector. If we try to support the
    semantic difference, a connector might write changes to Kafka using the
    tombstone flag to indicate when a row was truly deleted (vs an update that
    sets it to null but still present; this probably makes more sense for CDC
    from document stores or extracting single columns). There are various
    reasons we might want to maintain the full log and not turn compaction on
    (or just use a time-based retention policy), but downstream applications
    might care to know the difference between a delete and a null value. In
    fact both versions of the same log (compacted and time-retention) could be
    useful and I don't think it'll be uncommon to maintain both or use KIP-71
    to maintain a hybrid compacted/retention topic.
    
    -Ewen
    
    On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <Mi...@ig.com>
    wrote:
    
    > Hi Jay,
    >
    > Why wouldn't that work, the tombstone value is only looked at by the
    > broker, on a topic configured for compaction as such is benign on non
    > compacted topics. This is as much as sending a null value currently
    >
    >
    > Regards
    > Mike
    >
    >
    >
    > Sent using OWA for iPhone
    > ________________________________________
    > From: Jay Kreps <ja...@confluent.io>
    > Sent: Sunday, December 11, 2016 8:58:53 PM
    > To: dev@kafka.apache.org
    > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >
    > Hey Michael,
    >
    > I'm not quite sure that works as that would translate ALL null values to
    > tombstones, even for non-compacted topics that use null as an acceptable
    > value sent by the producer and expected by the consumer.
    >
    > -Jay
    >
    > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Mi...@ig.com>
    > wrote:
    >
    > > Hi Ewen,
    > >
    > > I think the easiest way to show this is with code.
    > >
    > > As you can see we keep the existing behaviour for code/binaries calling
    > > the pre-existing constructors, whereby if the value is null the tombstone
    > > is set to true.
    > >
    > > Regards
    > > Mike
    > >
    > >
    > >
    > >     /**
    > >      * Creates a record with a specified timestamp to be sent to a
    > > specified topic and partition
    > >      *
    > >      * @param topic The topic the record will be appended to
    > >      * @param partition The partition to which the record should be sent
    > >      * @param timestamp The timestamp of the record
    > >      * @param tombstone if the record should be treated as a tombstone if
    > > the topic is compacted
    > >      * @param key The key that will be included in the record
    > >      * @param value The record contents
    > >      */
    > >     public ProducerRecord(String topic, Integer partition, Boolean
    > > tombstone, Long timestamp, K key, V value) {
    > >         if (topic == null)
    > >             throw new IllegalArgumentException("Topic cannot be null.");
    > >         if (timestamp != null && timestamp < 0)
    > >             throw new IllegalArgumentException(
    > >                     String.format("Invalid timestamp: %d. Timestamp
    > should
    > > always be non-negative or null.", timestamp));
    > >         if (partition != null && partition < 0)
    > >             throw new IllegalArgumentException(
    > >                     String.format("Invalid partition: %d. Partition
    > number
    > > should always be non-negative or null.", partition));
    > >         if (!tombstone && value == null){
    > >             throw new IllegalArgumentException(
    > >                     String.format("Tombstone must be true if null
    > value"));
    > >         }
    > >         this.topic = topic;
    > >         this.partition = partition;
    > >         this.tombstone = tombstone;
    > >         this.key = key;
    > >         this.value = value;
    > >         this.timestamp = timestamp;
    > >     }
    > >
    > >     public ProducerRecord(String topic, Integer partition, Long
    > timestamp,
    > > K key, V value) {
    > >         this(topic, partition, value == null, timestamp, key, value);
    > >     }
    > > ________________________________________
    > > From: Ewen Cheslack-Postava <ew...@confluent.io>
    > > Sent: Sunday, December 11, 2016 5:45 AM
    > > To: dev@kafka.apache.org
    > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > >
    > > It seemed like this was addressed in the migration section, wasn't it?
    > The
    > > V2 vs V3 requests and the fact that the broker will downgrade the message
    > > format (losing zero copy) if you issues a V2 request to a broker using V3
    > > format handles compatibility, does it not? The existing consumers won't
    > see
    > > the extra metadata in the value, but they will get a null instead and
    > treat
    > > it as a tombstone. Certainly there is a performance impact, but it seemed
    > > compatible.
    > >
    > > I'm worried about this though:
    > >
    > >
    > >    - *NOTE *: With the new version of producer client using
    > ProduceRequest
    > >    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and null
    > > value
    > >    should not be allowed as the older version of consumer using
    > > FetchRequest
    > >    V2 will think of this as a tombstone when its actually not.
    > >
    > >
    > > Unless I'm misunderstanding, this ends up breaking binary compatibility
    > for
    > > the Java API. It sounds like this suggests that passing a null value to
    > the
    > > existing ProducerRecord constructors would generate an exception since
    > you
    > > didn't explicitly enable the tombstone (via whatever new constructor is
    > > provided). But that means you can't swap in a newer client jar without
    > > recompiling and get the same behavior if your app deletes keys using the
    > > current approach because your app will start throwing exceptions. Maybe
    > > this is a tradeoff we're ok with, but we've tried pretty hard to avoid
    > > breaking compatibility like this so far -- it makes picking up bug fixes
    > in
    > > the clients harder for users of frameworks that have to pin to earlier
    > > default versions for compatibility.
    > >
    > > But then later the KIP says:
    > >
    > >
    > >    - The old constructors for ProducerRecord without this parameter will
    > be
    > >    kept but updated so that their default behaviour if setting null value
    > > will
    > >    be the tombstone will be set to true to keep existing behaviour.
    > >
    > >
    > > So maybe I am misinterpreting something.
    > >
    > > And just a nit re: motivation section, item 6 would be clearer for a
    > union
    > > schema with null (or optional schemas in other formats), e.g. [null,
    > > string], in which case losing the schema truly is losing information
    > > (whereas null is already the only valid value for a pure null schema).
    > >
    > > -Ewen
    > >
    > >
    > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Mi...@ig.com>
    > > wrote:
    > >
    > > > Hi Jay,
    > > >
    > > > Good point this detail is missing in the KIP write up. Ive added this
    > > now.
    > > >
    > > > Essentially simply just upgrading the clients we do not expect any
    > client
    > > > app code change needed.
    > > >
    > > > Cheers
    > > > Mike
    > > > ________________________________________
    > > > From: Jay Kreps <ja...@confluent.io>
    > > > Sent: Saturday, December 10, 2016 10:51 PM
    > > > To: dev@kafka.apache.org
    > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > >
    > > > Michael,
    > > >
    > > > The compatibility section goes through the migration path, but isn't
    > the
    > > > bigger compatibility issue with existing apps? There are many (probably
    > > > thousands) of apps in production that use this feature and send null to
    > > > mean delete. It seems like this would break compatibility with them,
    > and
    > > > they would have to be rewritten to right?
    > > >
    > > > -Jay
    > > >
    > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Michael.Pearce@ig.com
    > >
    > > > wrote:
    > > >
    > > > > Hi Jun,
    > > > >
    > > > > 4) On v3 we honour the tombstone. As such we expect it to be set
    > > > correctly
    > > > > as per the KIP.
    > > > >
    > > > > 4.1) why would we want to produce an error when its v3? This is the
    > > exact
    > > > > purpose to support non-null tombstone’s
    > > > > 4.2) again here im unclear on the question on the v3, produce request
    > > we
    > > > > expect the tombstone flag to be set correctly.
    > > > >
    > > > > 4.4) compaction only occurs on compacted topics, the bit makes no
    > > > > difference and not looked at on un-compacted (time/size based
    > > eviction).
    > > > >
    > > > >
    > > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
    > > > >
    > > > >     Hi, Michael,
    > > > >
    > > > >     4. Then, I think I misunderstood this point. Could you document
    > the
    > > > >     following points in the wiki?
    > > > >     4.1 If producer V3 sets tombstone, but provides a non-null value,
    > > > does
    > > > > the
    > > > >     send() get an error or does the producer automatically set the
    > > value
    > > > to
    > > > >     null?
    > > > >     4.2 If producer V3 doesn't set tombstone, but provides a null
    > > value,
    > > > > does
    > > > >     the send() get an error or does the producer automatically sets
    > the
    > > > >     tombstone?
    > > > >     4.3 Does the broker only expect messages that (a) have no
    > tombstone
    > > > and
    > > > >     non-null value; (b) have tombstone and null value and reject the
    > > > > messages
    > > > >     with an error code otherwise?
    > > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on
    > > > not?
    > > > >
    > > > >     Thanks,
    > > > >
    > > > >     Jun
    > > > >
    > > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
    > > > Michael.Pearce@ig.com
    > > > > >
    > > > >     wrote:
    > > > >
    > > > >     > Not at all.  This only acts on compacted topics just as what
    > > occurs
    > > > > today
    > > > >     >
    > > > >     > Sent using OWA for iPhone
    > > > >     > ________________________________________
    > > > >     > From: Jun Rao <ju...@confluent.io>
    > > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
    > > > >     > To: dev@kafka.apache.org
    > > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > >     >
    > > > >     > Hi, Michael,
    > > > >     >
    > > > >     > 4. Hmm, does that mean the new client library can never send a
    > > null
    > > > > message
    > > > >     > even to a regular topic? This seems like a change of the
    > existing
    > > > > behavior.
    > > > >     >
    > > > >     > Thanks,
    > > > >     >
    > > > >     > Jun
    > > > >     >
    > > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
    > > > > Michael.Pearce@ig.com>
    > > > >     > wrote:
    > > > >     >
    > > > >     > > Hi Jun,
    > > > >     > >
    > > > >     > > Re 4) That's because we expect the tombstone value to be set
    > > > > correctly if
    > > > >     > > message bit is 2, as such if an older client sends in on old
    > > > > message the
    > > > >     > > message is upcast and the bit is set correctly. And such no
    > > > longer
    > > > > need
    > > > >     > to
    > > > >     > > check the value. Mayuresh can you confirm my thinking and
    > > > > understanding
    > > > >     > of
    > > > >     > > what we've discussed?
    > > > >     > >
    > > > >     > > The second point I understand what you're getting at now my
    > > > > apologies.
    > > > >     > Yes
    > > > >     > > this makes sense to save on touching the message, if we're
    > the
    > > > > only kip
    > > > >     > > going in, in this release.
    > > > >     > >
    > > > >     > > Cheers
    > > > >     > > Mike
    > > > >     > >
    > > > >     > > Sent using OWA for iPhone
    > > > >     > > ________________________________________
    > > > >     > > From: Jun Rao <ju...@confluent.io>
    > > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
    > > > >     > > To: dev@kafka.apache.org
    > > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > >     > >
    > > > >     > > Hi, Michael,
    > > > >     > >
    > > > >     > > 4. Is this updated in the wiki? The text "If the magic byte
    > on
    > > > > message is
    > > > >     > > 2, the broker should use the tombstone bit for log
    > compaction."
    > > > > doesn't
    > > > >     > > seem to have changed.
    > > > >     > >
    > > > >     > > 2. My point is that if we change the message format just for
    > > this
    > > > > KIP, we
    > > > >     > > should consider whether it's worth optimizing the down
    > > conversion
    > > > > path
    > > > >     > > (i.e., decide whether a conversion is needed by just looking
    > at
    > > > the
    > > > >     > > tombstone bit in the wrapper message) since tombstone will be
    > > > used
    > > > >     > rarely.
    > > > >     > > However, if the message format change here is combined with
    > > other
    > > > > KIPs,
    > > > >     > > then this optimization likely won't be needed. The latter
    > > > probably
    > > > > makes
    > > > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
    > > > >     > >
    > > > >     > > Other than those, +1 from me,
    > > > >     > >
    > > > >     > > Thanks,
    > > > >     > >
    > > > >     > > Jun
    > > > >     > >
    > > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
    > > > > Michael.Pearce@ig.com>
    > > > >     > > wrote:
    > > > >     > >
    > > > >     > > > Hi Jun
    > > > >     > > >
    > > > >     > > > do we have your vote on this now?
    > > > >     > > >
    > > > >     > > > Any other concerns?
    > > > >     > > >
    > > > >     > > > Cheers
    > > > >     > > > Mike
    > > > >     > > >
    > > > >     > > > Sent using OWA for iPhone
    > > > >     > > > ________________________________________
    > > > >     > > > From: Michael Pearce <Mi...@ig.com>
    > > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
    > > > >     > > > To: dev@kafka.apache.org
    > > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > > >     > > >
    > > > >     > > > Hi Jun,
    > > > >     > > >
    > > > >     > > > Have updated. Thanks again for the feedback.
    > > > >     > > >
    > > > >     > > > Agree yes we should align up when it gets to that, I assume
    > > > > you’ve
    > > > >     > > flagged
    > > > >     > > > the same in the other KIP?
    > > > >     > > >
    > > > >     > > > I think for now let’s get this KIP approved, then we can
    > > start
    > > > > the work
    > > > >     > > to
    > > > >     > > > get an initial PR, then we can discuss how to align the two
    > > if
    > > > > needed.
    > > > >     > > >
    > > > >     > > > Cheers,
    > > > >     > > > Mike
    > > > >     > > >
    > > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
    > > > >     > > >
    > > > >     > > >     Hi, Michael,
    > > > >     > > >
    > > > >     > > >     For 2), this is fine. Could you update the KIP wiki to
    > > make
    > > > > this
    > > > >     > and
    > > > >     > > > other
    > > > >     > > >     points clearer? Other than that, the KIP looks good to
    > > me.
    > > > >     > > >
    > > > >     > > >     An orthogonal thing is that there are other KIPs such
    > as
    > > > > KIP-98
    > > > >     > that
    > > > >     > > > also
    > > > >     > > >     intend to change the message format. If they all get
    > > > > approved, we
    > > > >     > > > should
    > > > >     > > >     think about whether it's better to just bump up the
    > magic
    > > > > byte once
    > > > >     > > to
    > > > >     > > >     incorporate multiple format changes like we did in
    > > > > KIP-31/KIP-32.
    > > > >     > > >
    > > > >     > > >     Thanks,
    > > > >     > > >
    > > > >     > > >     Jun
    > > > >     > > >
    > > > >     > > >
    > > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
    > > > >     > > Michael.Pearce@ig.com>
    > > > >     > > >     wrote:
    > > > >     > > >
    > > > >     > > >     > Hi Jao
    > > > >     > > >     >
    > > > >     > > >     > Thanks for the response. Sorry for slow reply, both
    > > with
    > > > > personal
    > > > >     > > > sickness
    > > > >     > > >     > and also battling some critical issues encountered
    > > since
    > > > >     > upgrading
    > > > >     > > to
    > > > >     > > >     > 0.10.1.0
    > > > >     > > >     >
    > > > >     > > >     > 1) Thans for spotting, Document error where we
    > branched
    > > > > this KIP
    > > > >     > > from
    > > > >     > > >     > KIP-82, will get that removed.
    > > > >     > > >     > 2) Intent is to do this just at the record message
    > > level.
    > > > >     > > >     > 3) Thanks for spotting, Will ensure this is
    > corrected.
    > > > >     > > >     > 4) As per discussion thread we will support
    > tombstone +
    > > > > null
    > > > >     > value,
    > > > >     > > >     > tombstone + non null value, no tombstone + null
    > value.
    > > > >     > > >     > 5) I believe this was in the discussion thread,
    > > @Mayuresh
    > > > > is this
    > > > >     > > >     > something we’ve overlooked? I thought we would down
    > > > > convert and
    > > > >     > > > remove the
    > > > >     > > >     > value so the old consumer had existing behavior, or
    > is
    > > > > there
    > > > >     > > > something we
    > > > >     > > >     > haven’t thought about?
    > > > >     > > >     >
    > > > >     > > >     > Cheers
    > > > >     > > >     > Mike
    > > > >     > > >     >
    > > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
    > > > wrote:
    > > > >     > > >     >
    > > > >     > > >     >     Hi, Michael,
    > > > >     > > >     >
    > > > >     > > >     >     Thanks for the KIP. A few comments below.
    > > > >     > > >     >
    > > > >     > > >     >     1. The message format change contains
    > > "HeadersLength
    > > > >     > Headers".
    > > > >     > > > Is that
    > > > >     > > >     >     intended?
    > > > >     > > >     >
    > > > >     > > >     >     2. For compressed messageset, is the tombstone
    > bit
    > > > > only set
    > > > >     > at
    > > > >     > > > the
    > > > >     > > >     > shallow
    > > > >     > > >     >     level? Do we always leave that bit in the wrapper
    > > > > message
    > > > >     > > unset?
    > > > >     > > > An
    > > > >     > > >     >     alternative is to set the tombstone bit in the
    > > > wrapper
    > > > > if at
    > > > >     > > > least one
    > > > >     > > >     >     inner message has the tombstone bit set. This
    > makes
    > > > > things a
    > > > >     > > bit
    > > > >     > > > more
    > > > >     > > >     >     complicated, but we could potentially exploit
    > that
    > > > for
    > > > >     > > > optimizing down
    > > > >     > > >     >     conversion. For example, we only need to convert
    > > > > messages
    > > > >     > with
    > > > >     > > > magic 2
    > > > >     > > >     > to
    > > > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
    > > > > (conversion is
    > > > >     > > > always
    > > > >     > > >     > needed
    > > > >     > > >     >     from magic 2 to magic 0). Not sure if the
    > > > optimization
    > > > > is
    > > > >     > worth
    > > > >     > > > the
    > > > >     > > >     >     complexity though.
    > > > >     > > >     >
    > > > >     > > >     >     3. The referencing of the new version of
    > > > >     > > > ProducerRequest/FetchRequest
    > > > >     > > >     > is
    > > > >     > > >     >     inconsistent (v4 vs v3). Since our convention
    > > starts
    > > > at
    > > > >     > version
    > > > >     > > > at 0, I
    > > > >     > > >     >     think the new version would be 3.
    > > > >     > > >     >
    > > > >     > > >     >     4. "If the magic byte on message is 2, the broker
    > > > > should use
    > > > >     > > the
    > > > >     > > >     > tombstone
    > > > >     > > >     >     bit for log compaction." What about null value?
    > My
    > > > >     > > understanding
    > > > >     > > > is
    > > > >     > > >     > that
    > > > >     > > >     >     null value will be treated the same as setting
    > the
    > > > > tombstone
    > > > >     > > bit.
    > > > >     > > >     >
    > > > >     > > >     >     5. For the migration path, it would be useful to
    > > > > describe the
    > > > >     > > > down
    > > > >     > > >     >     conversion path to consumers (i.e., brokers on
    > > > message
    > > > > format
    > > > >     > > > 0.10.2
    > > > >     > > >     > and
    > > > >     > > >     >     consumers on older version).
    > > > >     > > >     >
    > > > >     > > >     >     Thanks,
    > > > >     > > >     >
    > > > >     > > >     >     Jun
    > > > >     > > >     >
    > > > >     > > >     >
    > > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
    > > > >     > > > Michael.Pearce@ig.com
    > > > >     > > >     > >
    > > > >     > > >     >     wrote:
    > > > >     > > >     >
    > > > >     > > >     >     > Hi All,
    > > > >     > > >     >     >
    > > > >     > > >     >     > We have been discussing in the below thread and
    > > > final
    > > > >     > changes
    > > > >     > > > have
    > > > >     > > >     > been
    > > > >     > > >     >     > made to the KIP wiki based on these
    > discussions.
    > > > >     > > >     >     >
    > > > >     > > >     >     > We would now like to put to the vote the
    > > following
    > > > > KIP:
    > > > >     > > >     >     > https://cwiki.apache.org/
    > > > > confluence/display/KAFKA/KIP-
    > > > >     > 87+-+
    > > > >     > > >     >     > Add+Compaction+Tombstone+Flag
    > > > >     > > >     >     >
    > > > >     > > >     >     > This kip is for having a distinct compaction
    > > > > attribute
    > > > >     > > > “tombstone”
    > > > >     > > >     > flag
    > > > >     > > >     >     > instead of relying on null value, allowing
    > > non-null
    > > > > value
    > > > >     > > > delete
    > > > >     > > >     > messages.
    > > > >     > > >     >     >
    > > > >     > > >     >     > Many thanks,
    > > > >     > > >     >     > Michael
    > > > >     > > >     >     >
    > > > >     > > >     >     >
    > > > >     > > >     >     >
    > > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
    > > > >     > > Michael.Pearce@ig.com>
    > > > >     > > >     > wrote:
    > > > >     > > >     >     >
    > > > >     > > >     >     >     Hi Mayuresh,
    > > > >     > > >     >     >
    > > > >     > > >     >     >     LGTM. Ive just made one small adjustment
    > > > > updating the
    > > > >     > > wire
    > > > >     > > >     > protocol to
    > > > >     > > >     >     > show the magic byte bump.
    > > > >     > > >     >     >
    > > > >     > > >     >     >     Do we think we’re good to put to a vote? Is
    > > > > there any
    > > > >     > > > other bits
    > > > >     > > >     >     > needing discussion?
    > > > >     > > >     >     >
    > > > >     > > >     >     >     Cheers
    > > > >     > > >     >     >     Mike
    > > > >     > > >     >     >
    > > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
    > > > >     > > >     > gharatmayuresh15@gmail.com>
    > > > >     > > >     >     > wrote:
    > > > >     > > >     >     >
    > > > >     > > >     >     >         Hi Michael,
    > > > >     > > >     >     >
    > > > >     > > >     >     >         I have updated the migration section of
    > > the
    > > > > KIP.
    > > > >     > Can
    > > > >     > > > you
    > > > >     > > >     > please
    > > > >     > > >     >     > take a look?
    > > > >     > > >     >     >
    > > > >     > > >     >     >         Thanks,
    > > > >     > > >     >     >
    > > > >     > > >     >     >         Mayuresh
    > > > >     > > >     >     >
    > > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM,
    > Mayuresh
    > > > > Gharat <
    > > > >     > > >     >     > gharatmayuresh15@gmail.com
    > > > >     > > >     >     >         > wrote:
    > > > >     > > >     >     >
    > > > >     > > >     >     >         > Hi Michael,
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         > That whilst sending tombstone and non
    > > > null
    > > > > value,
    > > > >     > > the
    > > > >     > > >     > consumer
    > > > >     > > >     >     > can expect
    > > > >     > > >     >     >         > only to receive the non-null message
    > > only
    > > > > in step
    > > > >     > > > (3) is
    > > > >     > > >     > this
    > > > >     > > >     >     > correct?
    > > > >     > > >     >     >         > ---> I do agree with you here.
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         > Becket, Ismael : can you guys review
    > > the
    > > > >     > migration
    > > > >     > > > plan
    > > > >     > > >     > listed
    > > > >     > > >     >     > above using
    > > > >     > > >     >     >         > magic byte?
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         > Thanks,
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         > Mayuresh
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
    > > Michael
    > > > > Pearce <
    > > > >     > > >     >     > Michael.Pearce@ig.com>
    > > > >     > > >     >     >         > wrote:
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         >> Many thanks for this Mayuresh. I
    > don't
    > > > > have any
    > > > >     > > >     > objections.
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> I assume we should state:
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> That whilst sending tombstone and
    > non
    > > > null
    > > > >     > value,
    > > > >     > > > the
    > > > >     > > >     > consumer
    > > > >     > > >     >     > can expect
    > > > >     > > >     >     >         >> only to receive the non-null message
    > > > only
    > > > > in
    > > > >     > step
    > > > >     > > > (3) is
    > > > >     > > >     > this
    > > > >     > > >     >     > correct?
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Cheers
    > > > >     > > >     >     >         >> Mike
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Sent using OWA for iPhone
    > > > >     > > >     >     >         >> ______________________________
    > > > __________
    > > > >     > > >     >     >         >> From: Mayuresh Gharat <
    > > > >     > gharatmayuresh15@gmail.com
    > > > >     > > >
    > > > >     > > >     >     >         >> Sent: Thursday, November 17, 2016
    > > > 5:18:41
    > > > > PM
    > > > >     > > >     >     >         >> To: dev@kafka.apache.org
    > > > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
    > > > > Compaction
    > > > >     > > > Tombstone
    > > > >     > > >     > Flag
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Hi Ismael,
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Thanks for the explanation.
    > > > >     > > >     >     >         >> Specially I like this part where in
    > > you
    > > > >     > mentioned
    > > > >     > > > we can
    > > > >     > > >     > get
    > > > >     > > >     >     > rid of the
    > > > >     > > >     >     >         >> older null value support for log
    > > > > compaction
    > > > >     > later
    > > > >     > > > on,
    > > > >     > > >     > here :
    > > > >     > > >     >     >         >> We can't change semantics of the
    > > message
    > > > > format
    > > > >     > > > without
    > > > >     > > >     > having
    > > > >     > > >     >     > a long
    > > > >     > > >     >     >         >> transition period. And we can't rely
    > > > >     > > >     >     >         >> on people reading documentation or
    > > > acting
    > > > > on a
    > > > >     > > > warning for
    > > > >     > > >     >     > something so
    > > > >     > > >     >     >         >> fundamental. As such, my take is
    > that
    > > we
    > > > > need to
    > > > >     > > > bump the
    > > > >     > > >     > magic
    > > > >     > > >     >     > byte. The
    > > > >     > > >     >     >         >> good news is
    > > > >     > > >     >     >         >> that we don't have to support all
    > > > versions
    > > > >     > > forever.
    > > > >     > > > We
    > > > >     > > >     > have
    > > > >     > > >     >     > said that we
    > > > >     > > >     >     >         >> will support direct upgrades for 2
    > > > years.
    > > > > That
    > > > >     > > > means that
    > > > >     > > >     >     > message format
    > > > >     > > >     >     >         >> version n could, in theory, be
    > > removed 2
    > > > > years
    > > > >     > > > after the
    > > > >     > > >     > it's
    > > > >     > > >     >     > introduced.
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Just a heads up, I would like to
    > > mention
    > > > > that
    > > > >     > even
    > > > >     > > > without
    > > > >     > > >     >     > bumping magic
    > > > >     > > >     >     >         >> byte, we will *NOT* loose zero copy
    > as
    > > > in
    > > > > the
    > > > >     > > > client(x+1)
    > > > >     > > >     > in my
    > > > >     > > >     >     >         >> explanation
    > > > >     > > >     >     >         >> above will convert internally a null
    > > > > value to
    > > > >     > > have a
    > > > >     > > >     > tombstone
    > > > >     > > >     >     > bit set and
    > > > >     > > >     >     >         >> a tombstone bit set to have a null
    > > value
    > > > >     > > > automatically
    > > > >     > > >     >     > internally and by
    > > > >     > > >     >     >         >> the time we move to version (x+2),
    > the
    > > > > clients
    > > > >     > > > would have
    > > > >     > > >     >     > upgraded.
    > > > >     > > >     >     >         >> Obviously if we support a request
    > from
    > > > >     > > consumer(x),
    > > > >     > > > we
    > > > >     > > >     > will
    > > > >     > > >     >     > loose zero
    > > > >     > > >     >     >         >> copy
    > > > >     > > >     >     >         >> but that is the same case with magic
    > > > byte.
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> But if magic byte bump makes life
    > > easier
    > > > > for
    > > > >     > > > transition
    > > > >     > > >     > for the
    > > > >     > > >     >     > above
    > > > >     > > >     >     >         >> reasons that you explained, I am OK
    > > with
    > > > > it
    > > > >     > since
    > > > >     > > > we are
    > > > >     > > >     > going
    > > > >     > > >     >     > to meet the
    > > > >     > > >     >     >         >> end goal down the road :)
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> On a side note can we update the doc
    > > > here
    > > > > on
    > > > >     > magic
    > > > >     > > > byte
    > > > >     > > >     > to say
    > > > >     > > >     >     > that "*it
    > > > >     > > >     >     >         >> should be bumped whenever the
    > message
    > > > > format is
    > > > >     > > > changed
    > > > >     > > >     > or the
    > > > >     > > >     >     >         >> interpretation of message format
    > > (usage
    > > > > of the
    > > > >     > > > reserved
    > > > >     > > >     > bits as
    > > > >     > > >     >     > well) is
    > > > >     > > >     >     >         >> changed*".
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Hi Michael,
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Here is the update plan that we
    > > > discussed
    > > > >     > offline
    > > > >     > > >     > yesterday :
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Currently the magic-byte which
    > > > > corresponds to
    > > > >     > the
    > > > >     > > >     >     > "message.format.version"
    > > > >     > > >     >     >         >> is set to 1.
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> 1) On broker it will be set to 1
    > > > > initially.
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> 2) When a producer client sends a
    > > > message
    > > > > with
    > > > >     > > > magic-byte
    > > > >     > > >     > = 2,
    > > > >     > > >     >     > since the
    > > > >     > > >     >     >         >> broker is on magic-byte = 1, we will
    > > > down
    > > > >     > convert
    > > > >     > > > it,
    > > > >     > > >     > which
    > > > >     > > >     >     > means if the
    > > > >     > > >     >     >         >> tombstone bit is set, the value will
    > > be
    > > > > set to
    > > > >     > > > null. A
    > > > >     > > >     > consumer
    > > > >     > > >     >     >         >> understanding magic-byte = 1, will
    > > still
    > > > > work
    > > > >     > with
    > > > >     > > > this. A
    > > > >     > > >     >     > consumer
    > > > >     > > >     >     >         >> working
    > > > >     > > >     >     >         >> with magic-byte =2 will also be able
    > > to
    > > > >     > understand
    > > > >     > > > this,
    > > > >     > > >     > since
    > > > >     > > >     >     > it
    > > > >     > > >     >     >         >> understands the tombstone.
    > > > >     > > >     >     >         >> Now there is still the question of
    > > > > supporting a
    > > > >     > > >     > non-tombstone
    > > > >     > > >     >     > and null
    > > > >     > > >     >     >         >> value from producer client with
    > > > > magic-byte = 2.*
    > > > >     > > (I
    > > > >     > > > am
    > > > >     > > >     > not sure
    > > > >     > > >     >     > if we
    > > > >     > > >     >     >         >> should support this. Ismael/Becket
    > can
    > > > > comment
    > > > >     > > > here)*
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> 3) When almost all the clients have
    > > > > upgraded,
    > > > >     > the
    > > > >     > > >     >     > message.format.version
    > > > >     > > >     >     >         >> on
    > > > >     > > >     >     >         >> the broker can be changed to 2,
    > where
    > > in
    > > > > the
    > > > >     > down
    > > > >     > > >     > conversion in
    > > > >     > > >     >     > the above
    > > > >     > > >     >     >         >> step will not happen. If at this
    > point
    > > > we
    > > > > get a
    > > > >     > > > consumer
    > > > >     > > >     >     > request from a
    > > > >     > > >     >     >         >> older consumer, we might have to
    > down
    > > > > convert
    > > > >     > > where
    > > > >     > > > in we
    > > > >     > > >     > loose
    > > > >     > > >     >     > zero copy,
    > > > >     > > >     >     >         >> but these cases should be rare.
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Becket can you review this plan and
    > > add
    > > > > more
    > > > >     > > > details if I
    > > > >     > > >     > have
    > > > >     > > >     >     >         >> missed/wronged something, before we
    > > put
    > > > > it on
    > > > >     > KIP.
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Thanks,
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> Mayuresh
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
    > > > Michael
    > > > >     > Pearce <
    > > > >     > > >     >     > Michael.Pearce@ig.com>
    > > > >     > > >     >     >         >> wrote:
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> > Thanks guys, for discussing this
    > > > > offline and
    > > > >     > > > getting
    > > > >     > > >     > some
    > > > >     > > >     >     > consensus.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > So its clear for myself and others
    > > > what
    > > > > is
    > > > >     > > > proposed now
    > > > >     > > >     > (i
    > > > >     > > >     >     > think i
    > > > >     > > >     >     >         >> > understand, but want to make sure)
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > Could i ask either directly update
    > > the
    > > > > kip to
    > > > >     > > > detail the
    > > > >     > > >     >     > migration
    > > > >     > > >     >     >         >> > strategy, or (re-)state your
    > offline
    > > > > discussed
    > > > >     > > and
    > > > >     > > >     > agreed
    > > > >     > > >     >     > migration
    > > > >     > > >     >     >         >> > strategy based on a magic byte is
    > in
    > > > > this
    > > > >     > > thread.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > The main original driver for the
    > KIP
    > > > > was to
    > > > >     > > > support
    > > > >     > > >     >     > compaction where
    > > > >     > > >     >     >         >> value
    > > > >     > > >     >     >         >> > isn't null, based off the
    > > discussions
    > > > on
    > > > >     > KIP-82
    > > > >     > > > thread.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > We should be able to support
    > > > > non-tombstone +
    > > > >     > > null
    > > > >     > > > value
    > > > >     > > >     > by the
    > > > >     > > >     >     >         >> completion
    > > > >     > > >     >     >         >> > of the KIP, as we noted when
    > > > discussing
    > > > > this
    > > > >     > > kip,
    > > > >     > > > having
    > > > >     > > >     >     > logic based on
    > > > >     > > >     >     >         >> a
    > > > >     > > >     >     >         >> > null value isn't very clean and
    > also
    > > > > separates
    > > > >     > > the
    > > > >     > > >     > concerns.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > As discussed already though we can
    > > > > split this
    > > > >     > > into
    > > > >     > > >     > KIP-87a
    > > > >     > > >     >     > and KIP-87b
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > Where we look to deliver KIP-87a
    > on
    > > a
    > > > >     > compacted
    > > > >     > > > topic
    > > > >     > > >     > (to
    > > > >     > > >     >     > address the
    > > > >     > > >     >     >         >> > immediate issues)
    > > > >     > > >     >     >         >> > * tombstone + null value
    > > > >     > > >     >     >         >> > * tombstone + non-null value
    > > > >     > > >     >     >         >> > * non-tombstone + non-null value
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > Then we can discuss once KIP-87a
    > is
    > > > > completed
    > > > >     > > > options
    > > > >     > > >     > later
    > > > >     > > >     >     > and how we
    > > > >     > > >     >     >         >> > support the second part KIP-87b to
    > > > > deliver:
    > > > >     > > >     >     >         >> > * non-tombstone + null value
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > Cheers
    > > > >     > > >     >     >         >> > Mike
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > ______________________________
    > > > > __________
    > > > >     > > >     >     >         >> > From: Becket Qin <
    > > > becket.qin@gmail.com>
    > > > >     > > >     >     >         >> > Sent: Thursday, November 17, 2016
    > > 1:43
    > > > > AM
    > > > >     > > >     >     >         >> > To: dev@kafka.apache.org
    > > > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 -
    > Add
    > > > > Compaction
    > > > >     > > >     > Tombstone Flag
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > Renu, Mayuresh and I had an
    > offline
    > > > >     > discussion,
    > > > >     > > > and
    > > > >     > > >     > following
    > > > >     > > >     >     > is a brief
    > > > >     > > >     >     >         >> > summary.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > 1. We agreed that not bumping up
    > > magic
    > > > > value
    > > > >     > may
    > > > >     > > > result
    > > > >     > > >     > in
    > > > >     > > >     >     > losing zero
    > > > >     > > >     >     >         >> copy
    > > > >     > > >     >     >         >> > during migration.
    > > > >     > > >     >     >         >> > 2. Given that bumping up magic
    > value
    > > > is
    > > > > almost
    > > > >     > > > free and
    > > > >     > > >     > has
    > > > >     > > >     >     > benefit of
    > > > >     > > >     >     >         >> > avoiding potential performance
    > > issue.
    > > > > It is
    > > > >     > > > probably
    > > > >     > > >     > worth
    > > > >     > > >     >     > doing.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > One issue we still need to think
    > > about
    > > > > is
    > > > >     > > whether
    > > > >     > > > we
    > > > >     > > >     > want to
    > > > >     > > >     >     > support a
    > > > >     > > >     >     >         >> > non-tombstone message with null
    > > value.
    > > > >     > > >     >     >         >> > Currently it is not supported by
    > > > Kafka.
    > > > > If we
    > > > >     > > > allow a
    > > > >     > > >     >     > non-tombstone null
    > > > >     > > >     >     >         >> > value message to exist after
    > KIP-87.
    > > > The
    > > > >     > problem
    > > > >     > > > is
    > > > >     > > >     > that such
    > > > >     > > >     >     > message
    > > > >     > > >     >     >         >> will
    > > > >     > > >     >     >         >> > not be supported by the consumers
    > > > prior
    > > > > to
    > > > >     > > KIP-87.
    > > > >     > > >     > Because a
    > > > >     > > >     >     > null value
    > > > >     > > >     >     >         >> > will always be interpreted to a
    > > > > tombstone.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > One option is that we keep the
    > > current
    > > > > way,
    > > > >     > i.e.
    > > > >     > > > do not
    > > > >     > > >     >     > support such
    > > > >     > > >     >     >         >> > message. It would be good to know
    > if
    > > > > there is
    > > > >     > a
    > > > >     > > >     > concrete use
    > > > >     > > >     >     > case for
    > > > >     > > >     >     >         >> such
    > > > >     > > >     >     >         >> > message. If there is not, we can
    > > > > probably just
    > > > >     > > not
    > > > >     > > >     > support it.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > Thanks,
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > JIangjie (Becket) Qin
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
    > > > > Mayuresh
    > > > >     > > Gharat <
    > > > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
    > > > >     > > >     >     >         >> > > wrote:
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >> > > Hi Ismael,
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > > This is something I can think of
    > > for
    > > > >     > migration
    > > > >     > > > plan:
    > > > >     > > >     >     >         >> > > So the migration plan can look
    > > > > something
    > > > >     > like
    > > > >     > > > this,
    > > > >     > > >     > with up
    > > > >     > > >     >     >         >> conversion :
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > > 1) Currently lets say we have
    > > Broker
    > > > > at
    > > > >     > > version
    > > > >     > > > x.
    > > > >     > > >     >     >         >> > > 2) Currently we have clients at
    > > > > version x.
    > > > >     > > >     >     >         >> > > 3) a) We move the version to
    > > > > Broker(x+1) :
    > > > >     > > > supports
    > > > >     > > >     > both
    > > > >     > > >     >     > tombstone and
    > > > >     > > >     >     >         >> > null
    > > > >     > > >     >     >         >> > > for log compaction.
    > > > >     > > >     >     >         >> > >     b) We upgrade the client to
    > > > > version
    > > > >     > > > client(x+1) :
    > > > >     > > >     > if in
    > > > >     > > >     >     > the
    > > > >     > > >     >     >         >> producer
    > > > >     > > >     >     >         >> > > client(x+1) the value is set to
    > > > null,
    > > > > we
    > > > >     > will
    > > > >     > > >     > automatically
    > > > >     > > >     >     > set the
    > > > >     > > >     >     >         >> > > Tombstone bit internally. If the
    > > > > producer
    > > > >     > > > client(x+1)
    > > > >     > > >     > sets
    > > > >     > > >     >     > the
    > > > >     > > >     >     >         >> tombstone
    > > > >     > > >     >     >         >> > > itself, well and good. For
    > > producer
    > > > >     > client(x),
    > > > >     > > > the
    > > > >     > > >     > broker
    > > > >     > > >     >     > will up
    > > > >     > > >     >     >         >> convert
    > > > >     > > >     >     >         >> > > to have the tombstone bit.
    > > > > Broker(x+1) is
    > > > >     > > > supporting
    > > > >     > > >     > both.
    > > > >     > > >     >     > Consumer
    > > > >     > > >     >     >         >> > > client(x+1) will be aware of
    > this
    > > > and
    > > > > should
    > > > >     > > be
    > > > >     > > > able
    > > > >     > > >     > to
    > > > >     > > >     >     > handle this.
    > > > >     > > >     >     >         >> For
    > > > >     > > >     >     >         >> > > consumer client(x) we will down
    > > > > convert the
    > > > >     > > > message
    > > > >     > > >     > on the
    > > > >     > > >     >     > broker
    > > > >     > > >     >     >         >> side.
    > > > >     > > >     >     >         >> > >     c) At this point we will
    > have
    > > to
    > > > >     > specify a
    > > > >     > > >     > warning or
    > > > >     > > >     >     > clearly
    > > > >     > > >     >     >         >> specify
    > > > >     > > >     >     >         >> > > in docs that this behavior is
    > > about
    > > > > to be
    > > > >     > > > changed for
    > > > >     > > >     > log
    > > > >     > > >     >     > compaction.
    > > > >     > > >     >     >         >> > > 4) a) In next release of the
    > > > > Broker(x+2), we
    > > > >     > > > say that
    > > > >     > > >     > only
    > > > >     > > >     >     > Tombstone
    > > > >     > > >     >     >         >> is
    > > > >     > > >     >     >         >> > > used for log compaction on the
    > > > Broker
    > > > > side.
    > > > >     > > >     > Clients(x+1)
    > > > >     > > >     >     > still is
    > > > >     > > >     >     >         >> > > supported.
    > > > >     > > >     >     >         >> > >     b) We upgrade the client to
    > > > > version
    > > > >     > > > client(x+2) :
    > > > >     > > >     > if
    > > > >     > > >     >     > value is set
    > > > >     > > >     >     >         >> to
    > > > >     > > >     >     >         >> > > null, tombstone will not be set
    > > > >     > automatically.
    > > > >     > > > The
    > > > >     > > >     > client
    > > > >     > > >     >     > will have to
    > > > >     > > >     >     >         >> > call
    > > > >     > > >     >     >         >> > > setTombstone() to actually set
    > the
    > > > >     > tombstone.
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > > We should compare this migration
    > > > plan
    > > > > with
    > > > >     > the
    > > > >     > > >     > migration
    > > > >     > > >     >     > plan for
    > > > >     > > >     >     >         >> magic
    > > > >     > > >     >     >         >> > > byte bump and do whatever looks
    > > > good.
    > > > >     > > >     >     >         >> > > I am just worried that if we go
    > > down
    > > > > magic
    > > > >     > > byte
    > > > >     > > > route,
    > > > >     > > >     >     > unless I am
    > > > >     > > >     >     >         >> > missing
    > > > >     > > >     >     >         >> > > something, it sounds like kafka
    > > will
    > > > > be
    > > > >     > stuck
    > > > >     > > > with
    > > > >     > > >     >     > supporting both
    > > > >     > > >     >     >         >> null
    > > > >     > > >     >     >         >> > > value and tombstone bit for log
    > > > > compaction
    > > > >     > for
    > > > >     > > > life
    > > > >     > > >     > long,
    > > > >     > > >     >     > which does
    > > > >     > > >     >     >         >> not
    > > > >     > > >     >     >         >> > > look like a good end state.
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > > Thanks,
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > > Mayuresh
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
    > > > > Mayuresh
    > > > >     > > > Gharat <
    > > > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
    > > > >     > > >     >     >         >> > > > wrote:
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > > > Hi Ismael,
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > That's a very good point
    > which I
    > > > > might
    > > > >     > have
    > > > >     > > > not
    > > > >     > > >     >     > considered earlier.
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > Here is a plan that I can
    > think
    > > > of:
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > Stage 1) The broker from now
    > on,
    > > > up
    > > > >     > converts
    > > > >     > > > the
    > > > >     > > >     > message
    > > > >     > > >     >     > to have the
    > > > >     > > >     >     >         >> > > > tombstone marker. The log
    > > > compaction
    > > > >     > thread
    > > > >     > > > does log
    > > > >     > > >     >     > compaction
    > > > >     > > >     >     >         >> based
    > > > >     > > >     >     >         >> > on
    > > > >     > > >     >     >         >> > > > both null and tombstone
    > marker.
    > > > > This is
    > > > >     > our
    > > > >     > > >     > transition
    > > > >     > > >     >     > period.
    > > > >     > > >     >     >         >> > > > Stage 2) The next release we
    > > only
    > > > > say that
    > > > >     > > log
    > > > >     > > >     > compaction
    > > > >     > > >     >     > is based
    > > > >     > > >     >     >         >> on
    > > > >     > > >     >     >         >> > > > tombstone marker. (Open source
    > > > > kafka makes
    > > > >     > > > this as a
    > > > >     > > >     >     > policy). By
    > > > >     > > >     >     >         >> this
    > > > >     > > >     >     >         >> > > time,
    > > > >     > > >     >     >         >> > > > the organization which is
    > moving
    > > > to
    > > > > this
    > > > >     > > > release
    > > > >     > > >     > will be
    > > > >     > > >     >     > sure that
    > > > >     > > >     >     >         >> they
    > > > >     > > >     >     >         >> > > > have gone through the entire
    > > > > transition
    > > > >     > > > period.
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > My only goal of doing this is
    > > that
    > > > > Kafka
    > > > >     > > > clearly
    > > > >     > > >     >     > specifies the end
    > > > >     > > >     >     >         >> > state
    > > > >     > > >     >     >         >> > > > about what log compaction
    > means
    > > > (is
    > > > > it
    > > > >     > null
    > > > >     > > > value
    > > > >     > > >     > or a
    > > > >     > > >     >     > tombstone
    > > > >     > > >     >     >         >> > marker,
    > > > >     > > >     >     >         >> > > > but not both).
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > What do you think?
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > Thanks,
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > Mayuresh
    > > > >     > > >     >     >         >> > > > .
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17
    > AM,
    > > > > Ismael
    > > > >     > > Juma <
    > > > >     > > >     >     > ismael@juma.me.uk>
    > > > >     > > >     >     >         >> > wrote:
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > >> One comment below.
    > > > >     > > >     >     >         >> > > >>
    > > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08
    > > PM,
    > > > > Mayuresh
    > > > >     > > > Gharat <
    > > > >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
    > > > >     > > >     >     >         >> > > >> > wrote:
    > > > >     > > >     >     >         >> > > >>
    > > > >     > > >     >     >         >> > > >> >    - If we don't bump up
    > the
    > > > > magic
    > > > >     > byte,
    > > > >     > > > on the
    > > > >     > > >     > broker
    > > > >     > > >     >     > side, the
    > > > >     > > >     >     >         >> > > broker
    > > > >     > > >     >     >         >> > > >> >    will always have to look
    > > at
    > > > > both
    > > > >     > > > tombstone
    > > > >     > > >     > bit and
    > > > >     > > >     >     > the value
    > > > >     > > >     >     >         >> when
    > > > >     > > >     >     >         >> > > do
    > > > >     > > >     >     >         >> > > >> the
    > > > >     > > >     >     >         >> > > >> >    compaction. Assuming we
    > do
    > > > > not bump
    > > > >     > up
    > > > >     > > > the
    > > > >     > > >     > magic
    > > > >     > > >     >     > byte,
    > > > >     > > >     >     >         >> > > >> >    imagine the broker sees
    > a
    > > > > message
    > > > >     > > which
    > > > >     > > > does
    > > > >     > > >     > not
    > > > >     > > >     >     > have a
    > > > >     > > >     >     >         >> tombstone
    > > > >     > > >     >     >         >> > > bit
    > > > >     > > >     >     >         >> > > >> >    set. The broker does not
    > > > know
    > > > > when
    > > > >     > the
    > > > >     > > >     > message was
    > > > >     > > >     >     > produced
    > > > >     > > >     >     >         >> (i.e.
    > > > >     > > >     >     >         >> > > >> > whether
    > > > >     > > >     >     >         >> > > >> >    the message has been up
    > > > > converted or
    > > > >     > > > not), it
    > > > >     > > >     > has
    > > > >     > > >     >     > to take a
    > > > >     > > >     >     >         >> > further
    > > > >     > > >     >     >         >> > > >> > look at
    > > > >     > > >     >     >         >> > > >> >    the value to see if it
    > is
    > > > > null or
    > > > >     > not
    > > > >     > > in
    > > > >     > > >     > order to
    > > > >     > > >     >     > determine
    > > > >     > > >     >     >         >> if it
    > > > >     > > >     >     >         >> > > is
    > > > >     > > >     >     >         >> > > >> a
    > > > >     > > >     >     >         >> > > >> >    tombstone. The same
    > logic
    > > > has
    > > > > to be
    > > > >     > > put
    > > > >     > > > on the
    > > > >     > > >     >     > consumer as
    > > > >     > > >     >     >         >> well
    > > > >     > > >     >     >         >> > > >> because
    > > > >     > > >     >     >         >> > > >> > the
    > > > >     > > >     >     >         >> > > >> >    consumer does not know
    > if
    > > > the
    > > > >     > message
    > > > >     > > > has
    > > > >     > > >     > been up
    > > > >     > > >     >     > converted or
    > > > >     > > >     >     >         >> > not.
    > > > >     > > >     >     >         >> > > >> >       - If we upconvert
    > while
    > > > >     > appending,
    > > > >     > > > this is
    > > > >     > > >     > not
    > > > >     > > >     >     > the case,
    > > > >     > > >     >     >         >> > right?
    > > > >     > > >     >     >         >> > > >>
    > > > >     > > >     >     >         >> > > >>
    > > > >     > > >     >     >         >> > > >> If I understand you
    > correctly,
    > > > > this is
    > > > >     > not
    > > > >     > > >     > sufficient
    > > > >     > > >     >     > because the
    > > > >     > > >     >     >         >> log
    > > > >     > > >     >     >         >> > > may
    > > > >     > > >     >     >         >> > > >> have messages appended before
    > > it
    > > > > was
    > > > >     > > > upgraded to
    > > > >     > > >     > include
    > > > >     > > >     >     > KIP-87.
    > > > >     > > >     >     >         >> > > >>
    > > > >     > > >     >     >         >> > > >> Ismael
    > > > >     > > >     >     >         >> > > >>
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > > > --
    > > > >     > > >     >     >         >> > > > -Regards,
    > > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
    > > > >     > > >     >     >         >> > > > (862) 250-7125
    > > > >     > > >     >     >         >> > > >
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > > --
    > > > >     > > >     >     >         >> > > -Regards,
    > > > >     > > >     >     >         >> > > Mayuresh R. Gharat
    > > > >     > > >     >     >         >> > > (862) 250-7125
    > > > >     > > >     >     >         >> > >
    > > > >     > > >     >     >         >> > The information contained in this
    > > > email
    > > > > is
    > > > >     > > > strictly
    > > > >     > > >     >     > confidential and for
    > > > >     > > >     >     >         >> > the use of the addressee only,
    > > unless
    > > > >     > otherwise
    > > > >     > > >     > indicated. If
    > > > >     > > >     >     > you are
    > > > >     > > >     >     >         >> not
    > > > >     > > >     >     >         >> > the intended recipient, please do
    > > not
    > > > > read,
    > > > >     > > copy,
    > > > >     > > > use or
    > > > >     > > >     >     > disclose to
    > > > >     > > >     >     >         >> others
    > > > >     > > >     >     >         >> > this message or any attachment.
    > > Please
    > > > > also
    > > > >     > > > notify the
    > > > >     > > >     > sender
    > > > >     > > >     >     > by
    > > > >     > > >     >     >         >> replying
    > > > >     > > >     >     >         >> > to this email or by telephone
    > > (+44(020
    > > > > 7896
    > > > >     > > 0011)
    > > > >     > > > and
    > > > >     > > >     > then
    > > > >     > > >     >     > delete the
    > > > >     > > >     >     >         >> email
    > > > >     > > >     >     >         >> > and any copies of it. Opinions,
    > > > > conclusion
    > > > >     > (etc)
    > > > >     > > > that
    > > > >     > > >     > do not
    > > > >     > > >     >     > relate to
    > > > >     > > >     >     >         >> the
    > > > >     > > >     >     >         >> > official business of this company
    > > > shall
    > > > > be
    > > > >     > > > understood as
    > > > >     > > >     >     > neither given
    > > > >     > > >     >     >         >> nor
    > > > >     > > >     >     >         >> > endorsed by it. IG is a trading
    > name
    > > > of
    > > > > IG
    > > > >     > > Markets
    > > > >     > > >     > Limited (a
    > > > >     > > >     >     > company
    > > > >     > > >     >     >         >> > registered in England and Wales,
    > > > company
    > > > >     > number
    > > > >     > > >     > 04008957) and
    > > > >     > > >     >     > IG Index
    > > > >     > > >     >     >         >> > Limited (a company registered in
    > > > > England and
    > > > >     > > > Wales,
    > > > >     > > >     > company
    > > > >     > > >     >     > number
    > > > >     > > >     >     >         >> > 01190902). Registered address at
    > > > Cannon
    > > > > Bridge
    > > > >     > > > House, 25
    > > > >     > > >     >     > Dowgate Hill,
    > > > >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
    > > > Limited
    > > > >     > > (register
    > > > >     > > >     > number
    > > > >     > > >     >     > 195355) and IG
    > > > >     > > >     >     >         >> > Index Limited (register number
    > > 114059)
    > > > > are
    > > > >     > > > authorised
    > > > >     > > >     > and
    > > > >     > > >     >     > regulated by
    > > > >     > > >     >     >         >> the
    > > > >     > > >     >     >         >> > Financial Conduct Authority.
    > > > >     > > >     >     >         >> >
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >> --
    > > > >     > > >     >     >         >> -Regards,
    > > > >     > > >     >     >         >> Mayuresh R. Gharat
    > > > >     > > >     >     >         >> (862) 250-7125
    > > > >     > > >     >     >         >> The information contained in this
    > > email
    > > > is
    > > > >     > > strictly
    > > > >     > > >     >     > confidential and for
    > > > >     > > >     >     >         >> the use of the addressee only,
    > unless
    > > > > otherwise
    > > > >     > > >     > indicated. If
    > > > >     > > >     >     > you are not
    > > > >     > > >     >     >         >> the intended recipient, please do
    > not
    > > > > read,
    > > > >     > copy,
    > > > >     > > > use or
    > > > >     > > >     >     > disclose to others
    > > > >     > > >     >     >         >> this message or any attachment.
    > Please
    > > > > also
    > > > >     > notify
    > > > >     > > > the
    > > > >     > > >     > sender
    > > > >     > > >     >     > by replying
    > > > >     > > >     >     >         >> to this email or by telephone
    > (+44(020
    > > > > 7896
    > > > >     > 0011)
    > > > >     > > > and then
    > > > >     > > >     >     > delete the email
    > > > >     > > >     >     >         >> and any copies of it. Opinions,
    > > > > conclusion (etc)
    > > > >     > > > that do
    > > > >     > > >     > not
    > > > >     > > >     >     > relate to the
    > > > >     > > >     >     >         >> official business of this company
    > > shall
    > > > be
    > > > >     > > > understood as
    > > > >     > > >     >     > neither given nor
    > > > >     > > >     >     >         >> endorsed by it. IG is a trading name
    > > of
    > > > IG
    > > > >     > Markets
    > > > >     > > >     > Limited (a
    > > > >     > > >     >     > company
    > > > >     > > >     >     >         >> registered in England and Wales,
    > > company
    > > > > number
    > > > >     > > > 04008957)
    > > > >     > > >     > and
    > > > >     > > >     >     > IG Index
    > > > >     > > >     >     >         >> Limited (a company registered in
    > > England
    > > > > and
    > > > >     > > Wales,
    > > > >     > > >     > company
    > > > >     > > >     >     > number
    > > > >     > > >     >     >         >> 01190902). Registered address at
    > > Cannon
    > > > > Bridge
    > > > >     > > > House, 25
    > > > >     > > >     >     > Dowgate Hill,
    > > > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
    > > Limited
    > > > >     > (register
    > > > >     > > > number
    > > > >     > > >     >     > 195355) and IG
    > > > >     > > >     >     >         >> Index Limited (register number
    > 114059)
    > > > are
    > > > >     > > > authorised and
    > > > >     > > >     >     > regulated by the
    > > > >     > > >     >     >         >> Financial Conduct Authority.
    > > > >     > > >     >     >         >>
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >         > --
    > > > >     > > >     >     >         > -Regards,
    > > > >     > > >     >     >         > Mayuresh R. Gharat
    > > > >     > > >     >     >         > (862) 250-7125
    > > > >     > > >     >     >         >
    > > > >     > > >     >     >
    > > > >     > > >     >     >
    > > > >     > > >     >     >
    > > > >     > > >     >     >         --
    > > > >     > > >     >     >         -Regards,
    > > > >     > > >     >     >         Mayuresh R. Gharat
    > > > >     > > >     >     >         (862) 250-7125
    > > > >     > > >     >     >
    > > > >     > > >     >     >
    > > > >     > > >     >     >     The information contained in this email is
    > > > > strictly
    > > > >     > > > confidential
    > > > >     > > >     > and
    > > > >     > > >     >     > for the use of the addressee only, unless
    > > otherwise
    > > > >     > > indicated.
    > > > >     > > > If
    > > > >     > > >     > you are
    > > > >     > > >     >     > not the intended recipient, please do not read,
    > > > > copy, use
    > > > >     > or
    > > > >     > > >     > disclose to
    > > > >     > > >     >     > others this message or any attachment. Please
    > > also
    > > > > notify
    > > > >     > the
    > > > >     > > > sender
    > > > >     > > >     > by
    > > > >     > > >     >     > replying to this email or by telephone (+44(020
    > > > 7896
    > > > > 0011)
    > > > >     > > and
    > > > >     > > > then
    > > > >     > > >     > delete
    > > > >     > > >     >     > the email and any copies of it. Opinions,
    > > > conclusion
    > > > > (etc)
    > > > >     > > > that do
    > > > >     > > >     > not
    > > > >     > > >     >     > relate to the official business of this company
    > > > > shall be
    > > > >     > > > understood
    > > > >     > > >     > as
    > > > >     > > >     >     > neither given nor endorsed by it. IG is a
    > trading
    > > > > name of
    > > > >     > IG
    > > > >     > > > Markets
    > > > >     > > >     >     > Limited (a company registered in England and
    > > Wales,
    > > > > company
    > > > >     > > > number
    > > > >     > > >     >     > 04008957) and IG Index Limited (a company
    > > > registered
    > > > > in
    > > > >     > > > England and
    > > > >     > > >     > Wales,
    > > > >     > > >     >     > company number 01190902). Registered address at
    > > > > Cannon
    > > > >     > Bridge
    > > > >     > > > House,
    > > > >     > > >     > 25
    > > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
    > > > > Limited
    > > > >     > > > (register
    > > > >     > > >     > number
    > > > >     > > >     >     > 195355) and IG Index Limited (register number
    > > > > 114059) are
    > > > >     > > > authorised
    > > > >     > > >     > and
    > > > >     > > >     >     > regulated by the Financial Conduct Authority.
    > > > >     > > >     >     >
    > > > >     > > >     >     >
    > > > >     > > >     >     >
    > > > >     > > >     >
    > > > >     > > >     >
    > > > >     > > >     > The information contained in this email is strictly
    > > > > confidential
    > > > >     > > and
    > > > >     > > > for
    > > > >     > > >     > the use of the addressee only, unless otherwise
    > > > indicated.
    > > > > If you
    > > > >     > > > are not
    > > > >     > > >     > the intended recipient, please do not read, copy, use
    > > or
    > > > > disclose
    > > > >     > > to
    > > > >     > > > others
    > > > >     > > >     > this message or any attachment. Please also notify
    > the
    > > > > sender by
    > > > >     > > > replying
    > > > >     > > >     > to this email or by telephone (+44(020 7896 0011) and
    > > > then
    > > > > delete
    > > > >     > > > the email
    > > > >     > > >     > and any copies of it. Opinions, conclusion (etc) that
    > > do
    > > > > not
    > > > >     > relate
    > > > >     > > > to the
    > > > >     > > >     > official business of this company shall be understood
    > > as
    > > > > neither
    > > > >     > > > given nor
    > > > >     > > >     > endorsed by it. IG is a trading name of IG Markets
    > > > Limited
    > > > > (a
    > > > >     > > company
    > > > >     > > >     > registered in England and Wales, company number
    > > 04008957)
    > > > > and IG
    > > > >     > > > Index
    > > > >     > > >     > Limited (a company registered in England and Wales,
    > > > company
    > > > >     > number
    > > > >     > > >     > 01190902). Registered address at Cannon Bridge House,
    > > 25
    > > > > Dowgate
    > > > >     > > > Hill,
    > > > >     > > >     > London EC4R 2YA. Both IG Markets Limited (register
    > > number
    > > > > 195355)
    > > > >     > > > and IG
    > > > >     > > >     > Index Limited (register number 114059) are authorised
    > > and
    > > > >     > regulated
    > > > >     > > > by the
    > > > >     > > >     > Financial Conduct Authority.
    > > > >     > > >     >
    > > > >     > > >
    > > > >     > > >
    > > > >     > > > The information contained in this email is strictly
    > > > confidential
    > > > > and
    > > > >     > for
    > > > >     > > > the use of the addressee only, unless otherwise indicated.
    > If
    > > > > you are
    > > > >     > not
    > > > >     > > > the intended recipient, please do not read, copy, use or
    > > > > disclose to
    > > > >     > > others
    > > > >     > > > this message or any attachment. Please also notify the
    > sender
    > > > by
    > > > >     > replying
    > > > >     > > > to this email or by telephone (+44(020 7896 0011) and then
    > > > > delete the
    > > > >     > > email
    > > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
    > not
    > > > > relate to
    > > > >     > > the
    > > > >     > > > official business of this company shall be understood as
    > > > neither
    > > > > given
    > > > >     > > nor
    > > > >     > > > endorsed by it. IG is a trading name of IG Markets Limited
    > (a
    > > > > company
    > > > >     > > > registered in England and Wales, company number 04008957)
    > and
    > > > IG
    > > > > Index
    > > > >     > > > Limited (a company registered in England and Wales, company
    > > > > number
    > > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
    > > > Dowgate
    > > > > Hill,
    > > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
    > > > > 195355) and
    > > > >     > IG
    > > > >     > > > Index Limited (register number 114059) are authorised and
    > > > > regulated by
    > > > >     > > the
    > > > >     > > > Financial Conduct Authority.
    > > > >     > > >
    > > > >     > > The information contained in this email is strictly
    > > confidential
    > > > > and for
    > > > >     > > the use of the addressee only, unless otherwise indicated. If
    > > you
    > > > > are not
    > > > >     > > the intended recipient, please do not read, copy, use or
    > > disclose
    > > > > to
    > > > >     > others
    > > > >     > > this message or any attachment. Please also notify the sender
    > > by
    > > > > replying
    > > > >     > > to this email or by telephone (+44(020 7896 0011) and then
    > > delete
    > > > > the
    > > > >     > email
    > > > >     > > and any copies of it. Opinions, conclusion (etc) that do not
    > > > > relate to
    > > > >     > the
    > > > >     > > official business of this company shall be understood as
    > > neither
    > > > > given
    > > > >     > nor
    > > > >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
    > > > > company
    > > > >     > > registered in England and Wales, company number 04008957) and
    > > IG
    > > > > Index
    > > > >     > > Limited (a company registered in England and Wales, company
    > > > number
    > > > >     > > 01190902). Registered address at Cannon Bridge House, 25
    > > Dowgate
    > > > > Hill,
    > > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
    > > 195355)
    > > > > and IG
    > > > >     > > Index Limited (register number 114059) are authorised and
    > > > > regulated by
    > > > >     > the
    > > > >     > > Financial Conduct Authority.
    > > > >     > >
    > > > >     > >
    > > > >     > The information contained in this email is strictly
    > confidential
    > > > and
    > > > > for
    > > > >     > the use of the addressee only, unless otherwise indicated. If
    > you
    > > > > are not
    > > > >     > the intended recipient, please do not read, copy, use or
    > disclose
    > > > to
    > > > > others
    > > > >     > this message or any attachment. Please also notify the sender
    > by
    > > > > replying
    > > > >     > to this email or by telephone (+44(020 7896 0011) and then
    > delete
    > > > > the email
    > > > >     > and any copies of it. Opinions, conclusion (etc) that do not
    > > relate
    > > > > to the
    > > > >     > official business of this company shall be understood as
    > neither
    > > > > given nor
    > > > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
    > > > company
    > > > >     > registered in England and Wales, company number 04008957) and
    > IG
    > > > > Index
    > > > >     > Limited (a company registered in England and Wales, company
    > > number
    > > > >     > 01190902). Registered address at Cannon Bridge House, 25
    > Dowgate
    > > > > Hill,
    > > > >     > London EC4R 2YA. Both IG Markets Limited (register number
    > 195355)
    > > > > and IG
    > > > >     > Index Limited (register number 114059) are authorised and
    > > regulated
    > > > > by the
    > > > >     > Financial Conduct Authority.
    > > > >     >
    > > > >     >
    > > > >
    > > > >
    > > > > The information contained in this email is strictly confidential and
    > > for
    > > > > the use of the addressee only, unless otherwise indicated. If you are
    > > not
    > > > > the intended recipient, please do not read, copy, use or disclose to
    > > > others
    > > > > this message or any attachment. Please also notify the sender by
    > > replying
    > > > > to this email or by telephone (+44(020 7896 0011) and then delete the
    > > > email
    > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
    > to
    > > > the
    > > > > official business of this company shall be understood as neither
    > given
    > > > nor
    > > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > > > registered in England and Wales, company number 04008957) and IG
    > Index
    > > > > Limited (a company registered in England and Wales, company number
    > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > Hill,
    > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    > > IG
    > > > > Index Limited (register number 114059) are authorised and regulated
    > by
    > > > the
    > > > > Financial Conduct Authority.
    > > > >
    > > > The information contained in this email is strictly confidential and
    > for
    > > > the use of the addressee only, unless otherwise indicated. If you are
    > not
    > > > the intended recipient, please do not read, copy, use or disclose to
    > > others
    > > > this message or any attachment. Please also notify the sender by
    > replying
    > > > to this email or by telephone (+44(020 7896 0011) and then delete the
    > > email
    > > > and any copies of it. Opinions, conclusion (etc) that do not relate to
    > > the
    > > > official business of this company shall be understood as neither given
    > > nor
    > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > > registered in England and Wales, company number 04008957) and IG Index
    > > > Limited (a company registered in England and Wales, company number
    > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    > IG
    > > > Index Limited (register number 114059) are authorised and regulated by
    > > the
    > > > Financial Conduct Authority.
    > > >
    > > The information contained in this email is strictly confidential and for
    > > the use of the addressee only, unless otherwise indicated. If you are not
    > > the intended recipient, please do not read, copy, use or disclose to
    > others
    > > this message or any attachment. Please also notify the sender by replying
    > > to this email or by telephone (+44(020 7896 0011) and then delete the
    > email
    > > and any copies of it. Opinions, conclusion (etc) that do not relate to
    > the
    > > official business of this company shall be understood as neither given
    > nor
    > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > registered in England and Wales, company number 04008957) and IG Index
    > > Limited (a company registered in England and Wales, company number
    > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    > > Index Limited (register number 114059) are authorised and regulated by
    > the
    > > Financial Conduct Authority.
    > >
    > The information contained in this email is strictly confidential and for
    > the use of the addressee only, unless otherwise indicated. If you are not
    > the intended recipient, please do not read, copy, use or disclose to others
    > this message or any attachment. Please also notify the sender by replying
    > to this email or by telephone (+44(020 7896 0011) and then delete the email
    > and any copies of it. Opinions, conclusion (etc) that do not relate to the
    > official business of this company shall be understood as neither given nor
    > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > registered in England and Wales, company number 04008957) and IG Index
    > Limited (a company registered in England and Wales, company number
    > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    > Index Limited (register number 114059) are authorised and regulated by the
    > Financial Conduct Authority.
    >
    The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
    

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Ewen

So this is what got discussed in the discussion thread. We wanted to go for supporting all four combinations

Tombstone + non null value
Tombstone + null value
Non tombstone + non null value
Non tombstone + null value

The issues were mostly around compatibility with old consumers.

In the end the consensus was for reasons of making an upgrade/compatibility we would get to the position of handling in this KIP:

Tombstone + null value
Tombstone + non null value
Non tombstone + non null value

Only later once either we can break client compatibility or time expires long enough we can say consumers should no longer be inferring from null value instead the isTombstone flag can we support:

Non tombstone + null value.

Saying that this puts in a lot of the work to enable the last piece.

Also re having the change events of that makes the state aka compacted topic by doing what we're doing would give this to the supported combinations above mentioned.

Ideally I'd love to have ability to support non tombstone + null value, unfortunately this is some of the tech debt in the original compaction design we can't easily undo, without a period of time.

Regards
Mike

Sent using OWA for iPhone
________________________________________
From: Ewen Cheslack-Postava <ew...@confluent.io>
Sent: Sunday, December 11, 2016 11:15:42 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Michael,

It kind of depends on how you want to interpret the tombstone flag. If it's
purely a producer-facing Kafka-level thing that we treat as internal to the
broker and log cleaner once the record is sent, then your approach makes
sense. You're just moving copying the null-indicates-delete behavior of the
old constructor into the tombstone flag.

However, if you want this change to more generally decouple the idea of
deletion and null values, then you are sometimes converting what might be a
completely valid null value that doesn't indicate deletion into a
tombstone. Downstream applications could potentially handle these cases
differently given the separation of deletion from value.

I guess the question is if we want to try to support the latter even for
topics where we have older produce requests. An example where this could
come up is in something like a CDC Connector. If we try to support the
semantic difference, a connector might write changes to Kafka using the
tombstone flag to indicate when a row was truly deleted (vs an update that
sets it to null but still present; this probably makes more sense for CDC
from document stores or extracting single columns). There are various
reasons we might want to maintain the full log and not turn compaction on
(or just use a time-based retention policy), but downstream applications
might care to know the difference between a delete and a null value. In
fact both versions of the same log (compacted and time-retention) could be
useful and I don't think it'll be uncommon to maintain both or use KIP-71
to maintain a hybrid compacted/retention topic.

-Ewen

On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jay,
>
> Why wouldn't that work, the tombstone value is only looked at by the
> broker, on a topic configured for compaction as such is benign on non
> compacted topics. This is as much as sending a null value currently
>
>
> Regards
> Mike
>
>
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Sunday, December 11, 2016 8:58:53 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hey Michael,
>
> I'm not quite sure that works as that would translate ALL null values to
> tombstones, even for non-compacted topics that use null as an acceptable
> value sent by the producer and expected by the consumer.
>
> -Jay
>
> On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Ewen,
> >
> > I think the easiest way to show this is with code.
> >
> > As you can see we keep the existing behaviour for code/binaries calling
> > the pre-existing constructors, whereby if the value is null the tombstone
> > is set to true.
> >
> > Regards
> > Mike
> >
> >
> >
> >     /**
> >      * Creates a record with a specified timestamp to be sent to a
> > specified topic and partition
> >      *
> >      * @param topic The topic the record will be appended to
> >      * @param partition The partition to which the record should be sent
> >      * @param timestamp The timestamp of the record
> >      * @param tombstone if the record should be treated as a tombstone if
> > the topic is compacted
> >      * @param key The key that will be included in the record
> >      * @param value The record contents
> >      */
> >     public ProducerRecord(String topic, Integer partition, Boolean
> > tombstone, Long timestamp, K key, V value) {
> >         if (topic == null)
> >             throw new IllegalArgumentException("Topic cannot be null.");
> >         if (timestamp != null && timestamp < 0)
> >             throw new IllegalArgumentException(
> >                     String.format("Invalid timestamp: %d. Timestamp
> should
> > always be non-negative or null.", timestamp));
> >         if (partition != null && partition < 0)
> >             throw new IllegalArgumentException(
> >                     String.format("Invalid partition: %d. Partition
> number
> > should always be non-negative or null.", partition));
> >         if (!tombstone && value == null){
> >             throw new IllegalArgumentException(
> >                     String.format("Tombstone must be true if null
> value"));
> >         }
> >         this.topic = topic;
> >         this.partition = partition;
> >         this.tombstone = tombstone;
> >         this.key = key;
> >         this.value = value;
> >         this.timestamp = timestamp;
> >     }
> >
> >     public ProducerRecord(String topic, Integer partition, Long
> timestamp,
> > K key, V value) {
> >         this(topic, partition, value == null, timestamp, key, value);
> >     }
> > ________________________________________
> > From: Ewen Cheslack-Postava <ew...@confluent.io>
> > Sent: Sunday, December 11, 2016 5:45 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > It seemed like this was addressed in the migration section, wasn't it?
> The
> > V2 vs V3 requests and the fact that the broker will downgrade the message
> > format (losing zero copy) if you issues a V2 request to a broker using V3
> > format handles compatibility, does it not? The existing consumers won't
> see
> > the extra metadata in the value, but they will get a null instead and
> treat
> > it as a tombstone. Certainly there is a performance impact, but it seemed
> > compatible.
> >
> > I'm worried about this though:
> >
> >
> >    - *NOTE *: With the new version of producer client using
> ProduceRequest
> >    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and null
> > value
> >    should not be allowed as the older version of consumer using
> > FetchRequest
> >    V2 will think of this as a tombstone when its actually not.
> >
> >
> > Unless I'm misunderstanding, this ends up breaking binary compatibility
> for
> > the Java API. It sounds like this suggests that passing a null value to
> the
> > existing ProducerRecord constructors would generate an exception since
> you
> > didn't explicitly enable the tombstone (via whatever new constructor is
> > provided). But that means you can't swap in a newer client jar without
> > recompiling and get the same behavior if your app deletes keys using the
> > current approach because your app will start throwing exceptions. Maybe
> > this is a tradeoff we're ok with, but we've tried pretty hard to avoid
> > breaking compatibility like this so far -- it makes picking up bug fixes
> in
> > the clients harder for users of frameworks that have to pin to earlier
> > default versions for compatibility.
> >
> > But then later the KIP says:
> >
> >
> >    - The old constructors for ProducerRecord without this parameter will
> be
> >    kept but updated so that their default behaviour if setting null value
> > will
> >    be the tombstone will be set to true to keep existing behaviour.
> >
> >
> > So maybe I am misinterpreting something.
> >
> > And just a nit re: motivation section, item 6 would be clearer for a
> union
> > schema with null (or optional schemas in other formats), e.g. [null,
> > string], in which case losing the schema truly is losing information
> > (whereas null is already the only valid value for a pure null schema).
> >
> > -Ewen
> >
> >
> > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Good point this detail is missing in the KIP write up. Ive added this
> > now.
> > >
> > > Essentially simply just upgrading the clients we do not expect any
> client
> > > app code change needed.
> > >
> > > Cheers
> > > Mike
> > > ________________________________________
> > > From: Jay Kreps <ja...@confluent.io>
> > > Sent: Saturday, December 10, 2016 10:51 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > > Michael,
> > >
> > > The compatibility section goes through the migration path, but isn't
> the
> > > bigger compatibility issue with existing apps? There are many (probably
> > > thousands) of apps in production that use this feature and send null to
> > > mean delete. It seems like this would break compatibility with them,
> and
> > > they would have to be rewritten to right?
> > >
> > > -Jay
> > >
> > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > 4) On v3 we honour the tombstone. As such we expect it to be set
> > > correctly
> > > > as per the KIP.
> > > >
> > > > 4.1) why would we want to produce an error when its v3? This is the
> > exact
> > > > purpose to support non-null tombstone’s
> > > > 4.2) again here im unclear on the question on the v3, produce request
> > we
> > > > expect the tombstone flag to be set correctly.
> > > >
> > > > 4.4) compaction only occurs on compacted topics, the bit makes no
> > > > difference and not looked at on un-compacted (time/size based
> > eviction).
> > > >
> > > >
> > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> > > >
> > > >     Hi, Michael,
> > > >
> > > >     4. Then, I think I misunderstood this point. Could you document
> the
> > > >     following points in the wiki?
> > > >     4.1 If producer V3 sets tombstone, but provides a non-null value,
> > > does
> > > > the
> > > >     send() get an error or does the producer automatically set the
> > value
> > > to
> > > >     null?
> > > >     4.2 If producer V3 doesn't set tombstone, but provides a null
> > value,
> > > > does
> > > >     the send() get an error or does the producer automatically sets
> the
> > > >     tombstone?
> > > >     4.3 Does the broker only expect messages that (a) have no
> tombstone
> > > and
> > > >     non-null value; (b) have tombstone and null value and reject the
> > > > messages
> > > >     with an error code otherwise?
> > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on
> > > not?
> > > >
> > > >     Thanks,
> > > >
> > > >     Jun
> > > >
> > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > >     wrote:
> > > >
> > > >     > Not at all.  This only acts on compacted topics just as what
> > occurs
> > > > today
> > > >     >
> > > >     > Sent using OWA for iPhone
> > > >     > ________________________________________
> > > >     > From: Jun Rao <ju...@confluent.io>
> > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > > >     > To: dev@kafka.apache.org
> > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >     >
> > > >     > Hi, Michael,
> > > >     >
> > > >     > 4. Hmm, does that mean the new client library can never send a
> > null
> > > > message
> > > >     > even to a regular topic? This seems like a change of the
> existing
> > > > behavior.
> > > >     >
> > > >     > Thanks,
> > > >     >
> > > >     > Jun
> > > >     >
> > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > >     > wrote:
> > > >     >
> > > >     > > Hi Jun,
> > > >     > >
> > > >     > > Re 4) That's because we expect the tombstone value to be set
> > > > correctly if
> > > >     > > message bit is 2, as such if an older client sends in on old
> > > > message the
> > > >     > > message is upcast and the bit is set correctly. And such no
> > > longer
> > > > need
> > > >     > to
> > > >     > > check the value. Mayuresh can you confirm my thinking and
> > > > understanding
> > > >     > of
> > > >     > > what we've discussed?
> > > >     > >
> > > >     > > The second point I understand what you're getting at now my
> > > > apologies.
> > > >     > Yes
> > > >     > > this makes sense to save on touching the message, if we're
> the
> > > > only kip
> > > >     > > going in, in this release.
> > > >     > >
> > > >     > > Cheers
> > > >     > > Mike
> > > >     > >
> > > >     > > Sent using OWA for iPhone
> > > >     > > ________________________________________
> > > >     > > From: Jun Rao <ju...@confluent.io>
> > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > > >     > > To: dev@kafka.apache.org
> > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >     > >
> > > >     > > Hi, Michael,
> > > >     > >
> > > >     > > 4. Is this updated in the wiki? The text "If the magic byte
> on
> > > > message is
> > > >     > > 2, the broker should use the tombstone bit for log
> compaction."
> > > > doesn't
> > > >     > > seem to have changed.
> > > >     > >
> > > >     > > 2. My point is that if we change the message format just for
> > this
> > > > KIP, we
> > > >     > > should consider whether it's worth optimizing the down
> > conversion
> > > > path
> > > >     > > (i.e., decide whether a conversion is needed by just looking
> at
> > > the
> > > >     > > tombstone bit in the wrapper message) since tombstone will be
> > > used
> > > >     > rarely.
> > > >     > > However, if the message format change here is combined with
> > other
> > > > KIPs,
> > > >     > > then this optimization likely won't be needed. The latter
> > > probably
> > > > makes
> > > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> > > >     > >
> > > >     > > Other than those, +1 from me,
> > > >     > >
> > > >     > > Thanks,
> > > >     > >
> > > >     > > Jun
> > > >     > >
> > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > >     > > wrote:
> > > >     > >
> > > >     > > > Hi Jun
> > > >     > > >
> > > >     > > > do we have your vote on this now?
> > > >     > > >
> > > >     > > > Any other concerns?
> > > >     > > >
> > > >     > > > Cheers
> > > >     > > > Mike
> > > >     > > >
> > > >     > > > Sent using OWA for iPhone
> > > >     > > > ________________________________________
> > > >     > > > From: Michael Pearce <Mi...@ig.com>
> > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > > >     > > > To: dev@kafka.apache.org
> > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >     > > >
> > > >     > > > Hi Jun,
> > > >     > > >
> > > >     > > > Have updated. Thanks again for the feedback.
> > > >     > > >
> > > >     > > > Agree yes we should align up when it gets to that, I assume
> > > > you’ve
> > > >     > > flagged
> > > >     > > > the same in the other KIP?
> > > >     > > >
> > > >     > > > I think for now let’s get this KIP approved, then we can
> > start
> > > > the work
> > > >     > > to
> > > >     > > > get an initial PR, then we can discuss how to align the two
> > if
> > > > needed.
> > > >     > > >
> > > >     > > > Cheers,
> > > >     > > > Mike
> > > >     > > >
> > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> > > >     > > >
> > > >     > > >     Hi, Michael,
> > > >     > > >
> > > >     > > >     For 2), this is fine. Could you update the KIP wiki to
> > make
> > > > this
> > > >     > and
> > > >     > > > other
> > > >     > > >     points clearer? Other than that, the KIP looks good to
> > me.
> > > >     > > >
> > > >     > > >     An orthogonal thing is that there are other KIPs such
> as
> > > > KIP-98
> > > >     > that
> > > >     > > > also
> > > >     > > >     intend to change the message format. If they all get
> > > > approved, we
> > > >     > > > should
> > > >     > > >     think about whether it's better to just bump up the
> magic
> > > > byte once
> > > >     > > to
> > > >     > > >     incorporate multiple format changes like we did in
> > > > KIP-31/KIP-32.
> > > >     > > >
> > > >     > > >     Thanks,
> > > >     > > >
> > > >     > > >     Jun
> > > >     > > >
> > > >     > > >
> > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > > >     > > Michael.Pearce@ig.com>
> > > >     > > >     wrote:
> > > >     > > >
> > > >     > > >     > Hi Jao
> > > >     > > >     >
> > > >     > > >     > Thanks for the response. Sorry for slow reply, both
> > with
> > > > personal
> > > >     > > > sickness
> > > >     > > >     > and also battling some critical issues encountered
> > since
> > > >     > upgrading
> > > >     > > to
> > > >     > > >     > 0.10.1.0
> > > >     > > >     >
> > > >     > > >     > 1) Thans for spotting, Document error where we
> branched
> > > > this KIP
> > > >     > > from
> > > >     > > >     > KIP-82, will get that removed.
> > > >     > > >     > 2) Intent is to do this just at the record message
> > level.
> > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> corrected.
> > > >     > > >     > 4) As per discussion thread we will support
> tombstone +
> > > > null
> > > >     > value,
> > > >     > > >     > tombstone + non null value, no tombstone + null
> value.
> > > >     > > >     > 5) I believe this was in the discussion thread,
> > @Mayuresh
> > > > is this
> > > >     > > >     > something we’ve overlooked? I thought we would down
> > > > convert and
> > > >     > > > remove the
> > > >     > > >     > value so the old consumer had existing behavior, or
> is
> > > > there
> > > >     > > > something we
> > > >     > > >     > haven’t thought about?
> > > >     > > >     >
> > > >     > > >     > Cheers
> > > >     > > >     > Mike
> > > >     > > >     >
> > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
> > > wrote:
> > > >     > > >     >
> > > >     > > >     >     Hi, Michael,
> > > >     > > >     >
> > > >     > > >     >     Thanks for the KIP. A few comments below.
> > > >     > > >     >
> > > >     > > >     >     1. The message format change contains
> > "HeadersLength
> > > >     > Headers".
> > > >     > > > Is that
> > > >     > > >     >     intended?
> > > >     > > >     >
> > > >     > > >     >     2. For compressed messageset, is the tombstone
> bit
> > > > only set
> > > >     > at
> > > >     > > > the
> > > >     > > >     > shallow
> > > >     > > >     >     level? Do we always leave that bit in the wrapper
> > > > message
> > > >     > > unset?
> > > >     > > > An
> > > >     > > >     >     alternative is to set the tombstone bit in the
> > > wrapper
> > > > if at
> > > >     > > > least one
> > > >     > > >     >     inner message has the tombstone bit set. This
> makes
> > > > things a
> > > >     > > bit
> > > >     > > > more
> > > >     > > >     >     complicated, but we could potentially exploit
> that
> > > for
> > > >     > > > optimizing down
> > > >     > > >     >     conversion. For example, we only need to convert
> > > > messages
> > > >     > with
> > > >     > > > magic 2
> > > >     > > >     > to
> > > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > > > (conversion is
> > > >     > > > always
> > > >     > > >     > needed
> > > >     > > >     >     from magic 2 to magic 0). Not sure if the
> > > optimization
> > > > is
> > > >     > worth
> > > >     > > > the
> > > >     > > >     >     complexity though.
> > > >     > > >     >
> > > >     > > >     >     3. The referencing of the new version of
> > > >     > > > ProducerRequest/FetchRequest
> > > >     > > >     > is
> > > >     > > >     >     inconsistent (v4 vs v3). Since our convention
> > starts
> > > at
> > > >     > version
> > > >     > > > at 0, I
> > > >     > > >     >     think the new version would be 3.
> > > >     > > >     >
> > > >     > > >     >     4. "If the magic byte on message is 2, the broker
> > > > should use
> > > >     > > the
> > > >     > > >     > tombstone
> > > >     > > >     >     bit for log compaction." What about null value?
> My
> > > >     > > understanding
> > > >     > > > is
> > > >     > > >     > that
> > > >     > > >     >     null value will be treated the same as setting
> the
> > > > tombstone
> > > >     > > bit.
> > > >     > > >     >
> > > >     > > >     >     5. For the migration path, it would be useful to
> > > > describe the
> > > >     > > > down
> > > >     > > >     >     conversion path to consumers (i.e., brokers on
> > > message
> > > > format
> > > >     > > > 0.10.2
> > > >     > > >     > and
> > > >     > > >     >     consumers on older version).
> > > >     > > >     >
> > > >     > > >     >     Thanks,
> > > >     > > >     >
> > > >     > > >     >     Jun
> > > >     > > >     >
> > > >     > > >     >
> > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > > >     > > > Michael.Pearce@ig.com
> > > >     > > >     > >
> > > >     > > >     >     wrote:
> > > >     > > >     >
> > > >     > > >     >     > Hi All,
> > > >     > > >     >     >
> > > >     > > >     >     > We have been discussing in the below thread and
> > > final
> > > >     > changes
> > > >     > > > have
> > > >     > > >     > been
> > > >     > > >     >     > made to the KIP wiki based on these
> discussions.
> > > >     > > >     >     >
> > > >     > > >     >     > We would now like to put to the vote the
> > following
> > > > KIP:
> > > >     > > >     >     > https://cwiki.apache.org/
> > > > confluence/display/KAFKA/KIP-
> > > >     > 87+-+
> > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > > >     > > >     >     >
> > > >     > > >     >     > This kip is for having a distinct compaction
> > > > attribute
> > > >     > > > “tombstone”
> > > >     > > >     > flag
> > > >     > > >     >     > instead of relying on null value, allowing
> > non-null
> > > > value
> > > >     > > > delete
> > > >     > > >     > messages.
> > > >     > > >     >     >
> > > >     > > >     >     > Many thanks,
> > > >     > > >     >     > Michael
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> > > >     > > Michael.Pearce@ig.com>
> > > >     > > >     > wrote:
> > > >     > > >     >     >
> > > >     > > >     >     >     Hi Mayuresh,
> > > >     > > >     >     >
> > > >     > > >     >     >     LGTM. Ive just made one small adjustment
> > > > updating the
> > > >     > > wire
> > > >     > > >     > protocol to
> > > >     > > >     >     > show the magic byte bump.
> > > >     > > >     >     >
> > > >     > > >     >     >     Do we think we’re good to put to a vote? Is
> > > > there any
> > > >     > > > other bits
> > > >     > > >     >     > needing discussion?
> > > >     > > >     >     >
> > > >     > > >     >     >     Cheers
> > > >     > > >     >     >     Mike
> > > >     > > >     >     >
> > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> > > >     > > >     > gharatmayuresh15@gmail.com>
> > > >     > > >     >     > wrote:
> > > >     > > >     >     >
> > > >     > > >     >     >         Hi Michael,
> > > >     > > >     >     >
> > > >     > > >     >     >         I have updated the migration section of
> > the
> > > > KIP.
> > > >     > Can
> > > >     > > > you
> > > >     > > >     > please
> > > >     > > >     >     > take a look?
> > > >     > > >     >     >
> > > >     > > >     >     >         Thanks,
> > > >     > > >     >     >
> > > >     > > >     >     >         Mayuresh
> > > >     > > >     >     >
> > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM,
> Mayuresh
> > > > Gharat <
> > > >     > > >     >     > gharatmayuresh15@gmail.com
> > > >     > > >     >     >         > wrote:
> > > >     > > >     >     >
> > > >     > > >     >     >         > Hi Michael,
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > That whilst sending tombstone and non
> > > null
> > > > value,
> > > >     > > the
> > > >     > > >     > consumer
> > > >     > > >     >     > can expect
> > > >     > > >     >     >         > only to receive the non-null message
> > only
> > > > in step
> > > >     > > > (3) is
> > > >     > > >     > this
> > > >     > > >     >     > correct?
> > > >     > > >     >     >         > ---> I do agree with you here.
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > Becket, Ismael : can you guys review
> > the
> > > >     > migration
> > > >     > > > plan
> > > >     > > >     > listed
> > > >     > > >     >     > above using
> > > >     > > >     >     >         > magic byte?
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > Thanks,
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > Mayuresh
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
> > Michael
> > > > Pearce <
> > > >     > > >     >     > Michael.Pearce@ig.com>
> > > >     > > >     >     >         > wrote:
> > > >     > > >     >     >         >
> > > >     > > >     >     >         >> Many thanks for this Mayuresh. I
> don't
> > > > have any
> > > >     > > >     > objections.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> I assume we should state:
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> That whilst sending tombstone and
> non
> > > null
> > > >     > value,
> > > >     > > > the
> > > >     > > >     > consumer
> > > >     > > >     >     > can expect
> > > >     > > >     >     >         >> only to receive the non-null message
> > > only
> > > > in
> > > >     > step
> > > >     > > > (3) is
> > > >     > > >     > this
> > > >     > > >     >     > correct?
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Cheers
> > > >     > > >     >     >         >> Mike
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Sent using OWA for iPhone
> > > >     > > >     >     >         >> ______________________________
> > > __________
> > > >     > > >     >     >         >> From: Mayuresh Gharat <
> > > >     > gharatmayuresh15@gmail.com
> > > >     > > >
> > > >     > > >     >     >         >> Sent: Thursday, November 17, 2016
> > > 5:18:41
> > > > PM
> > > >     > > >     >     >         >> To: dev@kafka.apache.org
> > > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
> > > > Compaction
> > > >     > > > Tombstone
> > > >     > > >     > Flag
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Hi Ismael,
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Thanks for the explanation.
> > > >     > > >     >     >         >> Specially I like this part where in
> > you
> > > >     > mentioned
> > > >     > > > we can
> > > >     > > >     > get
> > > >     > > >     >     > rid of the
> > > >     > > >     >     >         >> older null value support for log
> > > > compaction
> > > >     > later
> > > >     > > > on,
> > > >     > > >     > here :
> > > >     > > >     >     >         >> We can't change semantics of the
> > message
> > > > format
> > > >     > > > without
> > > >     > > >     > having
> > > >     > > >     >     > a long
> > > >     > > >     >     >         >> transition period. And we can't rely
> > > >     > > >     >     >         >> on people reading documentation or
> > > acting
> > > > on a
> > > >     > > > warning for
> > > >     > > >     >     > something so
> > > >     > > >     >     >         >> fundamental. As such, my take is
> that
> > we
> > > > need to
> > > >     > > > bump the
> > > >     > > >     > magic
> > > >     > > >     >     > byte. The
> > > >     > > >     >     >         >> good news is
> > > >     > > >     >     >         >> that we don't have to support all
> > > versions
> > > >     > > forever.
> > > >     > > > We
> > > >     > > >     > have
> > > >     > > >     >     > said that we
> > > >     > > >     >     >         >> will support direct upgrades for 2
> > > years.
> > > > That
> > > >     > > > means that
> > > >     > > >     >     > message format
> > > >     > > >     >     >         >> version n could, in theory, be
> > removed 2
> > > > years
> > > >     > > > after the
> > > >     > > >     > it's
> > > >     > > >     >     > introduced.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Just a heads up, I would like to
> > mention
> > > > that
> > > >     > even
> > > >     > > > without
> > > >     > > >     >     > bumping magic
> > > >     > > >     >     >         >> byte, we will *NOT* loose zero copy
> as
> > > in
> > > > the
> > > >     > > > client(x+1)
> > > >     > > >     > in my
> > > >     > > >     >     >         >> explanation
> > > >     > > >     >     >         >> above will convert internally a null
> > > > value to
> > > >     > > have a
> > > >     > > >     > tombstone
> > > >     > > >     >     > bit set and
> > > >     > > >     >     >         >> a tombstone bit set to have a null
> > value
> > > >     > > > automatically
> > > >     > > >     >     > internally and by
> > > >     > > >     >     >         >> the time we move to version (x+2),
> the
> > > > clients
> > > >     > > > would have
> > > >     > > >     >     > upgraded.
> > > >     > > >     >     >         >> Obviously if we support a request
> from
> > > >     > > consumer(x),
> > > >     > > > we
> > > >     > > >     > will
> > > >     > > >     >     > loose zero
> > > >     > > >     >     >         >> copy
> > > >     > > >     >     >         >> but that is the same case with magic
> > > byte.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> But if magic byte bump makes life
> > easier
> > > > for
> > > >     > > > transition
> > > >     > > >     > for the
> > > >     > > >     >     > above
> > > >     > > >     >     >         >> reasons that you explained, I am OK
> > with
> > > > it
> > > >     > since
> > > >     > > > we are
> > > >     > > >     > going
> > > >     > > >     >     > to meet the
> > > >     > > >     >     >         >> end goal down the road :)
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> On a side note can we update the doc
> > > here
> > > > on
> > > >     > magic
> > > >     > > > byte
> > > >     > > >     > to say
> > > >     > > >     >     > that "*it
> > > >     > > >     >     >         >> should be bumped whenever the
> message
> > > > format is
> > > >     > > > changed
> > > >     > > >     > or the
> > > >     > > >     >     >         >> interpretation of message format
> > (usage
> > > > of the
> > > >     > > > reserved
> > > >     > > >     > bits as
> > > >     > > >     >     > well) is
> > > >     > > >     >     >         >> changed*".
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Hi Michael,
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Here is the update plan that we
> > > discussed
> > > >     > offline
> > > >     > > >     > yesterday :
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Currently the magic-byte which
> > > > corresponds to
> > > >     > the
> > > >     > > >     >     > "message.format.version"
> > > >     > > >     >     >         >> is set to 1.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> 1) On broker it will be set to 1
> > > > initially.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> 2) When a producer client sends a
> > > message
> > > > with
> > > >     > > > magic-byte
> > > >     > > >     > = 2,
> > > >     > > >     >     > since the
> > > >     > > >     >     >         >> broker is on magic-byte = 1, we will
> > > down
> > > >     > convert
> > > >     > > > it,
> > > >     > > >     > which
> > > >     > > >     >     > means if the
> > > >     > > >     >     >         >> tombstone bit is set, the value will
> > be
> > > > set to
> > > >     > > > null. A
> > > >     > > >     > consumer
> > > >     > > >     >     >         >> understanding magic-byte = 1, will
> > still
> > > > work
> > > >     > with
> > > >     > > > this. A
> > > >     > > >     >     > consumer
> > > >     > > >     >     >         >> working
> > > >     > > >     >     >         >> with magic-byte =2 will also be able
> > to
> > > >     > understand
> > > >     > > > this,
> > > >     > > >     > since
> > > >     > > >     >     > it
> > > >     > > >     >     >         >> understands the tombstone.
> > > >     > > >     >     >         >> Now there is still the question of
> > > > supporting a
> > > >     > > >     > non-tombstone
> > > >     > > >     >     > and null
> > > >     > > >     >     >         >> value from producer client with
> > > > magic-byte = 2.*
> > > >     > > (I
> > > >     > > > am
> > > >     > > >     > not sure
> > > >     > > >     >     > if we
> > > >     > > >     >     >         >> should support this. Ismael/Becket
> can
> > > > comment
> > > >     > > > here)*
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> 3) When almost all the clients have
> > > > upgraded,
> > > >     > the
> > > >     > > >     >     > message.format.version
> > > >     > > >     >     >         >> on
> > > >     > > >     >     >         >> the broker can be changed to 2,
> where
> > in
> > > > the
> > > >     > down
> > > >     > > >     > conversion in
> > > >     > > >     >     > the above
> > > >     > > >     >     >         >> step will not happen. If at this
> point
> > > we
> > > > get a
> > > >     > > > consumer
> > > >     > > >     >     > request from a
> > > >     > > >     >     >         >> older consumer, we might have to
> down
> > > > convert
> > > >     > > where
> > > >     > > > in we
> > > >     > > >     > loose
> > > >     > > >     >     > zero copy,
> > > >     > > >     >     >         >> but these cases should be rare.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Becket can you review this plan and
> > add
> > > > more
> > > >     > > > details if I
> > > >     > > >     > have
> > > >     > > >     >     >         >> missed/wronged something, before we
> > put
> > > > it on
> > > >     > KIP.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Thanks,
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Mayuresh
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
> > > Michael
> > > >     > Pearce <
> > > >     > > >     >     > Michael.Pearce@ig.com>
> > > >     > > >     >     >         >> wrote:
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> > Thanks guys, for discussing this
> > > > offline and
> > > >     > > > getting
> > > >     > > >     > some
> > > >     > > >     >     > consensus.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > So its clear for myself and others
> > > what
> > > > is
> > > >     > > > proposed now
> > > >     > > >     > (i
> > > >     > > >     >     > think i
> > > >     > > >     >     >         >> > understand, but want to make sure)
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Could i ask either directly update
> > the
> > > > kip to
> > > >     > > > detail the
> > > >     > > >     >     > migration
> > > >     > > >     >     >         >> > strategy, or (re-)state your
> offline
> > > > discussed
> > > >     > > and
> > > >     > > >     > agreed
> > > >     > > >     >     > migration
> > > >     > > >     >     >         >> > strategy based on a magic byte is
> in
> > > > this
> > > >     > > thread.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > The main original driver for the
> KIP
> > > > was to
> > > >     > > > support
> > > >     > > >     >     > compaction where
> > > >     > > >     >     >         >> value
> > > >     > > >     >     >         >> > isn't null, based off the
> > discussions
> > > on
> > > >     > KIP-82
> > > >     > > > thread.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > We should be able to support
> > > > non-tombstone +
> > > >     > > null
> > > >     > > > value
> > > >     > > >     > by the
> > > >     > > >     >     >         >> completion
> > > >     > > >     >     >         >> > of the KIP, as we noted when
> > > discussing
> > > > this
> > > >     > > kip,
> > > >     > > > having
> > > >     > > >     >     > logic based on
> > > >     > > >     >     >         >> a
> > > >     > > >     >     >         >> > null value isn't very clean and
> also
> > > > separates
> > > >     > > the
> > > >     > > >     > concerns.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > As discussed already though we can
> > > > split this
> > > >     > > into
> > > >     > > >     > KIP-87a
> > > >     > > >     >     > and KIP-87b
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Where we look to deliver KIP-87a
> on
> > a
> > > >     > compacted
> > > >     > > > topic
> > > >     > > >     > (to
> > > >     > > >     >     > address the
> > > >     > > >     >     >         >> > immediate issues)
> > > >     > > >     >     >         >> > * tombstone + null value
> > > >     > > >     >     >         >> > * tombstone + non-null value
> > > >     > > >     >     >         >> > * non-tombstone + non-null value
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Then we can discuss once KIP-87a
> is
> > > > completed
> > > >     > > > options
> > > >     > > >     > later
> > > >     > > >     >     > and how we
> > > >     > > >     >     >         >> > support the second part KIP-87b to
> > > > deliver:
> > > >     > > >     >     >         >> > * non-tombstone + null value
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Cheers
> > > >     > > >     >     >         >> > Mike
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > ______________________________
> > > > __________
> > > >     > > >     >     >         >> > From: Becket Qin <
> > > becket.qin@gmail.com>
> > > >     > > >     >     >         >> > Sent: Thursday, November 17, 2016
> > 1:43
> > > > AM
> > > >     > > >     >     >         >> > To: dev@kafka.apache.org
> > > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 -
> Add
> > > > Compaction
> > > >     > > >     > Tombstone Flag
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Renu, Mayuresh and I had an
> offline
> > > >     > discussion,
> > > >     > > > and
> > > >     > > >     > following
> > > >     > > >     >     > is a brief
> > > >     > > >     >     >         >> > summary.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > 1. We agreed that not bumping up
> > magic
> > > > value
> > > >     > may
> > > >     > > > result
> > > >     > > >     > in
> > > >     > > >     >     > losing zero
> > > >     > > >     >     >         >> copy
> > > >     > > >     >     >         >> > during migration.
> > > >     > > >     >     >         >> > 2. Given that bumping up magic
> value
> > > is
> > > > almost
> > > >     > > > free and
> > > >     > > >     > has
> > > >     > > >     >     > benefit of
> > > >     > > >     >     >         >> > avoiding potential performance
> > issue.
> > > > It is
> > > >     > > > probably
> > > >     > > >     > worth
> > > >     > > >     >     > doing.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > One issue we still need to think
> > about
> > > > is
> > > >     > > whether
> > > >     > > > we
> > > >     > > >     > want to
> > > >     > > >     >     > support a
> > > >     > > >     >     >         >> > non-tombstone message with null
> > value.
> > > >     > > >     >     >         >> > Currently it is not supported by
> > > Kafka.
> > > > If we
> > > >     > > > allow a
> > > >     > > >     >     > non-tombstone null
> > > >     > > >     >     >         >> > value message to exist after
> KIP-87.
> > > The
> > > >     > problem
> > > >     > > > is
> > > >     > > >     > that such
> > > >     > > >     >     > message
> > > >     > > >     >     >         >> will
> > > >     > > >     >     >         >> > not be supported by the consumers
> > > prior
> > > > to
> > > >     > > KIP-87.
> > > >     > > >     > Because a
> > > >     > > >     >     > null value
> > > >     > > >     >     >         >> > will always be interpreted to a
> > > > tombstone.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > One option is that we keep the
> > current
> > > > way,
> > > >     > i.e.
> > > >     > > > do not
> > > >     > > >     >     > support such
> > > >     > > >     >     >         >> > message. It would be good to know
> if
> > > > there is
> > > >     > a
> > > >     > > >     > concrete use
> > > >     > > >     >     > case for
> > > >     > > >     >     >         >> such
> > > >     > > >     >     >         >> > message. If there is not, we can
> > > > probably just
> > > >     > > not
> > > >     > > >     > support it.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Thanks,
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> > > > Mayuresh
> > > >     > > Gharat <
> > > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> > > >     > > >     >     >         >> > > wrote:
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > > Hi Ismael,
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > This is something I can think of
> > for
> > > >     > migration
> > > >     > > > plan:
> > > >     > > >     >     >         >> > > So the migration plan can look
> > > > something
> > > >     > like
> > > >     > > > this,
> > > >     > > >     > with up
> > > >     > > >     >     >         >> conversion :
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > 1) Currently lets say we have
> > Broker
> > > > at
> > > >     > > version
> > > >     > > > x.
> > > >     > > >     >     >         >> > > 2) Currently we have clients at
> > > > version x.
> > > >     > > >     >     >         >> > > 3) a) We move the version to
> > > > Broker(x+1) :
> > > >     > > > supports
> > > >     > > >     > both
> > > >     > > >     >     > tombstone and
> > > >     > > >     >     >         >> > null
> > > >     > > >     >     >         >> > > for log compaction.
> > > >     > > >     >     >         >> > >     b) We upgrade the client to
> > > > version
> > > >     > > > client(x+1) :
> > > >     > > >     > if in
> > > >     > > >     >     > the
> > > >     > > >     >     >         >> producer
> > > >     > > >     >     >         >> > > client(x+1) the value is set to
> > > null,
> > > > we
> > > >     > will
> > > >     > > >     > automatically
> > > >     > > >     >     > set the
> > > >     > > >     >     >         >> > > Tombstone bit internally. If the
> > > > producer
> > > >     > > > client(x+1)
> > > >     > > >     > sets
> > > >     > > >     >     > the
> > > >     > > >     >     >         >> tombstone
> > > >     > > >     >     >         >> > > itself, well and good. For
> > producer
> > > >     > client(x),
> > > >     > > > the
> > > >     > > >     > broker
> > > >     > > >     >     > will up
> > > >     > > >     >     >         >> convert
> > > >     > > >     >     >         >> > > to have the tombstone bit.
> > > > Broker(x+1) is
> > > >     > > > supporting
> > > >     > > >     > both.
> > > >     > > >     >     > Consumer
> > > >     > > >     >     >         >> > > client(x+1) will be aware of
> this
> > > and
> > > > should
> > > >     > > be
> > > >     > > > able
> > > >     > > >     > to
> > > >     > > >     >     > handle this.
> > > >     > > >     >     >         >> For
> > > >     > > >     >     >         >> > > consumer client(x) we will down
> > > > convert the
> > > >     > > > message
> > > >     > > >     > on the
> > > >     > > >     >     > broker
> > > >     > > >     >     >         >> side.
> > > >     > > >     >     >         >> > >     c) At this point we will
> have
> > to
> > > >     > specify a
> > > >     > > >     > warning or
> > > >     > > >     >     > clearly
> > > >     > > >     >     >         >> specify
> > > >     > > >     >     >         >> > > in docs that this behavior is
> > about
> > > > to be
> > > >     > > > changed for
> > > >     > > >     > log
> > > >     > > >     >     > compaction.
> > > >     > > >     >     >         >> > > 4) a) In next release of the
> > > > Broker(x+2), we
> > > >     > > > say that
> > > >     > > >     > only
> > > >     > > >     >     > Tombstone
> > > >     > > >     >     >         >> is
> > > >     > > >     >     >         >> > > used for log compaction on the
> > > Broker
> > > > side.
> > > >     > > >     > Clients(x+1)
> > > >     > > >     >     > still is
> > > >     > > >     >     >         >> > > supported.
> > > >     > > >     >     >         >> > >     b) We upgrade the client to
> > > > version
> > > >     > > > client(x+2) :
> > > >     > > >     > if
> > > >     > > >     >     > value is set
> > > >     > > >     >     >         >> to
> > > >     > > >     >     >         >> > > null, tombstone will not be set
> > > >     > automatically.
> > > >     > > > The
> > > >     > > >     > client
> > > >     > > >     >     > will have to
> > > >     > > >     >     >         >> > call
> > > >     > > >     >     >         >> > > setTombstone() to actually set
> the
> > > >     > tombstone.
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > We should compare this migration
> > > plan
> > > > with
> > > >     > the
> > > >     > > >     > migration
> > > >     > > >     >     > plan for
> > > >     > > >     >     >         >> magic
> > > >     > > >     >     >         >> > > byte bump and do whatever looks
> > > good.
> > > >     > > >     >     >         >> > > I am just worried that if we go
> > down
> > > > magic
> > > >     > > byte
> > > >     > > > route,
> > > >     > > >     >     > unless I am
> > > >     > > >     >     >         >> > missing
> > > >     > > >     >     >         >> > > something, it sounds like kafka
> > will
> > > > be
> > > >     > stuck
> > > >     > > > with
> > > >     > > >     >     > supporting both
> > > >     > > >     >     >         >> null
> > > >     > > >     >     >         >> > > value and tombstone bit for log
> > > > compaction
> > > >     > for
> > > >     > > > life
> > > >     > > >     > long,
> > > >     > > >     >     > which does
> > > >     > > >     >     >         >> not
> > > >     > > >     >     >         >> > > look like a good end state.
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > Thanks,
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > Mayuresh
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
> > > > Mayuresh
> > > >     > > > Gharat <
> > > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> > > >     > > >     >     >         >> > > > wrote:
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > > Hi Ismael,
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > That's a very good point
> which I
> > > > might
> > > >     > have
> > > >     > > > not
> > > >     > > >     >     > considered earlier.
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > Here is a plan that I can
> think
> > > of:
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > Stage 1) The broker from now
> on,
> > > up
> > > >     > converts
> > > >     > > > the
> > > >     > > >     > message
> > > >     > > >     >     > to have the
> > > >     > > >     >     >         >> > > > tombstone marker. The log
> > > compaction
> > > >     > thread
> > > >     > > > does log
> > > >     > > >     >     > compaction
> > > >     > > >     >     >         >> based
> > > >     > > >     >     >         >> > on
> > > >     > > >     >     >         >> > > > both null and tombstone
> marker.
> > > > This is
> > > >     > our
> > > >     > > >     > transition
> > > >     > > >     >     > period.
> > > >     > > >     >     >         >> > > > Stage 2) The next release we
> > only
> > > > say that
> > > >     > > log
> > > >     > > >     > compaction
> > > >     > > >     >     > is based
> > > >     > > >     >     >         >> on
> > > >     > > >     >     >         >> > > > tombstone marker. (Open source
> > > > kafka makes
> > > >     > > > this as a
> > > >     > > >     >     > policy). By
> > > >     > > >     >     >         >> this
> > > >     > > >     >     >         >> > > time,
> > > >     > > >     >     >         >> > > > the organization which is
> moving
> > > to
> > > > this
> > > >     > > > release
> > > >     > > >     > will be
> > > >     > > >     >     > sure that
> > > >     > > >     >     >         >> they
> > > >     > > >     >     >         >> > > > have gone through the entire
> > > > transition
> > > >     > > > period.
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > My only goal of doing this is
> > that
> > > > Kafka
> > > >     > > > clearly
> > > >     > > >     >     > specifies the end
> > > >     > > >     >     >         >> > state
> > > >     > > >     >     >         >> > > > about what log compaction
> means
> > > (is
> > > > it
> > > >     > null
> > > >     > > > value
> > > >     > > >     > or a
> > > >     > > >     >     > tombstone
> > > >     > > >     >     >         >> > marker,
> > > >     > > >     >     >         >> > > > but not both).
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > What do you think?
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > Thanks,
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > Mayuresh
> > > >     > > >     >     >         >> > > > .
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17
> AM,
> > > > Ismael
> > > >     > > Juma <
> > > >     > > >     >     > ismael@juma.me.uk>
> > > >     > > >     >     >         >> > wrote:
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > >> One comment below.
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08
> > PM,
> > > > Mayuresh
> > > >     > > > Gharat <
> > > >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> > > >     > > >     >     >         >> > > >> > wrote:
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >> >    - If we don't bump up
> the
> > > > magic
> > > >     > byte,
> > > >     > > > on the
> > > >     > > >     > broker
> > > >     > > >     >     > side, the
> > > >     > > >     >     >         >> > > broker
> > > >     > > >     >     >         >> > > >> >    will always have to look
> > at
> > > > both
> > > >     > > > tombstone
> > > >     > > >     > bit and
> > > >     > > >     >     > the value
> > > >     > > >     >     >         >> when
> > > >     > > >     >     >         >> > > do
> > > >     > > >     >     >         >> > > >> the
> > > >     > > >     >     >         >> > > >> >    compaction. Assuming we
> do
> > > > not bump
> > > >     > up
> > > >     > > > the
> > > >     > > >     > magic
> > > >     > > >     >     > byte,
> > > >     > > >     >     >         >> > > >> >    imagine the broker sees
> a
> > > > message
> > > >     > > which
> > > >     > > > does
> > > >     > > >     > not
> > > >     > > >     >     > have a
> > > >     > > >     >     >         >> tombstone
> > > >     > > >     >     >         >> > > bit
> > > >     > > >     >     >         >> > > >> >    set. The broker does not
> > > know
> > > > when
> > > >     > the
> > > >     > > >     > message was
> > > >     > > >     >     > produced
> > > >     > > >     >     >         >> (i.e.
> > > >     > > >     >     >         >> > > >> > whether
> > > >     > > >     >     >         >> > > >> >    the message has been up
> > > > converted or
> > > >     > > > not), it
> > > >     > > >     > has
> > > >     > > >     >     > to take a
> > > >     > > >     >     >         >> > further
> > > >     > > >     >     >         >> > > >> > look at
> > > >     > > >     >     >         >> > > >> >    the value to see if it
> is
> > > > null or
> > > >     > not
> > > >     > > in
> > > >     > > >     > order to
> > > >     > > >     >     > determine
> > > >     > > >     >     >         >> if it
> > > >     > > >     >     >         >> > > is
> > > >     > > >     >     >         >> > > >> a
> > > >     > > >     >     >         >> > > >> >    tombstone. The same
> logic
> > > has
> > > > to be
> > > >     > > put
> > > >     > > > on the
> > > >     > > >     >     > consumer as
> > > >     > > >     >     >         >> well
> > > >     > > >     >     >         >> > > >> because
> > > >     > > >     >     >         >> > > >> > the
> > > >     > > >     >     >         >> > > >> >    consumer does not know
> if
> > > the
> > > >     > message
> > > >     > > > has
> > > >     > > >     > been up
> > > >     > > >     >     > converted or
> > > >     > > >     >     >         >> > not.
> > > >     > > >     >     >         >> > > >> >       - If we upconvert
> while
> > > >     > appending,
> > > >     > > > this is
> > > >     > > >     > not
> > > >     > > >     >     > the case,
> > > >     > > >     >     >         >> > right?
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >> If I understand you
> correctly,
> > > > this is
> > > >     > not
> > > >     > > >     > sufficient
> > > >     > > >     >     > because the
> > > >     > > >     >     >         >> log
> > > >     > > >     >     >         >> > > may
> > > >     > > >     >     >         >> > > >> have messages appended before
> > it
> > > > was
> > > >     > > > upgraded to
> > > >     > > >     > include
> > > >     > > >     >     > KIP-87.
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >> Ismael
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > --
> > > >     > > >     >     >         >> > > > -Regards,
> > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> > > >     > > >     >     >         >> > > > (862) 250-7125
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > --
> > > >     > > >     >     >         >> > > -Regards,
> > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > > >     > > >     >     >         >> > > (862) 250-7125
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > The information contained in this
> > > email
> > > > is
> > > >     > > > strictly
> > > >     > > >     >     > confidential and for
> > > >     > > >     >     >         >> > the use of the addressee only,
> > unless
> > > >     > otherwise
> > > >     > > >     > indicated. If
> > > >     > > >     >     > you are
> > > >     > > >     >     >         >> not
> > > >     > > >     >     >         >> > the intended recipient, please do
> > not
> > > > read,
> > > >     > > copy,
> > > >     > > > use or
> > > >     > > >     >     > disclose to
> > > >     > > >     >     >         >> others
> > > >     > > >     >     >         >> > this message or any attachment.
> > Please
> > > > also
> > > >     > > > notify the
> > > >     > > >     > sender
> > > >     > > >     >     > by
> > > >     > > >     >     >         >> replying
> > > >     > > >     >     >         >> > to this email or by telephone
> > (+44(020
> > > > 7896
> > > >     > > 0011)
> > > >     > > > and
> > > >     > > >     > then
> > > >     > > >     >     > delete the
> > > >     > > >     >     >         >> email
> > > >     > > >     >     >         >> > and any copies of it. Opinions,
> > > > conclusion
> > > >     > (etc)
> > > >     > > > that
> > > >     > > >     > do not
> > > >     > > >     >     > relate to
> > > >     > > >     >     >         >> the
> > > >     > > >     >     >         >> > official business of this company
> > > shall
> > > > be
> > > >     > > > understood as
> > > >     > > >     >     > neither given
> > > >     > > >     >     >         >> nor
> > > >     > > >     >     >         >> > endorsed by it. IG is a trading
> name
> > > of
> > > > IG
> > > >     > > Markets
> > > >     > > >     > Limited (a
> > > >     > > >     >     > company
> > > >     > > >     >     >         >> > registered in England and Wales,
> > > company
> > > >     > number
> > > >     > > >     > 04008957) and
> > > >     > > >     >     > IG Index
> > > >     > > >     >     >         >> > Limited (a company registered in
> > > > England and
> > > >     > > > Wales,
> > > >     > > >     > company
> > > >     > > >     >     > number
> > > >     > > >     >     >         >> > 01190902). Registered address at
> > > Cannon
> > > > Bridge
> > > >     > > > House, 25
> > > >     > > >     >     > Dowgate Hill,
> > > >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
> > > Limited
> > > >     > > (register
> > > >     > > >     > number
> > > >     > > >     >     > 195355) and IG
> > > >     > > >     >     >         >> > Index Limited (register number
> > 114059)
> > > > are
> > > >     > > > authorised
> > > >     > > >     > and
> > > >     > > >     >     > regulated by
> > > >     > > >     >     >         >> the
> > > >     > > >     >     >         >> > Financial Conduct Authority.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> --
> > > >     > > >     >     >         >> -Regards,
> > > >     > > >     >     >         >> Mayuresh R. Gharat
> > > >     > > >     >     >         >> (862) 250-7125
> > > >     > > >     >     >         >> The information contained in this
> > email
> > > is
> > > >     > > strictly
> > > >     > > >     >     > confidential and for
> > > >     > > >     >     >         >> the use of the addressee only,
> unless
> > > > otherwise
> > > >     > > >     > indicated. If
> > > >     > > >     >     > you are not
> > > >     > > >     >     >         >> the intended recipient, please do
> not
> > > > read,
> > > >     > copy,
> > > >     > > > use or
> > > >     > > >     >     > disclose to others
> > > >     > > >     >     >         >> this message or any attachment.
> Please
> > > > also
> > > >     > notify
> > > >     > > > the
> > > >     > > >     > sender
> > > >     > > >     >     > by replying
> > > >     > > >     >     >         >> to this email or by telephone
> (+44(020
> > > > 7896
> > > >     > 0011)
> > > >     > > > and then
> > > >     > > >     >     > delete the email
> > > >     > > >     >     >         >> and any copies of it. Opinions,
> > > > conclusion (etc)
> > > >     > > > that do
> > > >     > > >     > not
> > > >     > > >     >     > relate to the
> > > >     > > >     >     >         >> official business of this company
> > shall
> > > be
> > > >     > > > understood as
> > > >     > > >     >     > neither given nor
> > > >     > > >     >     >         >> endorsed by it. IG is a trading name
> > of
> > > IG
> > > >     > Markets
> > > >     > > >     > Limited (a
> > > >     > > >     >     > company
> > > >     > > >     >     >         >> registered in England and Wales,
> > company
> > > > number
> > > >     > > > 04008957)
> > > >     > > >     > and
> > > >     > > >     >     > IG Index
> > > >     > > >     >     >         >> Limited (a company registered in
> > England
> > > > and
> > > >     > > Wales,
> > > >     > > >     > company
> > > >     > > >     >     > number
> > > >     > > >     >     >         >> 01190902). Registered address at
> > Cannon
> > > > Bridge
> > > >     > > > House, 25
> > > >     > > >     >     > Dowgate Hill,
> > > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
> > Limited
> > > >     > (register
> > > >     > > > number
> > > >     > > >     >     > 195355) and IG
> > > >     > > >     >     >         >> Index Limited (register number
> 114059)
> > > are
> > > >     > > > authorised and
> > > >     > > >     >     > regulated by the
> > > >     > > >     >     >         >> Financial Conduct Authority.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >
> > > >     > > >     >     >         >
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > --
> > > >     > > >     >     >         > -Regards,
> > > >     > > >     >     >         > Mayuresh R. Gharat
> > > >     > > >     >     >         > (862) 250-7125
> > > >     > > >     >     >         >
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >         --
> > > >     > > >     >     >         -Regards,
> > > >     > > >     >     >         Mayuresh R. Gharat
> > > >     > > >     >     >         (862) 250-7125
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >     The information contained in this email is
> > > > strictly
> > > >     > > > confidential
> > > >     > > >     > and
> > > >     > > >     >     > for the use of the addressee only, unless
> > otherwise
> > > >     > > indicated.
> > > >     > > > If
> > > >     > > >     > you are
> > > >     > > >     >     > not the intended recipient, please do not read,
> > > > copy, use
> > > >     > or
> > > >     > > >     > disclose to
> > > >     > > >     >     > others this message or any attachment. Please
> > also
> > > > notify
> > > >     > the
> > > >     > > > sender
> > > >     > > >     > by
> > > >     > > >     >     > replying to this email or by telephone (+44(020
> > > 7896
> > > > 0011)
> > > >     > > and
> > > >     > > > then
> > > >     > > >     > delete
> > > >     > > >     >     > the email and any copies of it. Opinions,
> > > conclusion
> > > > (etc)
> > > >     > > > that do
> > > >     > > >     > not
> > > >     > > >     >     > relate to the official business of this company
> > > > shall be
> > > >     > > > understood
> > > >     > > >     > as
> > > >     > > >     >     > neither given nor endorsed by it. IG is a
> trading
> > > > name of
> > > >     > IG
> > > >     > > > Markets
> > > >     > > >     >     > Limited (a company registered in England and
> > Wales,
> > > > company
> > > >     > > > number
> > > >     > > >     >     > 04008957) and IG Index Limited (a company
> > > registered
> > > > in
> > > >     > > > England and
> > > >     > > >     > Wales,
> > > >     > > >     >     > company number 01190902). Registered address at
> > > > Cannon
> > > >     > Bridge
> > > >     > > > House,
> > > >     > > >     > 25
> > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > > > Limited
> > > >     > > > (register
> > > >     > > >     > number
> > > >     > > >     >     > 195355) and IG Index Limited (register number
> > > > 114059) are
> > > >     > > > authorised
> > > >     > > >     > and
> > > >     > > >     >     > regulated by the Financial Conduct Authority.
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >
> > > >     > > >     >
> > > >     > > >     > The information contained in this email is strictly
> > > > confidential
> > > >     > > and
> > > >     > > > for
> > > >     > > >     > the use of the addressee only, unless otherwise
> > > indicated.
> > > > If you
> > > >     > > > are not
> > > >     > > >     > the intended recipient, please do not read, copy, use
> > or
> > > > disclose
> > > >     > > to
> > > >     > > > others
> > > >     > > >     > this message or any attachment. Please also notify
> the
> > > > sender by
> > > >     > > > replying
> > > >     > > >     > to this email or by telephone (+44(020 7896 0011) and
> > > then
> > > > delete
> > > >     > > > the email
> > > >     > > >     > and any copies of it. Opinions, conclusion (etc) that
> > do
> > > > not
> > > >     > relate
> > > >     > > > to the
> > > >     > > >     > official business of this company shall be understood
> > as
> > > > neither
> > > >     > > > given nor
> > > >     > > >     > endorsed by it. IG is a trading name of IG Markets
> > > Limited
> > > > (a
> > > >     > > company
> > > >     > > >     > registered in England and Wales, company number
> > 04008957)
> > > > and IG
> > > >     > > > Index
> > > >     > > >     > Limited (a company registered in England and Wales,
> > > company
> > > >     > number
> > > >     > > >     > 01190902). Registered address at Cannon Bridge House,
> > 25
> > > > Dowgate
> > > >     > > > Hill,
> > > >     > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > number
> > > > 195355)
> > > >     > > > and IG
> > > >     > > >     > Index Limited (register number 114059) are authorised
> > and
> > > >     > regulated
> > > >     > > > by the
> > > >     > > >     > Financial Conduct Authority.
> > > >     > > >     >
> > > >     > > >
> > > >     > > >
> > > >     > > > The information contained in this email is strictly
> > > confidential
> > > > and
> > > >     > for
> > > >     > > > the use of the addressee only, unless otherwise indicated.
> If
> > > > you are
> > > >     > not
> > > >     > > > the intended recipient, please do not read, copy, use or
> > > > disclose to
> > > >     > > others
> > > >     > > > this message or any attachment. Please also notify the
> sender
> > > by
> > > >     > replying
> > > >     > > > to this email or by telephone (+44(020 7896 0011) and then
> > > > delete the
> > > >     > > email
> > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > > relate to
> > > >     > > the
> > > >     > > > official business of this company shall be understood as
> > > neither
> > > > given
> > > >     > > nor
> > > >     > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > > company
> > > >     > > > registered in England and Wales, company number 04008957)
> and
> > > IG
> > > > Index
> > > >     > > > Limited (a company registered in England and Wales, company
> > > > number
> > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > > Hill,
> > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > > 195355) and
> > > >     > IG
> > > >     > > > Index Limited (register number 114059) are authorised and
> > > > regulated by
> > > >     > > the
> > > >     > > > Financial Conduct Authority.
> > > >     > > >
> > > >     > > The information contained in this email is strictly
> > confidential
> > > > and for
> > > >     > > the use of the addressee only, unless otherwise indicated. If
> > you
> > > > are not
> > > >     > > the intended recipient, please do not read, copy, use or
> > disclose
> > > > to
> > > >     > others
> > > >     > > this message or any attachment. Please also notify the sender
> > by
> > > > replying
> > > >     > > to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > > the
> > > >     > email
> > > >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > > > relate to
> > > >     > the
> > > >     > > official business of this company shall be understood as
> > neither
> > > > given
> > > >     > nor
> > > >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > > company
> > > >     > > registered in England and Wales, company number 04008957) and
> > IG
> > > > Index
> > > >     > > Limited (a company registered in England and Wales, company
> > > number
> > > >     > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > Hill,
> > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > and IG
> > > >     > > Index Limited (register number 114059) are authorised and
> > > > regulated by
> > > >     > the
> > > >     > > Financial Conduct Authority.
> > > >     > >
> > > >     > >
> > > >     > The information contained in this email is strictly
> confidential
> > > and
> > > > for
> > > >     > the use of the addressee only, unless otherwise indicated. If
> you
> > > > are not
> > > >     > the intended recipient, please do not read, copy, use or
> disclose
> > > to
> > > > others
> > > >     > this message or any attachment. Please also notify the sender
> by
> > > > replying
> > > >     > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > > > the email
> > > >     > and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> > > > to the
> > > >     > official business of this company shall be understood as
> neither
> > > > given nor
> > > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > > >     > registered in England and Wales, company number 04008957) and
> IG
> > > > Index
> > > >     > Limited (a company registered in England and Wales, company
> > number
> > > >     > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > > > Hill,
> > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > > and IG
> > > >     > Index Limited (register number 114059) are authorised and
> > regulated
> > > > by the
> > > >     > Financial Conduct Authority.
> > > >     >
> > > >     >
> > > >
> > > >
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Ewen Cheslack-Postava <ew...@confluent.io>]

Michael,

It kind of depends on how you want to interpret the tombstone flag. If it's
purely a producer-facing Kafka-level thing that we treat as internal to the
broker and log cleaner once the record is sent, then your approach makes
sense. You're just moving copying the null-indicates-delete behavior of the
old constructor into the tombstone flag.

However, if you want this change to more generally decouple the idea of
deletion and null values, then you are sometimes converting what might be a
completely valid null value that doesn't indicate deletion into a
tombstone. Downstream applications could potentially handle these cases
differently given the separation of deletion from value.

I guess the question is if we want to try to support the latter even for
topics where we have older produce requests. An example where this could
come up is in something like a CDC Connector. If we try to support the
semantic difference, a connector might write changes to Kafka using the
tombstone flag to indicate when a row was truly deleted (vs an update that
sets it to null but still present; this probably makes more sense for CDC
from document stores or extracting single columns). There are various
reasons we might want to maintain the full log and not turn compaction on
(or just use a time-based retention policy), but downstream applications
might care to know the difference between a delete and a null value. In
fact both versions of the same log (compacted and time-retention) could be
useful and I don't think it'll be uncommon to maintain both or use KIP-71
to maintain a hybrid compacted/retention topic.

-Ewen

On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jay,
>
> Why wouldn't that work, the tombstone value is only looked at by the
> broker, on a topic configured for compaction as such is benign on non
> compacted topics. This is as much as sending a null value currently
>
>
> Regards
> Mike
>
>
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Sunday, December 11, 2016 8:58:53 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hey Michael,
>
> I'm not quite sure that works as that would translate ALL null values to
> tombstones, even for non-compacted topics that use null as an acceptable
> value sent by the producer and expected by the consumer.
>
> -Jay
>
> On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Ewen,
> >
> > I think the easiest way to show this is with code.
> >
> > As you can see we keep the existing behaviour for code/binaries calling
> > the pre-existing constructors, whereby if the value is null the tombstone
> > is set to true.
> >
> > Regards
> > Mike
> >
> >
> >
> >     /**
> >      * Creates a record with a specified timestamp to be sent to a
> > specified topic and partition
> >      *
> >      * @param topic The topic the record will be appended to
> >      * @param partition The partition to which the record should be sent
> >      * @param timestamp The timestamp of the record
> >      * @param tombstone if the record should be treated as a tombstone if
> > the topic is compacted
> >      * @param key The key that will be included in the record
> >      * @param value The record contents
> >      */
> >     public ProducerRecord(String topic, Integer partition, Boolean
> > tombstone, Long timestamp, K key, V value) {
> >         if (topic == null)
> >             throw new IllegalArgumentException("Topic cannot be null.");
> >         if (timestamp != null && timestamp < 0)
> >             throw new IllegalArgumentException(
> >                     String.format("Invalid timestamp: %d. Timestamp
> should
> > always be non-negative or null.", timestamp));
> >         if (partition != null && partition < 0)
> >             throw new IllegalArgumentException(
> >                     String.format("Invalid partition: %d. Partition
> number
> > should always be non-negative or null.", partition));
> >         if (!tombstone && value == null){
> >             throw new IllegalArgumentException(
> >                     String.format("Tombstone must be true if null
> value"));
> >         }
> >         this.topic = topic;
> >         this.partition = partition;
> >         this.tombstone = tombstone;
> >         this.key = key;
> >         this.value = value;
> >         this.timestamp = timestamp;
> >     }
> >
> >     public ProducerRecord(String topic, Integer partition, Long
> timestamp,
> > K key, V value) {
> >         this(topic, partition, value == null, timestamp, key, value);
> >     }
> > ________________________________________
> > From: Ewen Cheslack-Postava <ew...@confluent.io>
> > Sent: Sunday, December 11, 2016 5:45 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > It seemed like this was addressed in the migration section, wasn't it?
> The
> > V2 vs V3 requests and the fact that the broker will downgrade the message
> > format (losing zero copy) if you issues a V2 request to a broker using V3
> > format handles compatibility, does it not? The existing consumers won't
> see
> > the extra metadata in the value, but they will get a null instead and
> treat
> > it as a tombstone. Certainly there is a performance impact, but it seemed
> > compatible.
> >
> > I'm worried about this though:
> >
> >
> >    - *NOTE *: With the new version of producer client using
> ProduceRequest
> >    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and null
> > value
> >    should not be allowed as the older version of consumer using
> > FetchRequest
> >    V2 will think of this as a tombstone when its actually not.
> >
> >
> > Unless I'm misunderstanding, this ends up breaking binary compatibility
> for
> > the Java API. It sounds like this suggests that passing a null value to
> the
> > existing ProducerRecord constructors would generate an exception since
> you
> > didn't explicitly enable the tombstone (via whatever new constructor is
> > provided). But that means you can't swap in a newer client jar without
> > recompiling and get the same behavior if your app deletes keys using the
> > current approach because your app will start throwing exceptions. Maybe
> > this is a tradeoff we're ok with, but we've tried pretty hard to avoid
> > breaking compatibility like this so far -- it makes picking up bug fixes
> in
> > the clients harder for users of frameworks that have to pin to earlier
> > default versions for compatibility.
> >
> > But then later the KIP says:
> >
> >
> >    - The old constructors for ProducerRecord without this parameter will
> be
> >    kept but updated so that their default behaviour if setting null value
> > will
> >    be the tombstone will be set to true to keep existing behaviour.
> >
> >
> > So maybe I am misinterpreting something.
> >
> > And just a nit re: motivation section, item 6 would be clearer for a
> union
> > schema with null (or optional schemas in other formats), e.g. [null,
> > string], in which case losing the schema truly is losing information
> > (whereas null is already the only valid value for a pure null schema).
> >
> > -Ewen
> >
> >
> > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Good point this detail is missing in the KIP write up. Ive added this
> > now.
> > >
> > > Essentially simply just upgrading the clients we do not expect any
> client
> > > app code change needed.
> > >
> > > Cheers
> > > Mike
> > > ________________________________________
> > > From: Jay Kreps <ja...@confluent.io>
> > > Sent: Saturday, December 10, 2016 10:51 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > > Michael,
> > >
> > > The compatibility section goes through the migration path, but isn't
> the
> > > bigger compatibility issue with existing apps? There are many (probably
> > > thousands) of apps in production that use this feature and send null to
> > > mean delete. It seems like this would break compatibility with them,
> and
> > > they would have to be rewritten to right?
> > >
> > > -Jay
> > >
> > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > 4) On v3 we honour the tombstone. As such we expect it to be set
> > > correctly
> > > > as per the KIP.
> > > >
> > > > 4.1) why would we want to produce an error when its v3? This is the
> > exact
> > > > purpose to support non-null tombstone’s
> > > > 4.2) again here im unclear on the question on the v3, produce request
> > we
> > > > expect the tombstone flag to be set correctly.
> > > >
> > > > 4.4) compaction only occurs on compacted topics, the bit makes no
> > > > difference and not looked at on un-compacted (time/size based
> > eviction).
> > > >
> > > >
> > > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> > > >
> > > >     Hi, Michael,
> > > >
> > > >     4. Then, I think I misunderstood this point. Could you document
> the
> > > >     following points in the wiki?
> > > >     4.1 If producer V3 sets tombstone, but provides a non-null value,
> > > does
> > > > the
> > > >     send() get an error or does the producer automatically set the
> > value
> > > to
> > > >     null?
> > > >     4.2 If producer V3 doesn't set tombstone, but provides a null
> > value,
> > > > does
> > > >     the send() get an error or does the producer automatically sets
> the
> > > >     tombstone?
> > > >     4.3 Does the broker only expect messages that (a) have no
> tombstone
> > > and
> > > >     non-null value; (b) have tombstone and null value and reject the
> > > > messages
> > > >     with an error code otherwise?
> > > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on
> > > not?
> > > >
> > > >     Thanks,
> > > >
> > > >     Jun
> > > >
> > > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > >     wrote:
> > > >
> > > >     > Not at all.  This only acts on compacted topics just as what
> > occurs
> > > > today
> > > >     >
> > > >     > Sent using OWA for iPhone
> > > >     > ________________________________________
> > > >     > From: Jun Rao <ju...@confluent.io>
> > > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > > >     > To: dev@kafka.apache.org
> > > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >     >
> > > >     > Hi, Michael,
> > > >     >
> > > >     > 4. Hmm, does that mean the new client library can never send a
> > null
> > > > message
> > > >     > even to a regular topic? This seems like a change of the
> existing
> > > > behavior.
> > > >     >
> > > >     > Thanks,
> > > >     >
> > > >     > Jun
> > > >     >
> > > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > >     > wrote:
> > > >     >
> > > >     > > Hi Jun,
> > > >     > >
> > > >     > > Re 4) That's because we expect the tombstone value to be set
> > > > correctly if
> > > >     > > message bit is 2, as such if an older client sends in on old
> > > > message the
> > > >     > > message is upcast and the bit is set correctly. And such no
> > > longer
> > > > need
> > > >     > to
> > > >     > > check the value. Mayuresh can you confirm my thinking and
> > > > understanding
> > > >     > of
> > > >     > > what we've discussed?
> > > >     > >
> > > >     > > The second point I understand what you're getting at now my
> > > > apologies.
> > > >     > Yes
> > > >     > > this makes sense to save on touching the message, if we're
> the
> > > > only kip
> > > >     > > going in, in this release.
> > > >     > >
> > > >     > > Cheers
> > > >     > > Mike
> > > >     > >
> > > >     > > Sent using OWA for iPhone
> > > >     > > ________________________________________
> > > >     > > From: Jun Rao <ju...@confluent.io>
> > > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > > >     > > To: dev@kafka.apache.org
> > > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >     > >
> > > >     > > Hi, Michael,
> > > >     > >
> > > >     > > 4. Is this updated in the wiki? The text "If the magic byte
> on
> > > > message is
> > > >     > > 2, the broker should use the tombstone bit for log
> compaction."
> > > > doesn't
> > > >     > > seem to have changed.
> > > >     > >
> > > >     > > 2. My point is that if we change the message format just for
> > this
> > > > KIP, we
> > > >     > > should consider whether it's worth optimizing the down
> > conversion
> > > > path
> > > >     > > (i.e., decide whether a conversion is needed by just looking
> at
> > > the
> > > >     > > tombstone bit in the wrapper message) since tombstone will be
> > > used
> > > >     > rarely.
> > > >     > > However, if the message format change here is combined with
> > other
> > > > KIPs,
> > > >     > > then this optimization likely won't be needed. The latter
> > > probably
> > > > makes
> > > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> > > >     > >
> > > >     > > Other than those, +1 from me,
> > > >     > >
> > > >     > > Thanks,
> > > >     > >
> > > >     > > Jun
> > > >     > >
> > > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > >     > > wrote:
> > > >     > >
> > > >     > > > Hi Jun
> > > >     > > >
> > > >     > > > do we have your vote on this now?
> > > >     > > >
> > > >     > > > Any other concerns?
> > > >     > > >
> > > >     > > > Cheers
> > > >     > > > Mike
> > > >     > > >
> > > >     > > > Sent using OWA for iPhone
> > > >     > > > ________________________________________
> > > >     > > > From: Michael Pearce <Mi...@ig.com>
> > > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > > >     > > > To: dev@kafka.apache.org
> > > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >     > > >
> > > >     > > > Hi Jun,
> > > >     > > >
> > > >     > > > Have updated. Thanks again for the feedback.
> > > >     > > >
> > > >     > > > Agree yes we should align up when it gets to that, I assume
> > > > you’ve
> > > >     > > flagged
> > > >     > > > the same in the other KIP?
> > > >     > > >
> > > >     > > > I think for now let’s get this KIP approved, then we can
> > start
> > > > the work
> > > >     > > to
> > > >     > > > get an initial PR, then we can discuss how to align the two
> > if
> > > > needed.
> > > >     > > >
> > > >     > > > Cheers,
> > > >     > > > Mike
> > > >     > > >
> > > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> > > >     > > >
> > > >     > > >     Hi, Michael,
> > > >     > > >
> > > >     > > >     For 2), this is fine. Could you update the KIP wiki to
> > make
> > > > this
> > > >     > and
> > > >     > > > other
> > > >     > > >     points clearer? Other than that, the KIP looks good to
> > me.
> > > >     > > >
> > > >     > > >     An orthogonal thing is that there are other KIPs such
> as
> > > > KIP-98
> > > >     > that
> > > >     > > > also
> > > >     > > >     intend to change the message format. If they all get
> > > > approved, we
> > > >     > > > should
> > > >     > > >     think about whether it's better to just bump up the
> magic
> > > > byte once
> > > >     > > to
> > > >     > > >     incorporate multiple format changes like we did in
> > > > KIP-31/KIP-32.
> > > >     > > >
> > > >     > > >     Thanks,
> > > >     > > >
> > > >     > > >     Jun
> > > >     > > >
> > > >     > > >
> > > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > > >     > > Michael.Pearce@ig.com>
> > > >     > > >     wrote:
> > > >     > > >
> > > >     > > >     > Hi Jao
> > > >     > > >     >
> > > >     > > >     > Thanks for the response. Sorry for slow reply, both
> > with
> > > > personal
> > > >     > > > sickness
> > > >     > > >     > and also battling some critical issues encountered
> > since
> > > >     > upgrading
> > > >     > > to
> > > >     > > >     > 0.10.1.0
> > > >     > > >     >
> > > >     > > >     > 1) Thans for spotting, Document error where we
> branched
> > > > this KIP
> > > >     > > from
> > > >     > > >     > KIP-82, will get that removed.
> > > >     > > >     > 2) Intent is to do this just at the record message
> > level.
> > > >     > > >     > 3) Thanks for spotting, Will ensure this is
> corrected.
> > > >     > > >     > 4) As per discussion thread we will support
> tombstone +
> > > > null
> > > >     > value,
> > > >     > > >     > tombstone + non null value, no tombstone + null
> value.
> > > >     > > >     > 5) I believe this was in the discussion thread,
> > @Mayuresh
> > > > is this
> > > >     > > >     > something we’ve overlooked? I thought we would down
> > > > convert and
> > > >     > > > remove the
> > > >     > > >     > value so the old consumer had existing behavior, or
> is
> > > > there
> > > >     > > > something we
> > > >     > > >     > haven’t thought about?
> > > >     > > >     >
> > > >     > > >     > Cheers
> > > >     > > >     > Mike
> > > >     > > >     >
> > > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
> > > wrote:
> > > >     > > >     >
> > > >     > > >     >     Hi, Michael,
> > > >     > > >     >
> > > >     > > >     >     Thanks for the KIP. A few comments below.
> > > >     > > >     >
> > > >     > > >     >     1. The message format change contains
> > "HeadersLength
> > > >     > Headers".
> > > >     > > > Is that
> > > >     > > >     >     intended?
> > > >     > > >     >
> > > >     > > >     >     2. For compressed messageset, is the tombstone
> bit
> > > > only set
> > > >     > at
> > > >     > > > the
> > > >     > > >     > shallow
> > > >     > > >     >     level? Do we always leave that bit in the wrapper
> > > > message
> > > >     > > unset?
> > > >     > > > An
> > > >     > > >     >     alternative is to set the tombstone bit in the
> > > wrapper
> > > > if at
> > > >     > > > least one
> > > >     > > >     >     inner message has the tombstone bit set. This
> makes
> > > > things a
> > > >     > > bit
> > > >     > > > more
> > > >     > > >     >     complicated, but we could potentially exploit
> that
> > > for
> > > >     > > > optimizing down
> > > >     > > >     >     conversion. For example, we only need to convert
> > > > messages
> > > >     > with
> > > >     > > > magic 2
> > > >     > > >     > to
> > > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > > > (conversion is
> > > >     > > > always
> > > >     > > >     > needed
> > > >     > > >     >     from magic 2 to magic 0). Not sure if the
> > > optimization
> > > > is
> > > >     > worth
> > > >     > > > the
> > > >     > > >     >     complexity though.
> > > >     > > >     >
> > > >     > > >     >     3. The referencing of the new version of
> > > >     > > > ProducerRequest/FetchRequest
> > > >     > > >     > is
> > > >     > > >     >     inconsistent (v4 vs v3). Since our convention
> > starts
> > > at
> > > >     > version
> > > >     > > > at 0, I
> > > >     > > >     >     think the new version would be 3.
> > > >     > > >     >
> > > >     > > >     >     4. "If the magic byte on message is 2, the broker
> > > > should use
> > > >     > > the
> > > >     > > >     > tombstone
> > > >     > > >     >     bit for log compaction." What about null value?
> My
> > > >     > > understanding
> > > >     > > > is
> > > >     > > >     > that
> > > >     > > >     >     null value will be treated the same as setting
> the
> > > > tombstone
> > > >     > > bit.
> > > >     > > >     >
> > > >     > > >     >     5. For the migration path, it would be useful to
> > > > describe the
> > > >     > > > down
> > > >     > > >     >     conversion path to consumers (i.e., brokers on
> > > message
> > > > format
> > > >     > > > 0.10.2
> > > >     > > >     > and
> > > >     > > >     >     consumers on older version).
> > > >     > > >     >
> > > >     > > >     >     Thanks,
> > > >     > > >     >
> > > >     > > >     >     Jun
> > > >     > > >     >
> > > >     > > >     >
> > > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > > >     > > > Michael.Pearce@ig.com
> > > >     > > >     > >
> > > >     > > >     >     wrote:
> > > >     > > >     >
> > > >     > > >     >     > Hi All,
> > > >     > > >     >     >
> > > >     > > >     >     > We have been discussing in the below thread and
> > > final
> > > >     > changes
> > > >     > > > have
> > > >     > > >     > been
> > > >     > > >     >     > made to the KIP wiki based on these
> discussions.
> > > >     > > >     >     >
> > > >     > > >     >     > We would now like to put to the vote the
> > following
> > > > KIP:
> > > >     > > >     >     > https://cwiki.apache.org/
> > > > confluence/display/KAFKA/KIP-
> > > >     > 87+-+
> > > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > > >     > > >     >     >
> > > >     > > >     >     > This kip is for having a distinct compaction
> > > > attribute
> > > >     > > > “tombstone”
> > > >     > > >     > flag
> > > >     > > >     >     > instead of relying on null value, allowing
> > non-null
> > > > value
> > > >     > > > delete
> > > >     > > >     > messages.
> > > >     > > >     >     >
> > > >     > > >     >     > Many thanks,
> > > >     > > >     >     > Michael
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> > > >     > > Michael.Pearce@ig.com>
> > > >     > > >     > wrote:
> > > >     > > >     >     >
> > > >     > > >     >     >     Hi Mayuresh,
> > > >     > > >     >     >
> > > >     > > >     >     >     LGTM. Ive just made one small adjustment
> > > > updating the
> > > >     > > wire
> > > >     > > >     > protocol to
> > > >     > > >     >     > show the magic byte bump.
> > > >     > > >     >     >
> > > >     > > >     >     >     Do we think we’re good to put to a vote? Is
> > > > there any
> > > >     > > > other bits
> > > >     > > >     >     > needing discussion?
> > > >     > > >     >     >
> > > >     > > >     >     >     Cheers
> > > >     > > >     >     >     Mike
> > > >     > > >     >     >
> > > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> > > >     > > >     > gharatmayuresh15@gmail.com>
> > > >     > > >     >     > wrote:
> > > >     > > >     >     >
> > > >     > > >     >     >         Hi Michael,
> > > >     > > >     >     >
> > > >     > > >     >     >         I have updated the migration section of
> > the
> > > > KIP.
> > > >     > Can
> > > >     > > > you
> > > >     > > >     > please
> > > >     > > >     >     > take a look?
> > > >     > > >     >     >
> > > >     > > >     >     >         Thanks,
> > > >     > > >     >     >
> > > >     > > >     >     >         Mayuresh
> > > >     > > >     >     >
> > > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM,
> Mayuresh
> > > > Gharat <
> > > >     > > >     >     > gharatmayuresh15@gmail.com
> > > >     > > >     >     >         > wrote:
> > > >     > > >     >     >
> > > >     > > >     >     >         > Hi Michael,
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > That whilst sending tombstone and non
> > > null
> > > > value,
> > > >     > > the
> > > >     > > >     > consumer
> > > >     > > >     >     > can expect
> > > >     > > >     >     >         > only to receive the non-null message
> > only
> > > > in step
> > > >     > > > (3) is
> > > >     > > >     > this
> > > >     > > >     >     > correct?
> > > >     > > >     >     >         > ---> I do agree with you here.
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > Becket, Ismael : can you guys review
> > the
> > > >     > migration
> > > >     > > > plan
> > > >     > > >     > listed
> > > >     > > >     >     > above using
> > > >     > > >     >     >         > magic byte?
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > Thanks,
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > Mayuresh
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
> > Michael
> > > > Pearce <
> > > >     > > >     >     > Michael.Pearce@ig.com>
> > > >     > > >     >     >         > wrote:
> > > >     > > >     >     >         >
> > > >     > > >     >     >         >> Many thanks for this Mayuresh. I
> don't
> > > > have any
> > > >     > > >     > objections.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> I assume we should state:
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> That whilst sending tombstone and
> non
> > > null
> > > >     > value,
> > > >     > > > the
> > > >     > > >     > consumer
> > > >     > > >     >     > can expect
> > > >     > > >     >     >         >> only to receive the non-null message
> > > only
> > > > in
> > > >     > step
> > > >     > > > (3) is
> > > >     > > >     > this
> > > >     > > >     >     > correct?
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Cheers
> > > >     > > >     >     >         >> Mike
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Sent using OWA for iPhone
> > > >     > > >     >     >         >> ______________________________
> > > __________
> > > >     > > >     >     >         >> From: Mayuresh Gharat <
> > > >     > gharatmayuresh15@gmail.com
> > > >     > > >
> > > >     > > >     >     >         >> Sent: Thursday, November 17, 2016
> > > 5:18:41
> > > > PM
> > > >     > > >     >     >         >> To: dev@kafka.apache.org
> > > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
> > > > Compaction
> > > >     > > > Tombstone
> > > >     > > >     > Flag
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Hi Ismael,
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Thanks for the explanation.
> > > >     > > >     >     >         >> Specially I like this part where in
> > you
> > > >     > mentioned
> > > >     > > > we can
> > > >     > > >     > get
> > > >     > > >     >     > rid of the
> > > >     > > >     >     >         >> older null value support for log
> > > > compaction
> > > >     > later
> > > >     > > > on,
> > > >     > > >     > here :
> > > >     > > >     >     >         >> We can't change semantics of the
> > message
> > > > format
> > > >     > > > without
> > > >     > > >     > having
> > > >     > > >     >     > a long
> > > >     > > >     >     >         >> transition period. And we can't rely
> > > >     > > >     >     >         >> on people reading documentation or
> > > acting
> > > > on a
> > > >     > > > warning for
> > > >     > > >     >     > something so
> > > >     > > >     >     >         >> fundamental. As such, my take is
> that
> > we
> > > > need to
> > > >     > > > bump the
> > > >     > > >     > magic
> > > >     > > >     >     > byte. The
> > > >     > > >     >     >         >> good news is
> > > >     > > >     >     >         >> that we don't have to support all
> > > versions
> > > >     > > forever.
> > > >     > > > We
> > > >     > > >     > have
> > > >     > > >     >     > said that we
> > > >     > > >     >     >         >> will support direct upgrades for 2
> > > years.
> > > > That
> > > >     > > > means that
> > > >     > > >     >     > message format
> > > >     > > >     >     >         >> version n could, in theory, be
> > removed 2
> > > > years
> > > >     > > > after the
> > > >     > > >     > it's
> > > >     > > >     >     > introduced.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Just a heads up, I would like to
> > mention
> > > > that
> > > >     > even
> > > >     > > > without
> > > >     > > >     >     > bumping magic
> > > >     > > >     >     >         >> byte, we will *NOT* loose zero copy
> as
> > > in
> > > > the
> > > >     > > > client(x+1)
> > > >     > > >     > in my
> > > >     > > >     >     >         >> explanation
> > > >     > > >     >     >         >> above will convert internally a null
> > > > value to
> > > >     > > have a
> > > >     > > >     > tombstone
> > > >     > > >     >     > bit set and
> > > >     > > >     >     >         >> a tombstone bit set to have a null
> > value
> > > >     > > > automatically
> > > >     > > >     >     > internally and by
> > > >     > > >     >     >         >> the time we move to version (x+2),
> the
> > > > clients
> > > >     > > > would have
> > > >     > > >     >     > upgraded.
> > > >     > > >     >     >         >> Obviously if we support a request
> from
> > > >     > > consumer(x),
> > > >     > > > we
> > > >     > > >     > will
> > > >     > > >     >     > loose zero
> > > >     > > >     >     >         >> copy
> > > >     > > >     >     >         >> but that is the same case with magic
> > > byte.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> But if magic byte bump makes life
> > easier
> > > > for
> > > >     > > > transition
> > > >     > > >     > for the
> > > >     > > >     >     > above
> > > >     > > >     >     >         >> reasons that you explained, I am OK
> > with
> > > > it
> > > >     > since
> > > >     > > > we are
> > > >     > > >     > going
> > > >     > > >     >     > to meet the
> > > >     > > >     >     >         >> end goal down the road :)
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> On a side note can we update the doc
> > > here
> > > > on
> > > >     > magic
> > > >     > > > byte
> > > >     > > >     > to say
> > > >     > > >     >     > that "*it
> > > >     > > >     >     >         >> should be bumped whenever the
> message
> > > > format is
> > > >     > > > changed
> > > >     > > >     > or the
> > > >     > > >     >     >         >> interpretation of message format
> > (usage
> > > > of the
> > > >     > > > reserved
> > > >     > > >     > bits as
> > > >     > > >     >     > well) is
> > > >     > > >     >     >         >> changed*".
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Hi Michael,
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Here is the update plan that we
> > > discussed
> > > >     > offline
> > > >     > > >     > yesterday :
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Currently the magic-byte which
> > > > corresponds to
> > > >     > the
> > > >     > > >     >     > "message.format.version"
> > > >     > > >     >     >         >> is set to 1.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> 1) On broker it will be set to 1
> > > > initially.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> 2) When a producer client sends a
> > > message
> > > > with
> > > >     > > > magic-byte
> > > >     > > >     > = 2,
> > > >     > > >     >     > since the
> > > >     > > >     >     >         >> broker is on magic-byte = 1, we will
> > > down
> > > >     > convert
> > > >     > > > it,
> > > >     > > >     > which
> > > >     > > >     >     > means if the
> > > >     > > >     >     >         >> tombstone bit is set, the value will
> > be
> > > > set to
> > > >     > > > null. A
> > > >     > > >     > consumer
> > > >     > > >     >     >         >> understanding magic-byte = 1, will
> > still
> > > > work
> > > >     > with
> > > >     > > > this. A
> > > >     > > >     >     > consumer
> > > >     > > >     >     >         >> working
> > > >     > > >     >     >         >> with magic-byte =2 will also be able
> > to
> > > >     > understand
> > > >     > > > this,
> > > >     > > >     > since
> > > >     > > >     >     > it
> > > >     > > >     >     >         >> understands the tombstone.
> > > >     > > >     >     >         >> Now there is still the question of
> > > > supporting a
> > > >     > > >     > non-tombstone
> > > >     > > >     >     > and null
> > > >     > > >     >     >         >> value from producer client with
> > > > magic-byte = 2.*
> > > >     > > (I
> > > >     > > > am
> > > >     > > >     > not sure
> > > >     > > >     >     > if we
> > > >     > > >     >     >         >> should support this. Ismael/Becket
> can
> > > > comment
> > > >     > > > here)*
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> 3) When almost all the clients have
> > > > upgraded,
> > > >     > the
> > > >     > > >     >     > message.format.version
> > > >     > > >     >     >         >> on
> > > >     > > >     >     >         >> the broker can be changed to 2,
> where
> > in
> > > > the
> > > >     > down
> > > >     > > >     > conversion in
> > > >     > > >     >     > the above
> > > >     > > >     >     >         >> step will not happen. If at this
> point
> > > we
> > > > get a
> > > >     > > > consumer
> > > >     > > >     >     > request from a
> > > >     > > >     >     >         >> older consumer, we might have to
> down
> > > > convert
> > > >     > > where
> > > >     > > > in we
> > > >     > > >     > loose
> > > >     > > >     >     > zero copy,
> > > >     > > >     >     >         >> but these cases should be rare.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Becket can you review this plan and
> > add
> > > > more
> > > >     > > > details if I
> > > >     > > >     > have
> > > >     > > >     >     >         >> missed/wronged something, before we
> > put
> > > > it on
> > > >     > KIP.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Thanks,
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> Mayuresh
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
> > > Michael
> > > >     > Pearce <
> > > >     > > >     >     > Michael.Pearce@ig.com>
> > > >     > > >     >     >         >> wrote:
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> > Thanks guys, for discussing this
> > > > offline and
> > > >     > > > getting
> > > >     > > >     > some
> > > >     > > >     >     > consensus.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > So its clear for myself and others
> > > what
> > > > is
> > > >     > > > proposed now
> > > >     > > >     > (i
> > > >     > > >     >     > think i
> > > >     > > >     >     >         >> > understand, but want to make sure)
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Could i ask either directly update
> > the
> > > > kip to
> > > >     > > > detail the
> > > >     > > >     >     > migration
> > > >     > > >     >     >         >> > strategy, or (re-)state your
> offline
> > > > discussed
> > > >     > > and
> > > >     > > >     > agreed
> > > >     > > >     >     > migration
> > > >     > > >     >     >         >> > strategy based on a magic byte is
> in
> > > > this
> > > >     > > thread.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > The main original driver for the
> KIP
> > > > was to
> > > >     > > > support
> > > >     > > >     >     > compaction where
> > > >     > > >     >     >         >> value
> > > >     > > >     >     >         >> > isn't null, based off the
> > discussions
> > > on
> > > >     > KIP-82
> > > >     > > > thread.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > We should be able to support
> > > > non-tombstone +
> > > >     > > null
> > > >     > > > value
> > > >     > > >     > by the
> > > >     > > >     >     >         >> completion
> > > >     > > >     >     >         >> > of the KIP, as we noted when
> > > discussing
> > > > this
> > > >     > > kip,
> > > >     > > > having
> > > >     > > >     >     > logic based on
> > > >     > > >     >     >         >> a
> > > >     > > >     >     >         >> > null value isn't very clean and
> also
> > > > separates
> > > >     > > the
> > > >     > > >     > concerns.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > As discussed already though we can
> > > > split this
> > > >     > > into
> > > >     > > >     > KIP-87a
> > > >     > > >     >     > and KIP-87b
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Where we look to deliver KIP-87a
> on
> > a
> > > >     > compacted
> > > >     > > > topic
> > > >     > > >     > (to
> > > >     > > >     >     > address the
> > > >     > > >     >     >         >> > immediate issues)
> > > >     > > >     >     >         >> > * tombstone + null value
> > > >     > > >     >     >         >> > * tombstone + non-null value
> > > >     > > >     >     >         >> > * non-tombstone + non-null value
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Then we can discuss once KIP-87a
> is
> > > > completed
> > > >     > > > options
> > > >     > > >     > later
> > > >     > > >     >     > and how we
> > > >     > > >     >     >         >> > support the second part KIP-87b to
> > > > deliver:
> > > >     > > >     >     >         >> > * non-tombstone + null value
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Cheers
> > > >     > > >     >     >         >> > Mike
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > ______________________________
> > > > __________
> > > >     > > >     >     >         >> > From: Becket Qin <
> > > becket.qin@gmail.com>
> > > >     > > >     >     >         >> > Sent: Thursday, November 17, 2016
> > 1:43
> > > > AM
> > > >     > > >     >     >         >> > To: dev@kafka.apache.org
> > > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 -
> Add
> > > > Compaction
> > > >     > > >     > Tombstone Flag
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Renu, Mayuresh and I had an
> offline
> > > >     > discussion,
> > > >     > > > and
> > > >     > > >     > following
> > > >     > > >     >     > is a brief
> > > >     > > >     >     >         >> > summary.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > 1. We agreed that not bumping up
> > magic
> > > > value
> > > >     > may
> > > >     > > > result
> > > >     > > >     > in
> > > >     > > >     >     > losing zero
> > > >     > > >     >     >         >> copy
> > > >     > > >     >     >         >> > during migration.
> > > >     > > >     >     >         >> > 2. Given that bumping up magic
> value
> > > is
> > > > almost
> > > >     > > > free and
> > > >     > > >     > has
> > > >     > > >     >     > benefit of
> > > >     > > >     >     >         >> > avoiding potential performance
> > issue.
> > > > It is
> > > >     > > > probably
> > > >     > > >     > worth
> > > >     > > >     >     > doing.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > One issue we still need to think
> > about
> > > > is
> > > >     > > whether
> > > >     > > > we
> > > >     > > >     > want to
> > > >     > > >     >     > support a
> > > >     > > >     >     >         >> > non-tombstone message with null
> > value.
> > > >     > > >     >     >         >> > Currently it is not supported by
> > > Kafka.
> > > > If we
> > > >     > > > allow a
> > > >     > > >     >     > non-tombstone null
> > > >     > > >     >     >         >> > value message to exist after
> KIP-87.
> > > The
> > > >     > problem
> > > >     > > > is
> > > >     > > >     > that such
> > > >     > > >     >     > message
> > > >     > > >     >     >         >> will
> > > >     > > >     >     >         >> > not be supported by the consumers
> > > prior
> > > > to
> > > >     > > KIP-87.
> > > >     > > >     > Because a
> > > >     > > >     >     > null value
> > > >     > > >     >     >         >> > will always be interpreted to a
> > > > tombstone.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > One option is that we keep the
> > current
> > > > way,
> > > >     > i.e.
> > > >     > > > do not
> > > >     > > >     >     > support such
> > > >     > > >     >     >         >> > message. It would be good to know
> if
> > > > there is
> > > >     > a
> > > >     > > >     > concrete use
> > > >     > > >     >     > case for
> > > >     > > >     >     >         >> such
> > > >     > > >     >     >         >> > message. If there is not, we can
> > > > probably just
> > > >     > > not
> > > >     > > >     > support it.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > Thanks,
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > JIangjie (Becket) Qin
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> > > > Mayuresh
> > > >     > > Gharat <
> > > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> > > >     > > >     >     >         >> > > wrote:
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >> > > Hi Ismael,
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > This is something I can think of
> > for
> > > >     > migration
> > > >     > > > plan:
> > > >     > > >     >     >         >> > > So the migration plan can look
> > > > something
> > > >     > like
> > > >     > > > this,
> > > >     > > >     > with up
> > > >     > > >     >     >         >> conversion :
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > 1) Currently lets say we have
> > Broker
> > > > at
> > > >     > > version
> > > >     > > > x.
> > > >     > > >     >     >         >> > > 2) Currently we have clients at
> > > > version x.
> > > >     > > >     >     >         >> > > 3) a) We move the version to
> > > > Broker(x+1) :
> > > >     > > > supports
> > > >     > > >     > both
> > > >     > > >     >     > tombstone and
> > > >     > > >     >     >         >> > null
> > > >     > > >     >     >         >> > > for log compaction.
> > > >     > > >     >     >         >> > >     b) We upgrade the client to
> > > > version
> > > >     > > > client(x+1) :
> > > >     > > >     > if in
> > > >     > > >     >     > the
> > > >     > > >     >     >         >> producer
> > > >     > > >     >     >         >> > > client(x+1) the value is set to
> > > null,
> > > > we
> > > >     > will
> > > >     > > >     > automatically
> > > >     > > >     >     > set the
> > > >     > > >     >     >         >> > > Tombstone bit internally. If the
> > > > producer
> > > >     > > > client(x+1)
> > > >     > > >     > sets
> > > >     > > >     >     > the
> > > >     > > >     >     >         >> tombstone
> > > >     > > >     >     >         >> > > itself, well and good. For
> > producer
> > > >     > client(x),
> > > >     > > > the
> > > >     > > >     > broker
> > > >     > > >     >     > will up
> > > >     > > >     >     >         >> convert
> > > >     > > >     >     >         >> > > to have the tombstone bit.
> > > > Broker(x+1) is
> > > >     > > > supporting
> > > >     > > >     > both.
> > > >     > > >     >     > Consumer
> > > >     > > >     >     >         >> > > client(x+1) will be aware of
> this
> > > and
> > > > should
> > > >     > > be
> > > >     > > > able
> > > >     > > >     > to
> > > >     > > >     >     > handle this.
> > > >     > > >     >     >         >> For
> > > >     > > >     >     >         >> > > consumer client(x) we will down
> > > > convert the
> > > >     > > > message
> > > >     > > >     > on the
> > > >     > > >     >     > broker
> > > >     > > >     >     >         >> side.
> > > >     > > >     >     >         >> > >     c) At this point we will
> have
> > to
> > > >     > specify a
> > > >     > > >     > warning or
> > > >     > > >     >     > clearly
> > > >     > > >     >     >         >> specify
> > > >     > > >     >     >         >> > > in docs that this behavior is
> > about
> > > > to be
> > > >     > > > changed for
> > > >     > > >     > log
> > > >     > > >     >     > compaction.
> > > >     > > >     >     >         >> > > 4) a) In next release of the
> > > > Broker(x+2), we
> > > >     > > > say that
> > > >     > > >     > only
> > > >     > > >     >     > Tombstone
> > > >     > > >     >     >         >> is
> > > >     > > >     >     >         >> > > used for log compaction on the
> > > Broker
> > > > side.
> > > >     > > >     > Clients(x+1)
> > > >     > > >     >     > still is
> > > >     > > >     >     >         >> > > supported.
> > > >     > > >     >     >         >> > >     b) We upgrade the client to
> > > > version
> > > >     > > > client(x+2) :
> > > >     > > >     > if
> > > >     > > >     >     > value is set
> > > >     > > >     >     >         >> to
> > > >     > > >     >     >         >> > > null, tombstone will not be set
> > > >     > automatically.
> > > >     > > > The
> > > >     > > >     > client
> > > >     > > >     >     > will have to
> > > >     > > >     >     >         >> > call
> > > >     > > >     >     >         >> > > setTombstone() to actually set
> the
> > > >     > tombstone.
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > We should compare this migration
> > > plan
> > > > with
> > > >     > the
> > > >     > > >     > migration
> > > >     > > >     >     > plan for
> > > >     > > >     >     >         >> magic
> > > >     > > >     >     >         >> > > byte bump and do whatever looks
> > > good.
> > > >     > > >     >     >         >> > > I am just worried that if we go
> > down
> > > > magic
> > > >     > > byte
> > > >     > > > route,
> > > >     > > >     >     > unless I am
> > > >     > > >     >     >         >> > missing
> > > >     > > >     >     >         >> > > something, it sounds like kafka
> > will
> > > > be
> > > >     > stuck
> > > >     > > > with
> > > >     > > >     >     > supporting both
> > > >     > > >     >     >         >> null
> > > >     > > >     >     >         >> > > value and tombstone bit for log
> > > > compaction
> > > >     > for
> > > >     > > > life
> > > >     > > >     > long,
> > > >     > > >     >     > which does
> > > >     > > >     >     >         >> not
> > > >     > > >     >     >         >> > > look like a good end state.
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > Thanks,
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > Mayuresh
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
> > > > Mayuresh
> > > >     > > > Gharat <
> > > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> > > >     > > >     >     >         >> > > > wrote:
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > > Hi Ismael,
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > That's a very good point
> which I
> > > > might
> > > >     > have
> > > >     > > > not
> > > >     > > >     >     > considered earlier.
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > Here is a plan that I can
> think
> > > of:
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > Stage 1) The broker from now
> on,
> > > up
> > > >     > converts
> > > >     > > > the
> > > >     > > >     > message
> > > >     > > >     >     > to have the
> > > >     > > >     >     >         >> > > > tombstone marker. The log
> > > compaction
> > > >     > thread
> > > >     > > > does log
> > > >     > > >     >     > compaction
> > > >     > > >     >     >         >> based
> > > >     > > >     >     >         >> > on
> > > >     > > >     >     >         >> > > > both null and tombstone
> marker.
> > > > This is
> > > >     > our
> > > >     > > >     > transition
> > > >     > > >     >     > period.
> > > >     > > >     >     >         >> > > > Stage 2) The next release we
> > only
> > > > say that
> > > >     > > log
> > > >     > > >     > compaction
> > > >     > > >     >     > is based
> > > >     > > >     >     >         >> on
> > > >     > > >     >     >         >> > > > tombstone marker. (Open source
> > > > kafka makes
> > > >     > > > this as a
> > > >     > > >     >     > policy). By
> > > >     > > >     >     >         >> this
> > > >     > > >     >     >         >> > > time,
> > > >     > > >     >     >         >> > > > the organization which is
> moving
> > > to
> > > > this
> > > >     > > > release
> > > >     > > >     > will be
> > > >     > > >     >     > sure that
> > > >     > > >     >     >         >> they
> > > >     > > >     >     >         >> > > > have gone through the entire
> > > > transition
> > > >     > > > period.
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > My only goal of doing this is
> > that
> > > > Kafka
> > > >     > > > clearly
> > > >     > > >     >     > specifies the end
> > > >     > > >     >     >         >> > state
> > > >     > > >     >     >         >> > > > about what log compaction
> means
> > > (is
> > > > it
> > > >     > null
> > > >     > > > value
> > > >     > > >     > or a
> > > >     > > >     >     > tombstone
> > > >     > > >     >     >         >> > marker,
> > > >     > > >     >     >         >> > > > but not both).
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > What do you think?
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > Thanks,
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > Mayuresh
> > > >     > > >     >     >         >> > > > .
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17
> AM,
> > > > Ismael
> > > >     > > Juma <
> > > >     > > >     >     > ismael@juma.me.uk>
> > > >     > > >     >     >         >> > wrote:
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > >> One comment below.
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08
> > PM,
> > > > Mayuresh
> > > >     > > > Gharat <
> > > >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> > > >     > > >     >     >         >> > > >> > wrote:
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >> >    - If we don't bump up
> the
> > > > magic
> > > >     > byte,
> > > >     > > > on the
> > > >     > > >     > broker
> > > >     > > >     >     > side, the
> > > >     > > >     >     >         >> > > broker
> > > >     > > >     >     >         >> > > >> >    will always have to look
> > at
> > > > both
> > > >     > > > tombstone
> > > >     > > >     > bit and
> > > >     > > >     >     > the value
> > > >     > > >     >     >         >> when
> > > >     > > >     >     >         >> > > do
> > > >     > > >     >     >         >> > > >> the
> > > >     > > >     >     >         >> > > >> >    compaction. Assuming we
> do
> > > > not bump
> > > >     > up
> > > >     > > > the
> > > >     > > >     > magic
> > > >     > > >     >     > byte,
> > > >     > > >     >     >         >> > > >> >    imagine the broker sees
> a
> > > > message
> > > >     > > which
> > > >     > > > does
> > > >     > > >     > not
> > > >     > > >     >     > have a
> > > >     > > >     >     >         >> tombstone
> > > >     > > >     >     >         >> > > bit
> > > >     > > >     >     >         >> > > >> >    set. The broker does not
> > > know
> > > > when
> > > >     > the
> > > >     > > >     > message was
> > > >     > > >     >     > produced
> > > >     > > >     >     >         >> (i.e.
> > > >     > > >     >     >         >> > > >> > whether
> > > >     > > >     >     >         >> > > >> >    the message has been up
> > > > converted or
> > > >     > > > not), it
> > > >     > > >     > has
> > > >     > > >     >     > to take a
> > > >     > > >     >     >         >> > further
> > > >     > > >     >     >         >> > > >> > look at
> > > >     > > >     >     >         >> > > >> >    the value to see if it
> is
> > > > null or
> > > >     > not
> > > >     > > in
> > > >     > > >     > order to
> > > >     > > >     >     > determine
> > > >     > > >     >     >         >> if it
> > > >     > > >     >     >         >> > > is
> > > >     > > >     >     >         >> > > >> a
> > > >     > > >     >     >         >> > > >> >    tombstone. The same
> logic
> > > has
> > > > to be
> > > >     > > put
> > > >     > > > on the
> > > >     > > >     >     > consumer as
> > > >     > > >     >     >         >> well
> > > >     > > >     >     >         >> > > >> because
> > > >     > > >     >     >         >> > > >> > the
> > > >     > > >     >     >         >> > > >> >    consumer does not know
> if
> > > the
> > > >     > message
> > > >     > > > has
> > > >     > > >     > been up
> > > >     > > >     >     > converted or
> > > >     > > >     >     >         >> > not.
> > > >     > > >     >     >         >> > > >> >       - If we upconvert
> while
> > > >     > appending,
> > > >     > > > this is
> > > >     > > >     > not
> > > >     > > >     >     > the case,
> > > >     > > >     >     >         >> > right?
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >> If I understand you
> correctly,
> > > > this is
> > > >     > not
> > > >     > > >     > sufficient
> > > >     > > >     >     > because the
> > > >     > > >     >     >         >> log
> > > >     > > >     >     >         >> > > may
> > > >     > > >     >     >         >> > > >> have messages appended before
> > it
> > > > was
> > > >     > > > upgraded to
> > > >     > > >     > include
> > > >     > > >     >     > KIP-87.
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >> Ismael
> > > >     > > >     >     >         >> > > >>
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > > > --
> > > >     > > >     >     >         >> > > > -Regards,
> > > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> > > >     > > >     >     >         >> > > > (862) 250-7125
> > > >     > > >     >     >         >> > > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > > --
> > > >     > > >     >     >         >> > > -Regards,
> > > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > > >     > > >     >     >         >> > > (862) 250-7125
> > > >     > > >     >     >         >> > >
> > > >     > > >     >     >         >> > The information contained in this
> > > email
> > > > is
> > > >     > > > strictly
> > > >     > > >     >     > confidential and for
> > > >     > > >     >     >         >> > the use of the addressee only,
> > unless
> > > >     > otherwise
> > > >     > > >     > indicated. If
> > > >     > > >     >     > you are
> > > >     > > >     >     >         >> not
> > > >     > > >     >     >         >> > the intended recipient, please do
> > not
> > > > read,
> > > >     > > copy,
> > > >     > > > use or
> > > >     > > >     >     > disclose to
> > > >     > > >     >     >         >> others
> > > >     > > >     >     >         >> > this message or any attachment.
> > Please
> > > > also
> > > >     > > > notify the
> > > >     > > >     > sender
> > > >     > > >     >     > by
> > > >     > > >     >     >         >> replying
> > > >     > > >     >     >         >> > to this email or by telephone
> > (+44(020
> > > > 7896
> > > >     > > 0011)
> > > >     > > > and
> > > >     > > >     > then
> > > >     > > >     >     > delete the
> > > >     > > >     >     >         >> email
> > > >     > > >     >     >         >> > and any copies of it. Opinions,
> > > > conclusion
> > > >     > (etc)
> > > >     > > > that
> > > >     > > >     > do not
> > > >     > > >     >     > relate to
> > > >     > > >     >     >         >> the
> > > >     > > >     >     >         >> > official business of this company
> > > shall
> > > > be
> > > >     > > > understood as
> > > >     > > >     >     > neither given
> > > >     > > >     >     >         >> nor
> > > >     > > >     >     >         >> > endorsed by it. IG is a trading
> name
> > > of
> > > > IG
> > > >     > > Markets
> > > >     > > >     > Limited (a
> > > >     > > >     >     > company
> > > >     > > >     >     >         >> > registered in England and Wales,
> > > company
> > > >     > number
> > > >     > > >     > 04008957) and
> > > >     > > >     >     > IG Index
> > > >     > > >     >     >         >> > Limited (a company registered in
> > > > England and
> > > >     > > > Wales,
> > > >     > > >     > company
> > > >     > > >     >     > number
> > > >     > > >     >     >         >> > 01190902). Registered address at
> > > Cannon
> > > > Bridge
> > > >     > > > House, 25
> > > >     > > >     >     > Dowgate Hill,
> > > >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
> > > Limited
> > > >     > > (register
> > > >     > > >     > number
> > > >     > > >     >     > 195355) and IG
> > > >     > > >     >     >         >> > Index Limited (register number
> > 114059)
> > > > are
> > > >     > > > authorised
> > > >     > > >     > and
> > > >     > > >     >     > regulated by
> > > >     > > >     >     >         >> the
> > > >     > > >     >     >         >> > Financial Conduct Authority.
> > > >     > > >     >     >         >> >
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >> --
> > > >     > > >     >     >         >> -Regards,
> > > >     > > >     >     >         >> Mayuresh R. Gharat
> > > >     > > >     >     >         >> (862) 250-7125
> > > >     > > >     >     >         >> The information contained in this
> > email
> > > is
> > > >     > > strictly
> > > >     > > >     >     > confidential and for
> > > >     > > >     >     >         >> the use of the addressee only,
> unless
> > > > otherwise
> > > >     > > >     > indicated. If
> > > >     > > >     >     > you are not
> > > >     > > >     >     >         >> the intended recipient, please do
> not
> > > > read,
> > > >     > copy,
> > > >     > > > use or
> > > >     > > >     >     > disclose to others
> > > >     > > >     >     >         >> this message or any attachment.
> Please
> > > > also
> > > >     > notify
> > > >     > > > the
> > > >     > > >     > sender
> > > >     > > >     >     > by replying
> > > >     > > >     >     >         >> to this email or by telephone
> (+44(020
> > > > 7896
> > > >     > 0011)
> > > >     > > > and then
> > > >     > > >     >     > delete the email
> > > >     > > >     >     >         >> and any copies of it. Opinions,
> > > > conclusion (etc)
> > > >     > > > that do
> > > >     > > >     > not
> > > >     > > >     >     > relate to the
> > > >     > > >     >     >         >> official business of this company
> > shall
> > > be
> > > >     > > > understood as
> > > >     > > >     >     > neither given nor
> > > >     > > >     >     >         >> endorsed by it. IG is a trading name
> > of
> > > IG
> > > >     > Markets
> > > >     > > >     > Limited (a
> > > >     > > >     >     > company
> > > >     > > >     >     >         >> registered in England and Wales,
> > company
> > > > number
> > > >     > > > 04008957)
> > > >     > > >     > and
> > > >     > > >     >     > IG Index
> > > >     > > >     >     >         >> Limited (a company registered in
> > England
> > > > and
> > > >     > > Wales,
> > > >     > > >     > company
> > > >     > > >     >     > number
> > > >     > > >     >     >         >> 01190902). Registered address at
> > Cannon
> > > > Bridge
> > > >     > > > House, 25
> > > >     > > >     >     > Dowgate Hill,
> > > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
> > Limited
> > > >     > (register
> > > >     > > > number
> > > >     > > >     >     > 195355) and IG
> > > >     > > >     >     >         >> Index Limited (register number
> 114059)
> > > are
> > > >     > > > authorised and
> > > >     > > >     >     > regulated by the
> > > >     > > >     >     >         >> Financial Conduct Authority.
> > > >     > > >     >     >         >>
> > > >     > > >     >     >         >
> > > >     > > >     >     >         >
> > > >     > > >     >     >         >
> > > >     > > >     >     >         > --
> > > >     > > >     >     >         > -Regards,
> > > >     > > >     >     >         > Mayuresh R. Gharat
> > > >     > > >     >     >         > (862) 250-7125
> > > >     > > >     >     >         >
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >         --
> > > >     > > >     >     >         -Regards,
> > > >     > > >     >     >         Mayuresh R. Gharat
> > > >     > > >     >     >         (862) 250-7125
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >     The information contained in this email is
> > > > strictly
> > > >     > > > confidential
> > > >     > > >     > and
> > > >     > > >     >     > for the use of the addressee only, unless
> > otherwise
> > > >     > > indicated.
> > > >     > > > If
> > > >     > > >     > you are
> > > >     > > >     >     > not the intended recipient, please do not read,
> > > > copy, use
> > > >     > or
> > > >     > > >     > disclose to
> > > >     > > >     >     > others this message or any attachment. Please
> > also
> > > > notify
> > > >     > the
> > > >     > > > sender
> > > >     > > >     > by
> > > >     > > >     >     > replying to this email or by telephone (+44(020
> > > 7896
> > > > 0011)
> > > >     > > and
> > > >     > > > then
> > > >     > > >     > delete
> > > >     > > >     >     > the email and any copies of it. Opinions,
> > > conclusion
> > > > (etc)
> > > >     > > > that do
> > > >     > > >     > not
> > > >     > > >     >     > relate to the official business of this company
> > > > shall be
> > > >     > > > understood
> > > >     > > >     > as
> > > >     > > >     >     > neither given nor endorsed by it. IG is a
> trading
> > > > name of
> > > >     > IG
> > > >     > > > Markets
> > > >     > > >     >     > Limited (a company registered in England and
> > Wales,
> > > > company
> > > >     > > > number
> > > >     > > >     >     > 04008957) and IG Index Limited (a company
> > > registered
> > > > in
> > > >     > > > England and
> > > >     > > >     > Wales,
> > > >     > > >     >     > company number 01190902). Registered address at
> > > > Cannon
> > > >     > Bridge
> > > >     > > > House,
> > > >     > > >     > 25
> > > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > > > Limited
> > > >     > > > (register
> > > >     > > >     > number
> > > >     > > >     >     > 195355) and IG Index Limited (register number
> > > > 114059) are
> > > >     > > > authorised
> > > >     > > >     > and
> > > >     > > >     >     > regulated by the Financial Conduct Authority.
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >     >
> > > >     > > >     >
> > > >     > > >     >
> > > >     > > >     > The information contained in this email is strictly
> > > > confidential
> > > >     > > and
> > > >     > > > for
> > > >     > > >     > the use of the addressee only, unless otherwise
> > > indicated.
> > > > If you
> > > >     > > > are not
> > > >     > > >     > the intended recipient, please do not read, copy, use
> > or
> > > > disclose
> > > >     > > to
> > > >     > > > others
> > > >     > > >     > this message or any attachment. Please also notify
> the
> > > > sender by
> > > >     > > > replying
> > > >     > > >     > to this email or by telephone (+44(020 7896 0011) and
> > > then
> > > > delete
> > > >     > > > the email
> > > >     > > >     > and any copies of it. Opinions, conclusion (etc) that
> > do
> > > > not
> > > >     > relate
> > > >     > > > to the
> > > >     > > >     > official business of this company shall be understood
> > as
> > > > neither
> > > >     > > > given nor
> > > >     > > >     > endorsed by it. IG is a trading name of IG Markets
> > > Limited
> > > > (a
> > > >     > > company
> > > >     > > >     > registered in England and Wales, company number
> > 04008957)
> > > > and IG
> > > >     > > > Index
> > > >     > > >     > Limited (a company registered in England and Wales,
> > > company
> > > >     > number
> > > >     > > >     > 01190902). Registered address at Cannon Bridge House,
> > 25
> > > > Dowgate
> > > >     > > > Hill,
> > > >     > > >     > London EC4R 2YA. Both IG Markets Limited (register
> > number
> > > > 195355)
> > > >     > > > and IG
> > > >     > > >     > Index Limited (register number 114059) are authorised
> > and
> > > >     > regulated
> > > >     > > > by the
> > > >     > > >     > Financial Conduct Authority.
> > > >     > > >     >
> > > >     > > >
> > > >     > > >
> > > >     > > > The information contained in this email is strictly
> > > confidential
> > > > and
> > > >     > for
> > > >     > > > the use of the addressee only, unless otherwise indicated.
> If
> > > > you are
> > > >     > not
> > > >     > > > the intended recipient, please do not read, copy, use or
> > > > disclose to
> > > >     > > others
> > > >     > > > this message or any attachment. Please also notify the
> sender
> > > by
> > > >     > replying
> > > >     > > > to this email or by telephone (+44(020 7896 0011) and then
> > > > delete the
> > > >     > > email
> > > >     > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > > relate to
> > > >     > > the
> > > >     > > > official business of this company shall be understood as
> > > neither
> > > > given
> > > >     > > nor
> > > >     > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > > company
> > > >     > > > registered in England and Wales, company number 04008957)
> and
> > > IG
> > > > Index
> > > >     > > > Limited (a company registered in England and Wales, company
> > > > number
> > > >     > > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > > Hill,
> > > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > > 195355) and
> > > >     > IG
> > > >     > > > Index Limited (register number 114059) are authorised and
> > > > regulated by
> > > >     > > the
> > > >     > > > Financial Conduct Authority.
> > > >     > > >
> > > >     > > The information contained in this email is strictly
> > confidential
> > > > and for
> > > >     > > the use of the addressee only, unless otherwise indicated. If
> > you
> > > > are not
> > > >     > > the intended recipient, please do not read, copy, use or
> > disclose
> > > > to
> > > >     > others
> > > >     > > this message or any attachment. Please also notify the sender
> > by
> > > > replying
> > > >     > > to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > > the
> > > >     > email
> > > >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > > > relate to
> > > >     > the
> > > >     > > official business of this company shall be understood as
> > neither
> > > > given
> > > >     > nor
> > > >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > > company
> > > >     > > registered in England and Wales, company number 04008957) and
> > IG
> > > > Index
> > > >     > > Limited (a company registered in England and Wales, company
> > > number
> > > >     > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > Hill,
> > > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > and IG
> > > >     > > Index Limited (register number 114059) are authorised and
> > > > regulated by
> > > >     > the
> > > >     > > Financial Conduct Authority.
> > > >     > >
> > > >     > >
> > > >     > The information contained in this email is strictly
> confidential
> > > and
> > > > for
> > > >     > the use of the addressee only, unless otherwise indicated. If
> you
> > > > are not
> > > >     > the intended recipient, please do not read, copy, use or
> disclose
> > > to
> > > > others
> > > >     > this message or any attachment. Please also notify the sender
> by
> > > > replying
> > > >     > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > > > the email
> > > >     > and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> > > > to the
> > > >     > official business of this company shall be understood as
> neither
> > > > given nor
> > > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > > >     > registered in England and Wales, company number 04008957) and
> IG
> > > > Index
> > > >     > Limited (a company registered in England and Wales, company
> > number
> > > >     > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > > > Hill,
> > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > > and IG
> > > >     > Index Limited (register number 114059) are authorised and
> > regulated
> > > > by the
> > > >     > Financial Conduct Authority.
> > > >     >
> > > >     >
> > > >
> > > >
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jay,

Why wouldn't that work, the tombstone value is only looked at by the broker, on a topic configured for compaction as such is benign on non compacted topics. This is as much as sending a null value currently


Regards
Mike



Sent using OWA for iPhone
________________________________________
From: Jay Kreps <ja...@confluent.io>
Sent: Sunday, December 11, 2016 8:58:53 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hey Michael,

I'm not quite sure that works as that would translate ALL null values to
tombstones, even for non-compacted topics that use null as an acceptable
value sent by the producer and expected by the consumer.

-Jay

On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Ewen,
>
> I think the easiest way to show this is with code.
>
> As you can see we keep the existing behaviour for code/binaries calling
> the pre-existing constructors, whereby if the value is null the tombstone
> is set to true.
>
> Regards
> Mike
>
>
>
>     /**
>      * Creates a record with a specified timestamp to be sent to a
> specified topic and partition
>      *
>      * @param topic The topic the record will be appended to
>      * @param partition The partition to which the record should be sent
>      * @param timestamp The timestamp of the record
>      * @param tombstone if the record should be treated as a tombstone if
> the topic is compacted
>      * @param key The key that will be included in the record
>      * @param value The record contents
>      */
>     public ProducerRecord(String topic, Integer partition, Boolean
> tombstone, Long timestamp, K key, V value) {
>         if (topic == null)
>             throw new IllegalArgumentException("Topic cannot be null.");
>         if (timestamp != null && timestamp < 0)
>             throw new IllegalArgumentException(
>                     String.format("Invalid timestamp: %d. Timestamp should
> always be non-negative or null.", timestamp));
>         if (partition != null && partition < 0)
>             throw new IllegalArgumentException(
>                     String.format("Invalid partition: %d. Partition number
> should always be non-negative or null.", partition));
>         if (!tombstone && value == null){
>             throw new IllegalArgumentException(
>                     String.format("Tombstone must be true if null value"));
>         }
>         this.topic = topic;
>         this.partition = partition;
>         this.tombstone = tombstone;
>         this.key = key;
>         this.value = value;
>         this.timestamp = timestamp;
>     }
>
>     public ProducerRecord(String topic, Integer partition, Long timestamp,
> K key, V value) {
>         this(topic, partition, value == null, timestamp, key, value);
>     }
> ________________________________________
> From: Ewen Cheslack-Postava <ew...@confluent.io>
> Sent: Sunday, December 11, 2016 5:45 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> It seemed like this was addressed in the migration section, wasn't it? The
> V2 vs V3 requests and the fact that the broker will downgrade the message
> format (losing zero copy) if you issues a V2 request to a broker using V3
> format handles compatibility, does it not? The existing consumers won't see
> the extra metadata in the value, but they will get a null instead and treat
> it as a tombstone. Certainly there is a performance impact, but it seemed
> compatible.
>
> I'm worried about this though:
>
>
>    - *NOTE *: With the new version of producer client using ProduceRequest
>    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and null
> value
>    should not be allowed as the older version of consumer using
> FetchRequest
>    V2 will think of this as a tombstone when its actually not.
>
>
> Unless I'm misunderstanding, this ends up breaking binary compatibility for
> the Java API. It sounds like this suggests that passing a null value to the
> existing ProducerRecord constructors would generate an exception since you
> didn't explicitly enable the tombstone (via whatever new constructor is
> provided). But that means you can't swap in a newer client jar without
> recompiling and get the same behavior if your app deletes keys using the
> current approach because your app will start throwing exceptions. Maybe
> this is a tradeoff we're ok with, but we've tried pretty hard to avoid
> breaking compatibility like this so far -- it makes picking up bug fixes in
> the clients harder for users of frameworks that have to pin to earlier
> default versions for compatibility.
>
> But then later the KIP says:
>
>
>    - The old constructors for ProducerRecord without this parameter will be
>    kept but updated so that their default behaviour if setting null value
> will
>    be the tombstone will be set to true to keep existing behaviour.
>
>
> So maybe I am misinterpreting something.
>
> And just a nit re: motivation section, item 6 would be clearer for a union
> schema with null (or optional schemas in other formats), e.g. [null,
> string], in which case losing the schema truly is losing information
> (whereas null is already the only valid value for a pure null schema).
>
> -Ewen
>
>
> On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jay,
> >
> > Good point this detail is missing in the KIP write up. Ive added this
> now.
> >
> > Essentially simply just upgrading the clients we do not expect any client
> > app code change needed.
> >
> > Cheers
> > Mike
> > ________________________________________
> > From: Jay Kreps <ja...@confluent.io>
> > Sent: Saturday, December 10, 2016 10:51 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Michael,
> >
> > The compatibility section goes through the migration path, but isn't the
> > bigger compatibility issue with existing apps? There are many (probably
> > thousands) of apps in production that use this feature and send null to
> > mean delete. It seems like this would break compatibility with them, and
> > they would have to be rewritten to right?
> >
> > -Jay
> >
> > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jun,
> > >
> > > 4) On v3 we honour the tombstone. As such we expect it to be set
> > correctly
> > > as per the KIP.
> > >
> > > 4.1) why would we want to produce an error when its v3? This is the
> exact
> > > purpose to support non-null tombstone’s
> > > 4.2) again here im unclear on the question on the v3, produce request
> we
> > > expect the tombstone flag to be set correctly.
> > >
> > > 4.4) compaction only occurs on compacted topics, the bit makes no
> > > difference and not looked at on un-compacted (time/size based
> eviction).
> > >
> > >
> > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> > >
> > >     Hi, Michael,
> > >
> > >     4. Then, I think I misunderstood this point. Could you document the
> > >     following points in the wiki?
> > >     4.1 If producer V3 sets tombstone, but provides a non-null value,
> > does
> > > the
> > >     send() get an error or does the producer automatically set the
> value
> > to
> > >     null?
> > >     4.2 If producer V3 doesn't set tombstone, but provides a null
> value,
> > > does
> > >     the send() get an error or does the producer automatically sets the
> > >     tombstone?
> > >     4.3 Does the broker only expect messages that (a) have no tombstone
> > and
> > >     non-null value; (b) have tombstone and null value and reject the
> > > messages
> > >     with an error code otherwise?
> > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on
> > not?
> > >
> > >     Thanks,
> > >
> > >     Jun
> > >
> > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > >     wrote:
> > >
> > >     > Not at all.  This only acts on compacted topics just as what
> occurs
> > > today
> > >     >
> > >     > Sent using OWA for iPhone
> > >     > ________________________________________
> > >     > From: Jun Rao <ju...@confluent.io>
> > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > >     > To: dev@kafka.apache.org
> > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     >
> > >     > Hi, Michael,
> > >     >
> > >     > 4. Hmm, does that mean the new client library can never send a
> null
> > > message
> > >     > even to a regular topic? This seems like a change of the existing
> > > behavior.
> > >     >
> > >     > Thanks,
> > >     >
> > >     > Jun
> > >     >
> > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > >     > wrote:
> > >     >
> > >     > > Hi Jun,
> > >     > >
> > >     > > Re 4) That's because we expect the tombstone value to be set
> > > correctly if
> > >     > > message bit is 2, as such if an older client sends in on old
> > > message the
> > >     > > message is upcast and the bit is set correctly. And such no
> > longer
> > > need
> > >     > to
> > >     > > check the value. Mayuresh can you confirm my thinking and
> > > understanding
> > >     > of
> > >     > > what we've discussed?
> > >     > >
> > >     > > The second point I understand what you're getting at now my
> > > apologies.
> > >     > Yes
> > >     > > this makes sense to save on touching the message, if we're the
> > > only kip
> > >     > > going in, in this release.
> > >     > >
> > >     > > Cheers
> > >     > > Mike
> > >     > >
> > >     > > Sent using OWA for iPhone
> > >     > > ________________________________________
> > >     > > From: Jun Rao <ju...@confluent.io>
> > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > >     > > To: dev@kafka.apache.org
> > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     > >
> > >     > > Hi, Michael,
> > >     > >
> > >     > > 4. Is this updated in the wiki? The text "If the magic byte on
> > > message is
> > >     > > 2, the broker should use the tombstone bit for log compaction."
> > > doesn't
> > >     > > seem to have changed.
> > >     > >
> > >     > > 2. My point is that if we change the message format just for
> this
> > > KIP, we
> > >     > > should consider whether it's worth optimizing the down
> conversion
> > > path
> > >     > > (i.e., decide whether a conversion is needed by just looking at
> > the
> > >     > > tombstone bit in the wrapper message) since tombstone will be
> > used
> > >     > rarely.
> > >     > > However, if the message format change here is combined with
> other
> > > KIPs,
> > >     > > then this optimization likely won't be needed. The latter
> > probably
> > > makes
> > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> > >     > >
> > >     > > Other than those, +1 from me,
> > >     > >
> > >     > > Thanks,
> > >     > >
> > >     > > Jun
> > >     > >
> > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > >     > > wrote:
> > >     > >
> > >     > > > Hi Jun
> > >     > > >
> > >     > > > do we have your vote on this now?
> > >     > > >
> > >     > > > Any other concerns?
> > >     > > >
> > >     > > > Cheers
> > >     > > > Mike
> > >     > > >
> > >     > > > Sent using OWA for iPhone
> > >     > > > ________________________________________
> > >     > > > From: Michael Pearce <Mi...@ig.com>
> > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > >     > > > To: dev@kafka.apache.org
> > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     > > >
> > >     > > > Hi Jun,
> > >     > > >
> > >     > > > Have updated. Thanks again for the feedback.
> > >     > > >
> > >     > > > Agree yes we should align up when it gets to that, I assume
> > > you’ve
> > >     > > flagged
> > >     > > > the same in the other KIP?
> > >     > > >
> > >     > > > I think for now let’s get this KIP approved, then we can
> start
> > > the work
> > >     > > to
> > >     > > > get an initial PR, then we can discuss how to align the two
> if
> > > needed.
> > >     > > >
> > >     > > > Cheers,
> > >     > > > Mike
> > >     > > >
> > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> > >     > > >
> > >     > > >     Hi, Michael,
> > >     > > >
> > >     > > >     For 2), this is fine. Could you update the KIP wiki to
> make
> > > this
> > >     > and
> > >     > > > other
> > >     > > >     points clearer? Other than that, the KIP looks good to
> me.
> > >     > > >
> > >     > > >     An orthogonal thing is that there are other KIPs such as
> > > KIP-98
> > >     > that
> > >     > > > also
> > >     > > >     intend to change the message format. If they all get
> > > approved, we
> > >     > > > should
> > >     > > >     think about whether it's better to just bump up the magic
> > > byte once
> > >     > > to
> > >     > > >     incorporate multiple format changes like we did in
> > > KIP-31/KIP-32.
> > >     > > >
> > >     > > >     Thanks,
> > >     > > >
> > >     > > >     Jun
> > >     > > >
> > >     > > >
> > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > >     > > Michael.Pearce@ig.com>
> > >     > > >     wrote:
> > >     > > >
> > >     > > >     > Hi Jao
> > >     > > >     >
> > >     > > >     > Thanks for the response. Sorry for slow reply, both
> with
> > > personal
> > >     > > > sickness
> > >     > > >     > and also battling some critical issues encountered
> since
> > >     > upgrading
> > >     > > to
> > >     > > >     > 0.10.1.0
> > >     > > >     >
> > >     > > >     > 1) Thans for spotting, Document error where we branched
> > > this KIP
> > >     > > from
> > >     > > >     > KIP-82, will get that removed.
> > >     > > >     > 2) Intent is to do this just at the record message
> level.
> > >     > > >     > 3) Thanks for spotting, Will ensure this is corrected.
> > >     > > >     > 4) As per discussion thread we will support tombstone +
> > > null
> > >     > value,
> > >     > > >     > tombstone + non null value, no tombstone + null value.
> > >     > > >     > 5) I believe this was in the discussion thread,
> @Mayuresh
> > > is this
> > >     > > >     > something we’ve overlooked? I thought we would down
> > > convert and
> > >     > > > remove the
> > >     > > >     > value so the old consumer had existing behavior, or is
> > > there
> > >     > > > something we
> > >     > > >     > haven’t thought about?
> > >     > > >     >
> > >     > > >     > Cheers
> > >     > > >     > Mike
> > >     > > >     >
> > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
> > wrote:
> > >     > > >     >
> > >     > > >     >     Hi, Michael,
> > >     > > >     >
> > >     > > >     >     Thanks for the KIP. A few comments below.
> > >     > > >     >
> > >     > > >     >     1. The message format change contains
> "HeadersLength
> > >     > Headers".
> > >     > > > Is that
> > >     > > >     >     intended?
> > >     > > >     >
> > >     > > >     >     2. For compressed messageset, is the tombstone bit
> > > only set
> > >     > at
> > >     > > > the
> > >     > > >     > shallow
> > >     > > >     >     level? Do we always leave that bit in the wrapper
> > > message
> > >     > > unset?
> > >     > > > An
> > >     > > >     >     alternative is to set the tombstone bit in the
> > wrapper
> > > if at
> > >     > > > least one
> > >     > > >     >     inner message has the tombstone bit set. This makes
> > > things a
> > >     > > bit
> > >     > > > more
> > >     > > >     >     complicated, but we could potentially exploit that
> > for
> > >     > > > optimizing down
> > >     > > >     >     conversion. For example, we only need to convert
> > > messages
> > >     > with
> > >     > > > magic 2
> > >     > > >     > to
> > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > > (conversion is
> > >     > > > always
> > >     > > >     > needed
> > >     > > >     >     from magic 2 to magic 0). Not sure if the
> > optimization
> > > is
> > >     > worth
> > >     > > > the
> > >     > > >     >     complexity though.
> > >     > > >     >
> > >     > > >     >     3. The referencing of the new version of
> > >     > > > ProducerRequest/FetchRequest
> > >     > > >     > is
> > >     > > >     >     inconsistent (v4 vs v3). Since our convention
> starts
> > at
> > >     > version
> > >     > > > at 0, I
> > >     > > >     >     think the new version would be 3.
> > >     > > >     >
> > >     > > >     >     4. "If the magic byte on message is 2, the broker
> > > should use
> > >     > > the
> > >     > > >     > tombstone
> > >     > > >     >     bit for log compaction." What about null value? My
> > >     > > understanding
> > >     > > > is
> > >     > > >     > that
> > >     > > >     >     null value will be treated the same as setting the
> > > tombstone
> > >     > > bit.
> > >     > > >     >
> > >     > > >     >     5. For the migration path, it would be useful to
> > > describe the
> > >     > > > down
> > >     > > >     >     conversion path to consumers (i.e., brokers on
> > message
> > > format
> > >     > > > 0.10.2
> > >     > > >     > and
> > >     > > >     >     consumers on older version).
> > >     > > >     >
> > >     > > >     >     Thanks,
> > >     > > >     >
> > >     > > >     >     Jun
> > >     > > >     >
> > >     > > >     >
> > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > >     > > > Michael.Pearce@ig.com
> > >     > > >     > >
> > >     > > >     >     wrote:
> > >     > > >     >
> > >     > > >     >     > Hi All,
> > >     > > >     >     >
> > >     > > >     >     > We have been discussing in the below thread and
> > final
> > >     > changes
> > >     > > > have
> > >     > > >     > been
> > >     > > >     >     > made to the KIP wiki based on these discussions.
> > >     > > >     >     >
> > >     > > >     >     > We would now like to put to the vote the
> following
> > > KIP:
> > >     > > >     >     > https://cwiki.apache.org/
> > > confluence/display/KAFKA/KIP-
> > >     > 87+-+
> > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > >     > > >     >     >
> > >     > > >     >     > This kip is for having a distinct compaction
> > > attribute
> > >     > > > “tombstone”
> > >     > > >     > flag
> > >     > > >     >     > instead of relying on null value, allowing
> non-null
> > > value
> > >     > > > delete
> > >     > > >     > messages.
> > >     > > >     >     >
> > >     > > >     >     > Many thanks,
> > >     > > >     >     > Michael
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> > >     > > Michael.Pearce@ig.com>
> > >     > > >     > wrote:
> > >     > > >     >     >
> > >     > > >     >     >     Hi Mayuresh,
> > >     > > >     >     >
> > >     > > >     >     >     LGTM. Ive just made one small adjustment
> > > updating the
> > >     > > wire
> > >     > > >     > protocol to
> > >     > > >     >     > show the magic byte bump.
> > >     > > >     >     >
> > >     > > >     >     >     Do we think we’re good to put to a vote? Is
> > > there any
> > >     > > > other bits
> > >     > > >     >     > needing discussion?
> > >     > > >     >     >
> > >     > > >     >     >     Cheers
> > >     > > >     >     >     Mike
> > >     > > >     >     >
> > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> > >     > > >     > gharatmayuresh15@gmail.com>
> > >     > > >     >     > wrote:
> > >     > > >     >     >
> > >     > > >     >     >         Hi Michael,
> > >     > > >     >     >
> > >     > > >     >     >         I have updated the migration section of
> the
> > > KIP.
> > >     > Can
> > >     > > > you
> > >     > > >     > please
> > >     > > >     >     > take a look?
> > >     > > >     >     >
> > >     > > >     >     >         Thanks,
> > >     > > >     >     >
> > >     > > >     >     >         Mayuresh
> > >     > > >     >     >
> > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh
> > > Gharat <
> > >     > > >     >     > gharatmayuresh15@gmail.com
> > >     > > >     >     >         > wrote:
> > >     > > >     >     >
> > >     > > >     >     >         > Hi Michael,
> > >     > > >     >     >         >
> > >     > > >     >     >         > That whilst sending tombstone and non
> > null
> > > value,
> > >     > > the
> > >     > > >     > consumer
> > >     > > >     >     > can expect
> > >     > > >     >     >         > only to receive the non-null message
> only
> > > in step
> > >     > > > (3) is
> > >     > > >     > this
> > >     > > >     >     > correct?
> > >     > > >     >     >         > ---> I do agree with you here.
> > >     > > >     >     >         >
> > >     > > >     >     >         > Becket, Ismael : can you guys review
> the
> > >     > migration
> > >     > > > plan
> > >     > > >     > listed
> > >     > > >     >     > above using
> > >     > > >     >     >         > magic byte?
> > >     > > >     >     >         >
> > >     > > >     >     >         > Thanks,
> > >     > > >     >     >         >
> > >     > > >     >     >         > Mayuresh
> > >     > > >     >     >         >
> > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
> Michael
> > > Pearce <
> > >     > > >     >     > Michael.Pearce@ig.com>
> > >     > > >     >     >         > wrote:
> > >     > > >     >     >         >
> > >     > > >     >     >         >> Many thanks for this Mayuresh. I don't
> > > have any
> > >     > > >     > objections.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> I assume we should state:
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> That whilst sending tombstone and non
> > null
> > >     > value,
> > >     > > > the
> > >     > > >     > consumer
> > >     > > >     >     > can expect
> > >     > > >     >     >         >> only to receive the non-null message
> > only
> > > in
> > >     > step
> > >     > > > (3) is
> > >     > > >     > this
> > >     > > >     >     > correct?
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Cheers
> > >     > > >     >     >         >> Mike
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Sent using OWA for iPhone
> > >     > > >     >     >         >> ______________________________
> > __________
> > >     > > >     >     >         >> From: Mayuresh Gharat <
> > >     > gharatmayuresh15@gmail.com
> > >     > > >
> > >     > > >     >     >         >> Sent: Thursday, November 17, 2016
> > 5:18:41
> > > PM
> > >     > > >     >     >         >> To: dev@kafka.apache.org
> > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
> > > Compaction
> > >     > > > Tombstone
> > >     > > >     > Flag
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Hi Ismael,
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Thanks for the explanation.
> > >     > > >     >     >         >> Specially I like this part where in
> you
> > >     > mentioned
> > >     > > > we can
> > >     > > >     > get
> > >     > > >     >     > rid of the
> > >     > > >     >     >         >> older null value support for log
> > > compaction
> > >     > later
> > >     > > > on,
> > >     > > >     > here :
> > >     > > >     >     >         >> We can't change semantics of the
> message
> > > format
> > >     > > > without
> > >     > > >     > having
> > >     > > >     >     > a long
> > >     > > >     >     >         >> transition period. And we can't rely
> > >     > > >     >     >         >> on people reading documentation or
> > acting
> > > on a
> > >     > > > warning for
> > >     > > >     >     > something so
> > >     > > >     >     >         >> fundamental. As such, my take is that
> we
> > > need to
> > >     > > > bump the
> > >     > > >     > magic
> > >     > > >     >     > byte. The
> > >     > > >     >     >         >> good news is
> > >     > > >     >     >         >> that we don't have to support all
> > versions
> > >     > > forever.
> > >     > > > We
> > >     > > >     > have
> > >     > > >     >     > said that we
> > >     > > >     >     >         >> will support direct upgrades for 2
> > years.
> > > That
> > >     > > > means that
> > >     > > >     >     > message format
> > >     > > >     >     >         >> version n could, in theory, be
> removed 2
> > > years
> > >     > > > after the
> > >     > > >     > it's
> > >     > > >     >     > introduced.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Just a heads up, I would like to
> mention
> > > that
> > >     > even
> > >     > > > without
> > >     > > >     >     > bumping magic
> > >     > > >     >     >         >> byte, we will *NOT* loose zero copy as
> > in
> > > the
> > >     > > > client(x+1)
> > >     > > >     > in my
> > >     > > >     >     >         >> explanation
> > >     > > >     >     >         >> above will convert internally a null
> > > value to
> > >     > > have a
> > >     > > >     > tombstone
> > >     > > >     >     > bit set and
> > >     > > >     >     >         >> a tombstone bit set to have a null
> value
> > >     > > > automatically
> > >     > > >     >     > internally and by
> > >     > > >     >     >         >> the time we move to version (x+2), the
> > > clients
> > >     > > > would have
> > >     > > >     >     > upgraded.
> > >     > > >     >     >         >> Obviously if we support a request from
> > >     > > consumer(x),
> > >     > > > we
> > >     > > >     > will
> > >     > > >     >     > loose zero
> > >     > > >     >     >         >> copy
> > >     > > >     >     >         >> but that is the same case with magic
> > byte.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> But if magic byte bump makes life
> easier
> > > for
> > >     > > > transition
> > >     > > >     > for the
> > >     > > >     >     > above
> > >     > > >     >     >         >> reasons that you explained, I am OK
> with
> > > it
> > >     > since
> > >     > > > we are
> > >     > > >     > going
> > >     > > >     >     > to meet the
> > >     > > >     >     >         >> end goal down the road :)
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> On a side note can we update the doc
> > here
> > > on
> > >     > magic
> > >     > > > byte
> > >     > > >     > to say
> > >     > > >     >     > that "*it
> > >     > > >     >     >         >> should be bumped whenever the message
> > > format is
> > >     > > > changed
> > >     > > >     > or the
> > >     > > >     >     >         >> interpretation of message format
> (usage
> > > of the
> > >     > > > reserved
> > >     > > >     > bits as
> > >     > > >     >     > well) is
> > >     > > >     >     >         >> changed*".
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Hi Michael,
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Here is the update plan that we
> > discussed
> > >     > offline
> > >     > > >     > yesterday :
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Currently the magic-byte which
> > > corresponds to
> > >     > the
> > >     > > >     >     > "message.format.version"
> > >     > > >     >     >         >> is set to 1.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> 1) On broker it will be set to 1
> > > initially.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> 2) When a producer client sends a
> > message
> > > with
> > >     > > > magic-byte
> > >     > > >     > = 2,
> > >     > > >     >     > since the
> > >     > > >     >     >         >> broker is on magic-byte = 1, we will
> > down
> > >     > convert
> > >     > > > it,
> > >     > > >     > which
> > >     > > >     >     > means if the
> > >     > > >     >     >         >> tombstone bit is set, the value will
> be
> > > set to
> > >     > > > null. A
> > >     > > >     > consumer
> > >     > > >     >     >         >> understanding magic-byte = 1, will
> still
> > > work
> > >     > with
> > >     > > > this. A
> > >     > > >     >     > consumer
> > >     > > >     >     >         >> working
> > >     > > >     >     >         >> with magic-byte =2 will also be able
> to
> > >     > understand
> > >     > > > this,
> > >     > > >     > since
> > >     > > >     >     > it
> > >     > > >     >     >         >> understands the tombstone.
> > >     > > >     >     >         >> Now there is still the question of
> > > supporting a
> > >     > > >     > non-tombstone
> > >     > > >     >     > and null
> > >     > > >     >     >         >> value from producer client with
> > > magic-byte = 2.*
> > >     > > (I
> > >     > > > am
> > >     > > >     > not sure
> > >     > > >     >     > if we
> > >     > > >     >     >         >> should support this. Ismael/Becket can
> > > comment
> > >     > > > here)*
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> 3) When almost all the clients have
> > > upgraded,
> > >     > the
> > >     > > >     >     > message.format.version
> > >     > > >     >     >         >> on
> > >     > > >     >     >         >> the broker can be changed to 2, where
> in
> > > the
> > >     > down
> > >     > > >     > conversion in
> > >     > > >     >     > the above
> > >     > > >     >     >         >> step will not happen. If at this point
> > we
> > > get a
> > >     > > > consumer
> > >     > > >     >     > request from a
> > >     > > >     >     >         >> older consumer, we might have to down
> > > convert
> > >     > > where
> > >     > > > in we
> > >     > > >     > loose
> > >     > > >     >     > zero copy,
> > >     > > >     >     >         >> but these cases should be rare.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Becket can you review this plan and
> add
> > > more
> > >     > > > details if I
> > >     > > >     > have
> > >     > > >     >     >         >> missed/wronged something, before we
> put
> > > it on
> > >     > KIP.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Thanks,
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Mayuresh
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
> > Michael
> > >     > Pearce <
> > >     > > >     >     > Michael.Pearce@ig.com>
> > >     > > >     >     >         >> wrote:
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> > Thanks guys, for discussing this
> > > offline and
> > >     > > > getting
> > >     > > >     > some
> > >     > > >     >     > consensus.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > So its clear for myself and others
> > what
> > > is
> > >     > > > proposed now
> > >     > > >     > (i
> > >     > > >     >     > think i
> > >     > > >     >     >         >> > understand, but want to make sure)
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Could i ask either directly update
> the
> > > kip to
> > >     > > > detail the
> > >     > > >     >     > migration
> > >     > > >     >     >         >> > strategy, or (re-)state your offline
> > > discussed
> > >     > > and
> > >     > > >     > agreed
> > >     > > >     >     > migration
> > >     > > >     >     >         >> > strategy based on a magic byte is in
> > > this
> > >     > > thread.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > The main original driver for the KIP
> > > was to
> > >     > > > support
> > >     > > >     >     > compaction where
> > >     > > >     >     >         >> value
> > >     > > >     >     >         >> > isn't null, based off the
> discussions
> > on
> > >     > KIP-82
> > >     > > > thread.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > We should be able to support
> > > non-tombstone +
> > >     > > null
> > >     > > > value
> > >     > > >     > by the
> > >     > > >     >     >         >> completion
> > >     > > >     >     >         >> > of the KIP, as we noted when
> > discussing
> > > this
> > >     > > kip,
> > >     > > > having
> > >     > > >     >     > logic based on
> > >     > > >     >     >         >> a
> > >     > > >     >     >         >> > null value isn't very clean and also
> > > separates
> > >     > > the
> > >     > > >     > concerns.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > As discussed already though we can
> > > split this
> > >     > > into
> > >     > > >     > KIP-87a
> > >     > > >     >     > and KIP-87b
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Where we look to deliver KIP-87a on
> a
> > >     > compacted
> > >     > > > topic
> > >     > > >     > (to
> > >     > > >     >     > address the
> > >     > > >     >     >         >> > immediate issues)
> > >     > > >     >     >         >> > * tombstone + null value
> > >     > > >     >     >         >> > * tombstone + non-null value
> > >     > > >     >     >         >> > * non-tombstone + non-null value
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Then we can discuss once KIP-87a is
> > > completed
> > >     > > > options
> > >     > > >     > later
> > >     > > >     >     > and how we
> > >     > > >     >     >         >> > support the second part KIP-87b to
> > > deliver:
> > >     > > >     >     >         >> > * non-tombstone + null value
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Cheers
> > >     > > >     >     >         >> > Mike
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > ______________________________
> > > __________
> > >     > > >     >     >         >> > From: Becket Qin <
> > becket.qin@gmail.com>
> > >     > > >     >     >         >> > Sent: Thursday, November 17, 2016
> 1:43
> > > AM
> > >     > > >     >     >         >> > To: dev@kafka.apache.org
> > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add
> > > Compaction
> > >     > > >     > Tombstone Flag
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Renu, Mayuresh and I had an offline
> > >     > discussion,
> > >     > > > and
> > >     > > >     > following
> > >     > > >     >     > is a brief
> > >     > > >     >     >         >> > summary.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > 1. We agreed that not bumping up
> magic
> > > value
> > >     > may
> > >     > > > result
> > >     > > >     > in
> > >     > > >     >     > losing zero
> > >     > > >     >     >         >> copy
> > >     > > >     >     >         >> > during migration.
> > >     > > >     >     >         >> > 2. Given that bumping up magic value
> > is
> > > almost
> > >     > > > free and
> > >     > > >     > has
> > >     > > >     >     > benefit of
> > >     > > >     >     >         >> > avoiding potential performance
> issue.
> > > It is
> > >     > > > probably
> > >     > > >     > worth
> > >     > > >     >     > doing.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > One issue we still need to think
> about
> > > is
> > >     > > whether
> > >     > > > we
> > >     > > >     > want to
> > >     > > >     >     > support a
> > >     > > >     >     >         >> > non-tombstone message with null
> value.
> > >     > > >     >     >         >> > Currently it is not supported by
> > Kafka.
> > > If we
> > >     > > > allow a
> > >     > > >     >     > non-tombstone null
> > >     > > >     >     >         >> > value message to exist after KIP-87.
> > The
> > >     > problem
> > >     > > > is
> > >     > > >     > that such
> > >     > > >     >     > message
> > >     > > >     >     >         >> will
> > >     > > >     >     >         >> > not be supported by the consumers
> > prior
> > > to
> > >     > > KIP-87.
> > >     > > >     > Because a
> > >     > > >     >     > null value
> > >     > > >     >     >         >> > will always be interpreted to a
> > > tombstone.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > One option is that we keep the
> current
> > > way,
> > >     > i.e.
> > >     > > > do not
> > >     > > >     >     > support such
> > >     > > >     >     >         >> > message. It would be good to know if
> > > there is
> > >     > a
> > >     > > >     > concrete use
> > >     > > >     >     > case for
> > >     > > >     >     >         >> such
> > >     > > >     >     >         >> > message. If there is not, we can
> > > probably just
> > >     > > not
> > >     > > >     > support it.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Thanks,
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > JIangjie (Becket) Qin
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> > > Mayuresh
> > >     > > Gharat <
> > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> > >     > > >     >     >         >> > > wrote:
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > > Hi Ismael,
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > This is something I can think of
> for
> > >     > migration
> > >     > > > plan:
> > >     > > >     >     >         >> > > So the migration plan can look
> > > something
> > >     > like
> > >     > > > this,
> > >     > > >     > with up
> > >     > > >     >     >         >> conversion :
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > 1) Currently lets say we have
> Broker
> > > at
> > >     > > version
> > >     > > > x.
> > >     > > >     >     >         >> > > 2) Currently we have clients at
> > > version x.
> > >     > > >     >     >         >> > > 3) a) We move the version to
> > > Broker(x+1) :
> > >     > > > supports
> > >     > > >     > both
> > >     > > >     >     > tombstone and
> > >     > > >     >     >         >> > null
> > >     > > >     >     >         >> > > for log compaction.
> > >     > > >     >     >         >> > >     b) We upgrade the client to
> > > version
> > >     > > > client(x+1) :
> > >     > > >     > if in
> > >     > > >     >     > the
> > >     > > >     >     >         >> producer
> > >     > > >     >     >         >> > > client(x+1) the value is set to
> > null,
> > > we
> > >     > will
> > >     > > >     > automatically
> > >     > > >     >     > set the
> > >     > > >     >     >         >> > > Tombstone bit internally. If the
> > > producer
> > >     > > > client(x+1)
> > >     > > >     > sets
> > >     > > >     >     > the
> > >     > > >     >     >         >> tombstone
> > >     > > >     >     >         >> > > itself, well and good. For
> producer
> > >     > client(x),
> > >     > > > the
> > >     > > >     > broker
> > >     > > >     >     > will up
> > >     > > >     >     >         >> convert
> > >     > > >     >     >         >> > > to have the tombstone bit.
> > > Broker(x+1) is
> > >     > > > supporting
> > >     > > >     > both.
> > >     > > >     >     > Consumer
> > >     > > >     >     >         >> > > client(x+1) will be aware of this
> > and
> > > should
> > >     > > be
> > >     > > > able
> > >     > > >     > to
> > >     > > >     >     > handle this.
> > >     > > >     >     >         >> For
> > >     > > >     >     >         >> > > consumer client(x) we will down
> > > convert the
> > >     > > > message
> > >     > > >     > on the
> > >     > > >     >     > broker
> > >     > > >     >     >         >> side.
> > >     > > >     >     >         >> > >     c) At this point we will have
> to
> > >     > specify a
> > >     > > >     > warning or
> > >     > > >     >     > clearly
> > >     > > >     >     >         >> specify
> > >     > > >     >     >         >> > > in docs that this behavior is
> about
> > > to be
> > >     > > > changed for
> > >     > > >     > log
> > >     > > >     >     > compaction.
> > >     > > >     >     >         >> > > 4) a) In next release of the
> > > Broker(x+2), we
> > >     > > > say that
> > >     > > >     > only
> > >     > > >     >     > Tombstone
> > >     > > >     >     >         >> is
> > >     > > >     >     >         >> > > used for log compaction on the
> > Broker
> > > side.
> > >     > > >     > Clients(x+1)
> > >     > > >     >     > still is
> > >     > > >     >     >         >> > > supported.
> > >     > > >     >     >         >> > >     b) We upgrade the client to
> > > version
> > >     > > > client(x+2) :
> > >     > > >     > if
> > >     > > >     >     > value is set
> > >     > > >     >     >         >> to
> > >     > > >     >     >         >> > > null, tombstone will not be set
> > >     > automatically.
> > >     > > > The
> > >     > > >     > client
> > >     > > >     >     > will have to
> > >     > > >     >     >         >> > call
> > >     > > >     >     >         >> > > setTombstone() to actually set the
> > >     > tombstone.
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > We should compare this migration
> > plan
> > > with
> > >     > the
> > >     > > >     > migration
> > >     > > >     >     > plan for
> > >     > > >     >     >         >> magic
> > >     > > >     >     >         >> > > byte bump and do whatever looks
> > good.
> > >     > > >     >     >         >> > > I am just worried that if we go
> down
> > > magic
> > >     > > byte
> > >     > > > route,
> > >     > > >     >     > unless I am
> > >     > > >     >     >         >> > missing
> > >     > > >     >     >         >> > > something, it sounds like kafka
> will
> > > be
> > >     > stuck
> > >     > > > with
> > >     > > >     >     > supporting both
> > >     > > >     >     >         >> null
> > >     > > >     >     >         >> > > value and tombstone bit for log
> > > compaction
> > >     > for
> > >     > > > life
> > >     > > >     > long,
> > >     > > >     >     > which does
> > >     > > >     >     >         >> not
> > >     > > >     >     >         >> > > look like a good end state.
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > Thanks,
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > Mayuresh
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
> > > Mayuresh
> > >     > > > Gharat <
> > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> > >     > > >     >     >         >> > > > wrote:
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > > Hi Ismael,
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > That's a very good point which I
> > > might
> > >     > have
> > >     > > > not
> > >     > > >     >     > considered earlier.
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > Here is a plan that I can think
> > of:
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > Stage 1) The broker from now on,
> > up
> > >     > converts
> > >     > > > the
> > >     > > >     > message
> > >     > > >     >     > to have the
> > >     > > >     >     >         >> > > > tombstone marker. The log
> > compaction
> > >     > thread
> > >     > > > does log
> > >     > > >     >     > compaction
> > >     > > >     >     >         >> based
> > >     > > >     >     >         >> > on
> > >     > > >     >     >         >> > > > both null and tombstone marker.
> > > This is
> > >     > our
> > >     > > >     > transition
> > >     > > >     >     > period.
> > >     > > >     >     >         >> > > > Stage 2) The next release we
> only
> > > say that
> > >     > > log
> > >     > > >     > compaction
> > >     > > >     >     > is based
> > >     > > >     >     >         >> on
> > >     > > >     >     >         >> > > > tombstone marker. (Open source
> > > kafka makes
> > >     > > > this as a
> > >     > > >     >     > policy). By
> > >     > > >     >     >         >> this
> > >     > > >     >     >         >> > > time,
> > >     > > >     >     >         >> > > > the organization which is moving
> > to
> > > this
> > >     > > > release
> > >     > > >     > will be
> > >     > > >     >     > sure that
> > >     > > >     >     >         >> they
> > >     > > >     >     >         >> > > > have gone through the entire
> > > transition
> > >     > > > period.
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > My only goal of doing this is
> that
> > > Kafka
> > >     > > > clearly
> > >     > > >     >     > specifies the end
> > >     > > >     >     >         >> > state
> > >     > > >     >     >         >> > > > about what log compaction means
> > (is
> > > it
> > >     > null
> > >     > > > value
> > >     > > >     > or a
> > >     > > >     >     > tombstone
> > >     > > >     >     >         >> > marker,
> > >     > > >     >     >         >> > > > but not both).
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > What do you think?
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > Thanks,
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > Mayuresh
> > >     > > >     >     >         >> > > > .
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM,
> > > Ismael
> > >     > > Juma <
> > >     > > >     >     > ismael@juma.me.uk>
> > >     > > >     >     >         >> > wrote:
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > >> One comment below.
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08
> PM,
> > > Mayuresh
> > >     > > > Gharat <
> > >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> > >     > > >     >     >         >> > > >> > wrote:
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >> >    - If we don't bump up the
> > > magic
> > >     > byte,
> > >     > > > on the
> > >     > > >     > broker
> > >     > > >     >     > side, the
> > >     > > >     >     >         >> > > broker
> > >     > > >     >     >         >> > > >> >    will always have to look
> at
> > > both
> > >     > > > tombstone
> > >     > > >     > bit and
> > >     > > >     >     > the value
> > >     > > >     >     >         >> when
> > >     > > >     >     >         >> > > do
> > >     > > >     >     >         >> > > >> the
> > >     > > >     >     >         >> > > >> >    compaction. Assuming we do
> > > not bump
> > >     > up
> > >     > > > the
> > >     > > >     > magic
> > >     > > >     >     > byte,
> > >     > > >     >     >         >> > > >> >    imagine the broker sees a
> > > message
> > >     > > which
> > >     > > > does
> > >     > > >     > not
> > >     > > >     >     > have a
> > >     > > >     >     >         >> tombstone
> > >     > > >     >     >         >> > > bit
> > >     > > >     >     >         >> > > >> >    set. The broker does not
> > know
> > > when
> > >     > the
> > >     > > >     > message was
> > >     > > >     >     > produced
> > >     > > >     >     >         >> (i.e.
> > >     > > >     >     >         >> > > >> > whether
> > >     > > >     >     >         >> > > >> >    the message has been up
> > > converted or
> > >     > > > not), it
> > >     > > >     > has
> > >     > > >     >     > to take a
> > >     > > >     >     >         >> > further
> > >     > > >     >     >         >> > > >> > look at
> > >     > > >     >     >         >> > > >> >    the value to see if it is
> > > null or
> > >     > not
> > >     > > in
> > >     > > >     > order to
> > >     > > >     >     > determine
> > >     > > >     >     >         >> if it
> > >     > > >     >     >         >> > > is
> > >     > > >     >     >         >> > > >> a
> > >     > > >     >     >         >> > > >> >    tombstone. The same logic
> > has
> > > to be
> > >     > > put
> > >     > > > on the
> > >     > > >     >     > consumer as
> > >     > > >     >     >         >> well
> > >     > > >     >     >         >> > > >> because
> > >     > > >     >     >         >> > > >> > the
> > >     > > >     >     >         >> > > >> >    consumer does not know if
> > the
> > >     > message
> > >     > > > has
> > >     > > >     > been up
> > >     > > >     >     > converted or
> > >     > > >     >     >         >> > not.
> > >     > > >     >     >         >> > > >> >       - If we upconvert while
> > >     > appending,
> > >     > > > this is
> > >     > > >     > not
> > >     > > >     >     > the case,
> > >     > > >     >     >         >> > right?
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >> If I understand you correctly,
> > > this is
> > >     > not
> > >     > > >     > sufficient
> > >     > > >     >     > because the
> > >     > > >     >     >         >> log
> > >     > > >     >     >         >> > > may
> > >     > > >     >     >         >> > > >> have messages appended before
> it
> > > was
> > >     > > > upgraded to
> > >     > > >     > include
> > >     > > >     >     > KIP-87.
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >> Ismael
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > --
> > >     > > >     >     >         >> > > > -Regards,
> > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> > >     > > >     >     >         >> > > > (862) 250-7125
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > --
> > >     > > >     >     >         >> > > -Regards,
> > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > >     > > >     >     >         >> > > (862) 250-7125
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > The information contained in this
> > email
> > > is
> > >     > > > strictly
> > >     > > >     >     > confidential and for
> > >     > > >     >     >         >> > the use of the addressee only,
> unless
> > >     > otherwise
> > >     > > >     > indicated. If
> > >     > > >     >     > you are
> > >     > > >     >     >         >> not
> > >     > > >     >     >         >> > the intended recipient, please do
> not
> > > read,
> > >     > > copy,
> > >     > > > use or
> > >     > > >     >     > disclose to
> > >     > > >     >     >         >> others
> > >     > > >     >     >         >> > this message or any attachment.
> Please
> > > also
> > >     > > > notify the
> > >     > > >     > sender
> > >     > > >     >     > by
> > >     > > >     >     >         >> replying
> > >     > > >     >     >         >> > to this email or by telephone
> (+44(020
> > > 7896
> > >     > > 0011)
> > >     > > > and
> > >     > > >     > then
> > >     > > >     >     > delete the
> > >     > > >     >     >         >> email
> > >     > > >     >     >         >> > and any copies of it. Opinions,
> > > conclusion
> > >     > (etc)
> > >     > > > that
> > >     > > >     > do not
> > >     > > >     >     > relate to
> > >     > > >     >     >         >> the
> > >     > > >     >     >         >> > official business of this company
> > shall
> > > be
> > >     > > > understood as
> > >     > > >     >     > neither given
> > >     > > >     >     >         >> nor
> > >     > > >     >     >         >> > endorsed by it. IG is a trading name
> > of
> > > IG
> > >     > > Markets
> > >     > > >     > Limited (a
> > >     > > >     >     > company
> > >     > > >     >     >         >> > registered in England and Wales,
> > company
> > >     > number
> > >     > > >     > 04008957) and
> > >     > > >     >     > IG Index
> > >     > > >     >     >         >> > Limited (a company registered in
> > > England and
> > >     > > > Wales,
> > >     > > >     > company
> > >     > > >     >     > number
> > >     > > >     >     >         >> > 01190902). Registered address at
> > Cannon
> > > Bridge
> > >     > > > House, 25
> > >     > > >     >     > Dowgate Hill,
> > >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
> > Limited
> > >     > > (register
> > >     > > >     > number
> > >     > > >     >     > 195355) and IG
> > >     > > >     >     >         >> > Index Limited (register number
> 114059)
> > > are
> > >     > > > authorised
> > >     > > >     > and
> > >     > > >     >     > regulated by
> > >     > > >     >     >         >> the
> > >     > > >     >     >         >> > Financial Conduct Authority.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> --
> > >     > > >     >     >         >> -Regards,
> > >     > > >     >     >         >> Mayuresh R. Gharat
> > >     > > >     >     >         >> (862) 250-7125
> > >     > > >     >     >         >> The information contained in this
> email
> > is
> > >     > > strictly
> > >     > > >     >     > confidential and for
> > >     > > >     >     >         >> the use of the addressee only, unless
> > > otherwise
> > >     > > >     > indicated. If
> > >     > > >     >     > you are not
> > >     > > >     >     >         >> the intended recipient, please do not
> > > read,
> > >     > copy,
> > >     > > > use or
> > >     > > >     >     > disclose to others
> > >     > > >     >     >         >> this message or any attachment. Please
> > > also
> > >     > notify
> > >     > > > the
> > >     > > >     > sender
> > >     > > >     >     > by replying
> > >     > > >     >     >         >> to this email or by telephone (+44(020
> > > 7896
> > >     > 0011)
> > >     > > > and then
> > >     > > >     >     > delete the email
> > >     > > >     >     >         >> and any copies of it. Opinions,
> > > conclusion (etc)
> > >     > > > that do
> > >     > > >     > not
> > >     > > >     >     > relate to the
> > >     > > >     >     >         >> official business of this company
> shall
> > be
> > >     > > > understood as
> > >     > > >     >     > neither given nor
> > >     > > >     >     >         >> endorsed by it. IG is a trading name
> of
> > IG
> > >     > Markets
> > >     > > >     > Limited (a
> > >     > > >     >     > company
> > >     > > >     >     >         >> registered in England and Wales,
> company
> > > number
> > >     > > > 04008957)
> > >     > > >     > and
> > >     > > >     >     > IG Index
> > >     > > >     >     >         >> Limited (a company registered in
> England
> > > and
> > >     > > Wales,
> > >     > > >     > company
> > >     > > >     >     > number
> > >     > > >     >     >         >> 01190902). Registered address at
> Cannon
> > > Bridge
> > >     > > > House, 25
> > >     > > >     >     > Dowgate Hill,
> > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
> Limited
> > >     > (register
> > >     > > > number
> > >     > > >     >     > 195355) and IG
> > >     > > >     >     >         >> Index Limited (register number 114059)
> > are
> > >     > > > authorised and
> > >     > > >     >     > regulated by the
> > >     > > >     >     >         >> Financial Conduct Authority.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >
> > >     > > >     >     >         >
> > >     > > >     >     >         >
> > >     > > >     >     >         > --
> > >     > > >     >     >         > -Regards,
> > >     > > >     >     >         > Mayuresh R. Gharat
> > >     > > >     >     >         > (862) 250-7125
> > >     > > >     >     >         >
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >         --
> > >     > > >     >     >         -Regards,
> > >     > > >     >     >         Mayuresh R. Gharat
> > >     > > >     >     >         (862) 250-7125
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >     The information contained in this email is
> > > strictly
> > >     > > > confidential
> > >     > > >     > and
> > >     > > >     >     > for the use of the addressee only, unless
> otherwise
> > >     > > indicated.
> > >     > > > If
> > >     > > >     > you are
> > >     > > >     >     > not the intended recipient, please do not read,
> > > copy, use
> > >     > or
> > >     > > >     > disclose to
> > >     > > >     >     > others this message or any attachment. Please
> also
> > > notify
> > >     > the
> > >     > > > sender
> > >     > > >     > by
> > >     > > >     >     > replying to this email or by telephone (+44(020
> > 7896
> > > 0011)
> > >     > > and
> > >     > > > then
> > >     > > >     > delete
> > >     > > >     >     > the email and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > >     > > > that do
> > >     > > >     > not
> > >     > > >     >     > relate to the official business of this company
> > > shall be
> > >     > > > understood
> > >     > > >     > as
> > >     > > >     >     > neither given nor endorsed by it. IG is a trading
> > > name of
> > >     > IG
> > >     > > > Markets
> > >     > > >     >     > Limited (a company registered in England and
> Wales,
> > > company
> > >     > > > number
> > >     > > >     >     > 04008957) and IG Index Limited (a company
> > registered
> > > in
> > >     > > > England and
> > >     > > >     > Wales,
> > >     > > >     >     > company number 01190902). Registered address at
> > > Cannon
> > >     > Bridge
> > >     > > > House,
> > >     > > >     > 25
> > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > > Limited
> > >     > > > (register
> > >     > > >     > number
> > >     > > >     >     > 195355) and IG Index Limited (register number
> > > 114059) are
> > >     > > > authorised
> > >     > > >     > and
> > >     > > >     >     > regulated by the Financial Conduct Authority.
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >
> > >     > > >     >
> > >     > > >     > The information contained in this email is strictly
> > > confidential
> > >     > > and
> > >     > > > for
> > >     > > >     > the use of the addressee only, unless otherwise
> > indicated.
> > > If you
> > >     > > > are not
> > >     > > >     > the intended recipient, please do not read, copy, use
> or
> > > disclose
> > >     > > to
> > >     > > > others
> > >     > > >     > this message or any attachment. Please also notify the
> > > sender by
> > >     > > > replying
> > >     > > >     > to this email or by telephone (+44(020 7896 0011) and
> > then
> > > delete
> > >     > > > the email
> > >     > > >     > and any copies of it. Opinions, conclusion (etc) that
> do
> > > not
> > >     > relate
> > >     > > > to the
> > >     > > >     > official business of this company shall be understood
> as
> > > neither
> > >     > > > given nor
> > >     > > >     > endorsed by it. IG is a trading name of IG Markets
> > Limited
> > > (a
> > >     > > company
> > >     > > >     > registered in England and Wales, company number
> 04008957)
> > > and IG
> > >     > > > Index
> > >     > > >     > Limited (a company registered in England and Wales,
> > company
> > >     > number
> > >     > > >     > 01190902). Registered address at Cannon Bridge House,
> 25
> > > Dowgate
> > >     > > > Hill,
> > >     > > >     > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355)
> > >     > > > and IG
> > >     > > >     > Index Limited (register number 114059) are authorised
> and
> > >     > regulated
> > >     > > > by the
> > >     > > >     > Financial Conduct Authority.
> > >     > > >     >
> > >     > > >
> > >     > > >
> > >     > > > The information contained in this email is strictly
> > confidential
> > > and
> > >     > for
> > >     > > > the use of the addressee only, unless otherwise indicated. If
> > > you are
> > >     > not
> > >     > > > the intended recipient, please do not read, copy, use or
> > > disclose to
> > >     > > others
> > >     > > > this message or any attachment. Please also notify the sender
> > by
> > >     > replying
> > >     > > > to this email or by telephone (+44(020 7896 0011) and then
> > > delete the
> > >     > > email
> > >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to
> > >     > > the
> > >     > > > official business of this company shall be understood as
> > neither
> > > given
> > >     > > nor
> > >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > >     > > > registered in England and Wales, company number 04008957) and
> > IG
> > > Index
> > >     > > > Limited (a company registered in England and Wales, company
> > > number
> > >     > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355) and
> > >     > IG
> > >     > > > Index Limited (register number 114059) are authorised and
> > > regulated by
> > >     > > the
> > >     > > > Financial Conduct Authority.
> > >     > > >
> > >     > > The information contained in this email is strictly
> confidential
> > > and for
> > >     > > the use of the addressee only, unless otherwise indicated. If
> you
> > > are not
> > >     > > the intended recipient, please do not read, copy, use or
> disclose
> > > to
> > >     > others
> > >     > > this message or any attachment. Please also notify the sender
> by
> > > replying
> > >     > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > > the
> > >     > email
> > >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to
> > >     > the
> > >     > > official business of this company shall be understood as
> neither
> > > given
> > >     > nor
> > >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > >     > > registered in England and Wales, company number 04008957) and
> IG
> > > Index
> > >     > > Limited (a company registered in England and Wales, company
> > number
> > >     > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > > Hill,
> > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > and IG
> > >     > > Index Limited (register number 114059) are authorised and
> > > regulated by
> > >     > the
> > >     > > Financial Conduct Authority.
> > >     > >
> > >     > >
> > >     > The information contained in this email is strictly confidential
> > and
> > > for
> > >     > the use of the addressee only, unless otherwise indicated. If you
> > > are not
> > >     > the intended recipient, please do not read, copy, use or disclose
> > to
> > > others
> > >     > this message or any attachment. Please also notify the sender by
> > > replying
> > >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > > the email
> > >     > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > > to the
> > >     > official business of this company shall be understood as neither
> > > given nor
> > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > >     > registered in England and Wales, company number 04008957) and IG
> > > Index
> > >     > Limited (a company registered in England and Wales, company
> number
> > >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > > and IG
> > >     > Index Limited (register number 114059) are authorised and
> regulated
> > > by the
> > >     > Financial Conduct Authority.
> > >     >
> > >     >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jay Kreps <ja...@confluent.io>]

Hey Michael,

I'm not quite sure that works as that would translate ALL null values to
tombstones, even for non-compacted topics that use null as an acceptable
value sent by the producer and expected by the consumer.

-Jay

On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Ewen,
>
> I think the easiest way to show this is with code.
>
> As you can see we keep the existing behaviour for code/binaries calling
> the pre-existing constructors, whereby if the value is null the tombstone
> is set to true.
>
> Regards
> Mike
>
>
>
>     /**
>      * Creates a record with a specified timestamp to be sent to a
> specified topic and partition
>      *
>      * @param topic The topic the record will be appended to
>      * @param partition The partition to which the record should be sent
>      * @param timestamp The timestamp of the record
>      * @param tombstone if the record should be treated as a tombstone if
> the topic is compacted
>      * @param key The key that will be included in the record
>      * @param value The record contents
>      */
>     public ProducerRecord(String topic, Integer partition, Boolean
> tombstone, Long timestamp, K key, V value) {
>         if (topic == null)
>             throw new IllegalArgumentException("Topic cannot be null.");
>         if (timestamp != null && timestamp < 0)
>             throw new IllegalArgumentException(
>                     String.format("Invalid timestamp: %d. Timestamp should
> always be non-negative or null.", timestamp));
>         if (partition != null && partition < 0)
>             throw new IllegalArgumentException(
>                     String.format("Invalid partition: %d. Partition number
> should always be non-negative or null.", partition));
>         if (!tombstone && value == null){
>             throw new IllegalArgumentException(
>                     String.format("Tombstone must be true if null value"));
>         }
>         this.topic = topic;
>         this.partition = partition;
>         this.tombstone = tombstone;
>         this.key = key;
>         this.value = value;
>         this.timestamp = timestamp;
>     }
>
>     public ProducerRecord(String topic, Integer partition, Long timestamp,
> K key, V value) {
>         this(topic, partition, value == null, timestamp, key, value);
>     }
> ________________________________________
> From: Ewen Cheslack-Postava <ew...@confluent.io>
> Sent: Sunday, December 11, 2016 5:45 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> It seemed like this was addressed in the migration section, wasn't it? The
> V2 vs V3 requests and the fact that the broker will downgrade the message
> format (losing zero copy) if you issues a V2 request to a broker using V3
> format handles compatibility, does it not? The existing consumers won't see
> the extra metadata in the value, but they will get a null instead and treat
> it as a tombstone. Certainly there is a performance impact, but it seemed
> compatible.
>
> I'm worried about this though:
>
>
>    - *NOTE *: With the new version of producer client using ProduceRequest
>    V3 (magic byte = 2), a non tombstone (tombstone bit not set) and null
> value
>    should not be allowed as the older version of consumer using
> FetchRequest
>    V2 will think of this as a tombstone when its actually not.
>
>
> Unless I'm misunderstanding, this ends up breaking binary compatibility for
> the Java API. It sounds like this suggests that passing a null value to the
> existing ProducerRecord constructors would generate an exception since you
> didn't explicitly enable the tombstone (via whatever new constructor is
> provided). But that means you can't swap in a newer client jar without
> recompiling and get the same behavior if your app deletes keys using the
> current approach because your app will start throwing exceptions. Maybe
> this is a tradeoff we're ok with, but we've tried pretty hard to avoid
> breaking compatibility like this so far -- it makes picking up bug fixes in
> the clients harder for users of frameworks that have to pin to earlier
> default versions for compatibility.
>
> But then later the KIP says:
>
>
>    - The old constructors for ProducerRecord without this parameter will be
>    kept but updated so that their default behaviour if setting null value
> will
>    be the tombstone will be set to true to keep existing behaviour.
>
>
> So maybe I am misinterpreting something.
>
> And just a nit re: motivation section, item 6 would be clearer for a union
> schema with null (or optional schemas in other formats), e.g. [null,
> string], in which case losing the schema truly is losing information
> (whereas null is already the only valid value for a pure null schema).
>
> -Ewen
>
>
> On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jay,
> >
> > Good point this detail is missing in the KIP write up. Ive added this
> now.
> >
> > Essentially simply just upgrading the clients we do not expect any client
> > app code change needed.
> >
> > Cheers
> > Mike
> > ________________________________________
> > From: Jay Kreps <ja...@confluent.io>
> > Sent: Saturday, December 10, 2016 10:51 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Michael,
> >
> > The compatibility section goes through the migration path, but isn't the
> > bigger compatibility issue with existing apps? There are many (probably
> > thousands) of apps in production that use this feature and send null to
> > mean delete. It seems like this would break compatibility with them, and
> > they would have to be rewritten to right?
> >
> > -Jay
> >
> > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jun,
> > >
> > > 4) On v3 we honour the tombstone. As such we expect it to be set
> > correctly
> > > as per the KIP.
> > >
> > > 4.1) why would we want to produce an error when its v3? This is the
> exact
> > > purpose to support non-null tombstone’s
> > > 4.2) again here im unclear on the question on the v3, produce request
> we
> > > expect the tombstone flag to be set correctly.
> > >
> > > 4.4) compaction only occurs on compacted topics, the bit makes no
> > > difference and not looked at on un-compacted (time/size based
> eviction).
> > >
> > >
> > > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> > >
> > >     Hi, Michael,
> > >
> > >     4. Then, I think I misunderstood this point. Could you document the
> > >     following points in the wiki?
> > >     4.1 If producer V3 sets tombstone, but provides a non-null value,
> > does
> > > the
> > >     send() get an error or does the producer automatically set the
> value
> > to
> > >     null?
> > >     4.2 If producer V3 doesn't set tombstone, but provides a null
> value,
> > > does
> > >     the send() get an error or does the producer automatically sets the
> > >     tombstone?
> > >     4.3 Does the broker only expect messages that (a) have no tombstone
> > and
> > >     non-null value; (b) have tombstone and null value and reject the
> > > messages
> > >     with an error code otherwise?
> > >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on
> > not?
> > >
> > >     Thanks,
> > >
> > >     Jun
> > >
> > >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > >     wrote:
> > >
> > >     > Not at all.  This only acts on compacted topics just as what
> occurs
> > > today
> > >     >
> > >     > Sent using OWA for iPhone
> > >     > ________________________________________
> > >     > From: Jun Rao <ju...@confluent.io>
> > >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > >     > To: dev@kafka.apache.org
> > >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     >
> > >     > Hi, Michael,
> > >     >
> > >     > 4. Hmm, does that mean the new client library can never send a
> null
> > > message
> > >     > even to a regular topic? This seems like a change of the existing
> > > behavior.
> > >     >
> > >     > Thanks,
> > >     >
> > >     > Jun
> > >     >
> > >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > >     > wrote:
> > >     >
> > >     > > Hi Jun,
> > >     > >
> > >     > > Re 4) That's because we expect the tombstone value to be set
> > > correctly if
> > >     > > message bit is 2, as such if an older client sends in on old
> > > message the
> > >     > > message is upcast and the bit is set correctly. And such no
> > longer
> > > need
> > >     > to
> > >     > > check the value. Mayuresh can you confirm my thinking and
> > > understanding
> > >     > of
> > >     > > what we've discussed?
> > >     > >
> > >     > > The second point I understand what you're getting at now my
> > > apologies.
> > >     > Yes
> > >     > > this makes sense to save on touching the message, if we're the
> > > only kip
> > >     > > going in, in this release.
> > >     > >
> > >     > > Cheers
> > >     > > Mike
> > >     > >
> > >     > > Sent using OWA for iPhone
> > >     > > ________________________________________
> > >     > > From: Jun Rao <ju...@confluent.io>
> > >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > >     > > To: dev@kafka.apache.org
> > >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     > >
> > >     > > Hi, Michael,
> > >     > >
> > >     > > 4. Is this updated in the wiki? The text "If the magic byte on
> > > message is
> > >     > > 2, the broker should use the tombstone bit for log compaction."
> > > doesn't
> > >     > > seem to have changed.
> > >     > >
> > >     > > 2. My point is that if we change the message format just for
> this
> > > KIP, we
> > >     > > should consider whether it's worth optimizing the down
> conversion
> > > path
> > >     > > (i.e., decide whether a conversion is needed by just looking at
> > the
> > >     > > tombstone bit in the wrapper message) since tombstone will be
> > used
> > >     > rarely.
> > >     > > However, if the message format change here is combined with
> other
> > > KIPs,
> > >     > > then this optimization likely won't be needed. The latter
> > probably
> > > makes
> > >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> > >     > >
> > >     > > Other than those, +1 from me,
> > >     > >
> > >     > > Thanks,
> > >     > >
> > >     > > Jun
> > >     > >
> > >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > >     > > wrote:
> > >     > >
> > >     > > > Hi Jun
> > >     > > >
> > >     > > > do we have your vote on this now?
> > >     > > >
> > >     > > > Any other concerns?
> > >     > > >
> > >     > > > Cheers
> > >     > > > Mike
> > >     > > >
> > >     > > > Sent using OWA for iPhone
> > >     > > > ________________________________________
> > >     > > > From: Michael Pearce <Mi...@ig.com>
> > >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > >     > > > To: dev@kafka.apache.org
> > >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >     > > >
> > >     > > > Hi Jun,
> > >     > > >
> > >     > > > Have updated. Thanks again for the feedback.
> > >     > > >
> > >     > > > Agree yes we should align up when it gets to that, I assume
> > > you’ve
> > >     > > flagged
> > >     > > > the same in the other KIP?
> > >     > > >
> > >     > > > I think for now let’s get this KIP approved, then we can
> start
> > > the work
> > >     > > to
> > >     > > > get an initial PR, then we can discuss how to align the two
> if
> > > needed.
> > >     > > >
> > >     > > > Cheers,
> > >     > > > Mike
> > >     > > >
> > >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> > >     > > >
> > >     > > >     Hi, Michael,
> > >     > > >
> > >     > > >     For 2), this is fine. Could you update the KIP wiki to
> make
> > > this
> > >     > and
> > >     > > > other
> > >     > > >     points clearer? Other than that, the KIP looks good to
> me.
> > >     > > >
> > >     > > >     An orthogonal thing is that there are other KIPs such as
> > > KIP-98
> > >     > that
> > >     > > > also
> > >     > > >     intend to change the message format. If they all get
> > > approved, we
> > >     > > > should
> > >     > > >     think about whether it's better to just bump up the magic
> > > byte once
> > >     > > to
> > >     > > >     incorporate multiple format changes like we did in
> > > KIP-31/KIP-32.
> > >     > > >
> > >     > > >     Thanks,
> > >     > > >
> > >     > > >     Jun
> > >     > > >
> > >     > > >
> > >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > >     > > Michael.Pearce@ig.com>
> > >     > > >     wrote:
> > >     > > >
> > >     > > >     > Hi Jao
> > >     > > >     >
> > >     > > >     > Thanks for the response. Sorry for slow reply, both
> with
> > > personal
> > >     > > > sickness
> > >     > > >     > and also battling some critical issues encountered
> since
> > >     > upgrading
> > >     > > to
> > >     > > >     > 0.10.1.0
> > >     > > >     >
> > >     > > >     > 1) Thans for spotting, Document error where we branched
> > > this KIP
> > >     > > from
> > >     > > >     > KIP-82, will get that removed.
> > >     > > >     > 2) Intent is to do this just at the record message
> level.
> > >     > > >     > 3) Thanks for spotting, Will ensure this is corrected.
> > >     > > >     > 4) As per discussion thread we will support tombstone +
> > > null
> > >     > value,
> > >     > > >     > tombstone + non null value, no tombstone + null value.
> > >     > > >     > 5) I believe this was in the discussion thread,
> @Mayuresh
> > > is this
> > >     > > >     > something we’ve overlooked? I thought we would down
> > > convert and
> > >     > > > remove the
> > >     > > >     > value so the old consumer had existing behavior, or is
> > > there
> > >     > > > something we
> > >     > > >     > haven’t thought about?
> > >     > > >     >
> > >     > > >     > Cheers
> > >     > > >     > Mike
> > >     > > >     >
> > >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
> > wrote:
> > >     > > >     >
> > >     > > >     >     Hi, Michael,
> > >     > > >     >
> > >     > > >     >     Thanks for the KIP. A few comments below.
> > >     > > >     >
> > >     > > >     >     1. The message format change contains
> "HeadersLength
> > >     > Headers".
> > >     > > > Is that
> > >     > > >     >     intended?
> > >     > > >     >
> > >     > > >     >     2. For compressed messageset, is the tombstone bit
> > > only set
> > >     > at
> > >     > > > the
> > >     > > >     > shallow
> > >     > > >     >     level? Do we always leave that bit in the wrapper
> > > message
> > >     > > unset?
> > >     > > > An
> > >     > > >     >     alternative is to set the tombstone bit in the
> > wrapper
> > > if at
> > >     > > > least one
> > >     > > >     >     inner message has the tombstone bit set. This makes
> > > things a
> > >     > > bit
> > >     > > > more
> > >     > > >     >     complicated, but we could potentially exploit that
> > for
> > >     > > > optimizing down
> > >     > > >     >     conversion. For example, we only need to convert
> > > messages
> > >     > with
> > >     > > > magic 2
> > >     > > >     > to
> > >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > > (conversion is
> > >     > > > always
> > >     > > >     > needed
> > >     > > >     >     from magic 2 to magic 0). Not sure if the
> > optimization
> > > is
> > >     > worth
> > >     > > > the
> > >     > > >     >     complexity though.
> > >     > > >     >
> > >     > > >     >     3. The referencing of the new version of
> > >     > > > ProducerRequest/FetchRequest
> > >     > > >     > is
> > >     > > >     >     inconsistent (v4 vs v3). Since our convention
> starts
> > at
> > >     > version
> > >     > > > at 0, I
> > >     > > >     >     think the new version would be 3.
> > >     > > >     >
> > >     > > >     >     4. "If the magic byte on message is 2, the broker
> > > should use
> > >     > > the
> > >     > > >     > tombstone
> > >     > > >     >     bit for log compaction." What about null value? My
> > >     > > understanding
> > >     > > > is
> > >     > > >     > that
> > >     > > >     >     null value will be treated the same as setting the
> > > tombstone
> > >     > > bit.
> > >     > > >     >
> > >     > > >     >     5. For the migration path, it would be useful to
> > > describe the
> > >     > > > down
> > >     > > >     >     conversion path to consumers (i.e., brokers on
> > message
> > > format
> > >     > > > 0.10.2
> > >     > > >     > and
> > >     > > >     >     consumers on older version).
> > >     > > >     >
> > >     > > >     >     Thanks,
> > >     > > >     >
> > >     > > >     >     Jun
> > >     > > >     >
> > >     > > >     >
> > >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > >     > > > Michael.Pearce@ig.com
> > >     > > >     > >
> > >     > > >     >     wrote:
> > >     > > >     >
> > >     > > >     >     > Hi All,
> > >     > > >     >     >
> > >     > > >     >     > We have been discussing in the below thread and
> > final
> > >     > changes
> > >     > > > have
> > >     > > >     > been
> > >     > > >     >     > made to the KIP wiki based on these discussions.
> > >     > > >     >     >
> > >     > > >     >     > We would now like to put to the vote the
> following
> > > KIP:
> > >     > > >     >     > https://cwiki.apache.org/
> > > confluence/display/KAFKA/KIP-
> > >     > 87+-+
> > >     > > >     >     > Add+Compaction+Tombstone+Flag
> > >     > > >     >     >
> > >     > > >     >     > This kip is for having a distinct compaction
> > > attribute
> > >     > > > “tombstone”
> > >     > > >     > flag
> > >     > > >     >     > instead of relying on null value, allowing
> non-null
> > > value
> > >     > > > delete
> > >     > > >     > messages.
> > >     > > >     >     >
> > >     > > >     >     > Many thanks,
> > >     > > >     >     > Michael
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> > >     > > Michael.Pearce@ig.com>
> > >     > > >     > wrote:
> > >     > > >     >     >
> > >     > > >     >     >     Hi Mayuresh,
> > >     > > >     >     >
> > >     > > >     >     >     LGTM. Ive just made one small adjustment
> > > updating the
> > >     > > wire
> > >     > > >     > protocol to
> > >     > > >     >     > show the magic byte bump.
> > >     > > >     >     >
> > >     > > >     >     >     Do we think we’re good to put to a vote? Is
> > > there any
> > >     > > > other bits
> > >     > > >     >     > needing discussion?
> > >     > > >     >     >
> > >     > > >     >     >     Cheers
> > >     > > >     >     >     Mike
> > >     > > >     >     >
> > >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> > >     > > >     > gharatmayuresh15@gmail.com>
> > >     > > >     >     > wrote:
> > >     > > >     >     >
> > >     > > >     >     >         Hi Michael,
> > >     > > >     >     >
> > >     > > >     >     >         I have updated the migration section of
> the
> > > KIP.
> > >     > Can
> > >     > > > you
> > >     > > >     > please
> > >     > > >     >     > take a look?
> > >     > > >     >     >
> > >     > > >     >     >         Thanks,
> > >     > > >     >     >
> > >     > > >     >     >         Mayuresh
> > >     > > >     >     >
> > >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh
> > > Gharat <
> > >     > > >     >     > gharatmayuresh15@gmail.com
> > >     > > >     >     >         > wrote:
> > >     > > >     >     >
> > >     > > >     >     >         > Hi Michael,
> > >     > > >     >     >         >
> > >     > > >     >     >         > That whilst sending tombstone and non
> > null
> > > value,
> > >     > > the
> > >     > > >     > consumer
> > >     > > >     >     > can expect
> > >     > > >     >     >         > only to receive the non-null message
> only
> > > in step
> > >     > > > (3) is
> > >     > > >     > this
> > >     > > >     >     > correct?
> > >     > > >     >     >         > ---> I do agree with you here.
> > >     > > >     >     >         >
> > >     > > >     >     >         > Becket, Ismael : can you guys review
> the
> > >     > migration
> > >     > > > plan
> > >     > > >     > listed
> > >     > > >     >     > above using
> > >     > > >     >     >         > magic byte?
> > >     > > >     >     >         >
> > >     > > >     >     >         > Thanks,
> > >     > > >     >     >         >
> > >     > > >     >     >         > Mayuresh
> > >     > > >     >     >         >
> > >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM,
> Michael
> > > Pearce <
> > >     > > >     >     > Michael.Pearce@ig.com>
> > >     > > >     >     >         > wrote:
> > >     > > >     >     >         >
> > >     > > >     >     >         >> Many thanks for this Mayuresh. I don't
> > > have any
> > >     > > >     > objections.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> I assume we should state:
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> That whilst sending tombstone and non
> > null
> > >     > value,
> > >     > > > the
> > >     > > >     > consumer
> > >     > > >     >     > can expect
> > >     > > >     >     >         >> only to receive the non-null message
> > only
> > > in
> > >     > step
> > >     > > > (3) is
> > >     > > >     > this
> > >     > > >     >     > correct?
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Cheers
> > >     > > >     >     >         >> Mike
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Sent using OWA for iPhone
> > >     > > >     >     >         >> ______________________________
> > __________
> > >     > > >     >     >         >> From: Mayuresh Gharat <
> > >     > gharatmayuresh15@gmail.com
> > >     > > >
> > >     > > >     >     >         >> Sent: Thursday, November 17, 2016
> > 5:18:41
> > > PM
> > >     > > >     >     >         >> To: dev@kafka.apache.org
> > >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
> > > Compaction
> > >     > > > Tombstone
> > >     > > >     > Flag
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Hi Ismael,
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Thanks for the explanation.
> > >     > > >     >     >         >> Specially I like this part where in
> you
> > >     > mentioned
> > >     > > > we can
> > >     > > >     > get
> > >     > > >     >     > rid of the
> > >     > > >     >     >         >> older null value support for log
> > > compaction
> > >     > later
> > >     > > > on,
> > >     > > >     > here :
> > >     > > >     >     >         >> We can't change semantics of the
> message
> > > format
> > >     > > > without
> > >     > > >     > having
> > >     > > >     >     > a long
> > >     > > >     >     >         >> transition period. And we can't rely
> > >     > > >     >     >         >> on people reading documentation or
> > acting
> > > on a
> > >     > > > warning for
> > >     > > >     >     > something so
> > >     > > >     >     >         >> fundamental. As such, my take is that
> we
> > > need to
> > >     > > > bump the
> > >     > > >     > magic
> > >     > > >     >     > byte. The
> > >     > > >     >     >         >> good news is
> > >     > > >     >     >         >> that we don't have to support all
> > versions
> > >     > > forever.
> > >     > > > We
> > >     > > >     > have
> > >     > > >     >     > said that we
> > >     > > >     >     >         >> will support direct upgrades for 2
> > years.
> > > That
> > >     > > > means that
> > >     > > >     >     > message format
> > >     > > >     >     >         >> version n could, in theory, be
> removed 2
> > > years
> > >     > > > after the
> > >     > > >     > it's
> > >     > > >     >     > introduced.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Just a heads up, I would like to
> mention
> > > that
> > >     > even
> > >     > > > without
> > >     > > >     >     > bumping magic
> > >     > > >     >     >         >> byte, we will *NOT* loose zero copy as
> > in
> > > the
> > >     > > > client(x+1)
> > >     > > >     > in my
> > >     > > >     >     >         >> explanation
> > >     > > >     >     >         >> above will convert internally a null
> > > value to
> > >     > > have a
> > >     > > >     > tombstone
> > >     > > >     >     > bit set and
> > >     > > >     >     >         >> a tombstone bit set to have a null
> value
> > >     > > > automatically
> > >     > > >     >     > internally and by
> > >     > > >     >     >         >> the time we move to version (x+2), the
> > > clients
> > >     > > > would have
> > >     > > >     >     > upgraded.
> > >     > > >     >     >         >> Obviously if we support a request from
> > >     > > consumer(x),
> > >     > > > we
> > >     > > >     > will
> > >     > > >     >     > loose zero
> > >     > > >     >     >         >> copy
> > >     > > >     >     >         >> but that is the same case with magic
> > byte.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> But if magic byte bump makes life
> easier
> > > for
> > >     > > > transition
> > >     > > >     > for the
> > >     > > >     >     > above
> > >     > > >     >     >         >> reasons that you explained, I am OK
> with
> > > it
> > >     > since
> > >     > > > we are
> > >     > > >     > going
> > >     > > >     >     > to meet the
> > >     > > >     >     >         >> end goal down the road :)
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> On a side note can we update the doc
> > here
> > > on
> > >     > magic
> > >     > > > byte
> > >     > > >     > to say
> > >     > > >     >     > that "*it
> > >     > > >     >     >         >> should be bumped whenever the message
> > > format is
> > >     > > > changed
> > >     > > >     > or the
> > >     > > >     >     >         >> interpretation of message format
> (usage
> > > of the
> > >     > > > reserved
> > >     > > >     > bits as
> > >     > > >     >     > well) is
> > >     > > >     >     >         >> changed*".
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Hi Michael,
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Here is the update plan that we
> > discussed
> > >     > offline
> > >     > > >     > yesterday :
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Currently the magic-byte which
> > > corresponds to
> > >     > the
> > >     > > >     >     > "message.format.version"
> > >     > > >     >     >         >> is set to 1.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> 1) On broker it will be set to 1
> > > initially.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> 2) When a producer client sends a
> > message
> > > with
> > >     > > > magic-byte
> > >     > > >     > = 2,
> > >     > > >     >     > since the
> > >     > > >     >     >         >> broker is on magic-byte = 1, we will
> > down
> > >     > convert
> > >     > > > it,
> > >     > > >     > which
> > >     > > >     >     > means if the
> > >     > > >     >     >         >> tombstone bit is set, the value will
> be
> > > set to
> > >     > > > null. A
> > >     > > >     > consumer
> > >     > > >     >     >         >> understanding magic-byte = 1, will
> still
> > > work
> > >     > with
> > >     > > > this. A
> > >     > > >     >     > consumer
> > >     > > >     >     >         >> working
> > >     > > >     >     >         >> with magic-byte =2 will also be able
> to
> > >     > understand
> > >     > > > this,
> > >     > > >     > since
> > >     > > >     >     > it
> > >     > > >     >     >         >> understands the tombstone.
> > >     > > >     >     >         >> Now there is still the question of
> > > supporting a
> > >     > > >     > non-tombstone
> > >     > > >     >     > and null
> > >     > > >     >     >         >> value from producer client with
> > > magic-byte = 2.*
> > >     > > (I
> > >     > > > am
> > >     > > >     > not sure
> > >     > > >     >     > if we
> > >     > > >     >     >         >> should support this. Ismael/Becket can
> > > comment
> > >     > > > here)*
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> 3) When almost all the clients have
> > > upgraded,
> > >     > the
> > >     > > >     >     > message.format.version
> > >     > > >     >     >         >> on
> > >     > > >     >     >         >> the broker can be changed to 2, where
> in
> > > the
> > >     > down
> > >     > > >     > conversion in
> > >     > > >     >     > the above
> > >     > > >     >     >         >> step will not happen. If at this point
> > we
> > > get a
> > >     > > > consumer
> > >     > > >     >     > request from a
> > >     > > >     >     >         >> older consumer, we might have to down
> > > convert
> > >     > > where
> > >     > > > in we
> > >     > > >     > loose
> > >     > > >     >     > zero copy,
> > >     > > >     >     >         >> but these cases should be rare.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Becket can you review this plan and
> add
> > > more
> > >     > > > details if I
> > >     > > >     > have
> > >     > > >     >     >         >> missed/wronged something, before we
> put
> > > it on
> > >     > KIP.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Thanks,
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> Mayuresh
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
> > Michael
> > >     > Pearce <
> > >     > > >     >     > Michael.Pearce@ig.com>
> > >     > > >     >     >         >> wrote:
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> > Thanks guys, for discussing this
> > > offline and
> > >     > > > getting
> > >     > > >     > some
> > >     > > >     >     > consensus.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > So its clear for myself and others
> > what
> > > is
> > >     > > > proposed now
> > >     > > >     > (i
> > >     > > >     >     > think i
> > >     > > >     >     >         >> > understand, but want to make sure)
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Could i ask either directly update
> the
> > > kip to
> > >     > > > detail the
> > >     > > >     >     > migration
> > >     > > >     >     >         >> > strategy, or (re-)state your offline
> > > discussed
> > >     > > and
> > >     > > >     > agreed
> > >     > > >     >     > migration
> > >     > > >     >     >         >> > strategy based on a magic byte is in
> > > this
> > >     > > thread.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > The main original driver for the KIP
> > > was to
> > >     > > > support
> > >     > > >     >     > compaction where
> > >     > > >     >     >         >> value
> > >     > > >     >     >         >> > isn't null, based off the
> discussions
> > on
> > >     > KIP-82
> > >     > > > thread.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > We should be able to support
> > > non-tombstone +
> > >     > > null
> > >     > > > value
> > >     > > >     > by the
> > >     > > >     >     >         >> completion
> > >     > > >     >     >         >> > of the KIP, as we noted when
> > discussing
> > > this
> > >     > > kip,
> > >     > > > having
> > >     > > >     >     > logic based on
> > >     > > >     >     >         >> a
> > >     > > >     >     >         >> > null value isn't very clean and also
> > > separates
> > >     > > the
> > >     > > >     > concerns.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > As discussed already though we can
> > > split this
> > >     > > into
> > >     > > >     > KIP-87a
> > >     > > >     >     > and KIP-87b
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Where we look to deliver KIP-87a on
> a
> > >     > compacted
> > >     > > > topic
> > >     > > >     > (to
> > >     > > >     >     > address the
> > >     > > >     >     >         >> > immediate issues)
> > >     > > >     >     >         >> > * tombstone + null value
> > >     > > >     >     >         >> > * tombstone + non-null value
> > >     > > >     >     >         >> > * non-tombstone + non-null value
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Then we can discuss once KIP-87a is
> > > completed
> > >     > > > options
> > >     > > >     > later
> > >     > > >     >     > and how we
> > >     > > >     >     >         >> > support the second part KIP-87b to
> > > deliver:
> > >     > > >     >     >         >> > * non-tombstone + null value
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Cheers
> > >     > > >     >     >         >> > Mike
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > ______________________________
> > > __________
> > >     > > >     >     >         >> > From: Becket Qin <
> > becket.qin@gmail.com>
> > >     > > >     >     >         >> > Sent: Thursday, November 17, 2016
> 1:43
> > > AM
> > >     > > >     >     >         >> > To: dev@kafka.apache.org
> > >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add
> > > Compaction
> > >     > > >     > Tombstone Flag
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Renu, Mayuresh and I had an offline
> > >     > discussion,
> > >     > > > and
> > >     > > >     > following
> > >     > > >     >     > is a brief
> > >     > > >     >     >         >> > summary.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > 1. We agreed that not bumping up
> magic
> > > value
> > >     > may
> > >     > > > result
> > >     > > >     > in
> > >     > > >     >     > losing zero
> > >     > > >     >     >         >> copy
> > >     > > >     >     >         >> > during migration.
> > >     > > >     >     >         >> > 2. Given that bumping up magic value
> > is
> > > almost
> > >     > > > free and
> > >     > > >     > has
> > >     > > >     >     > benefit of
> > >     > > >     >     >         >> > avoiding potential performance
> issue.
> > > It is
> > >     > > > probably
> > >     > > >     > worth
> > >     > > >     >     > doing.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > One issue we still need to think
> about
> > > is
> > >     > > whether
> > >     > > > we
> > >     > > >     > want to
> > >     > > >     >     > support a
> > >     > > >     >     >         >> > non-tombstone message with null
> value.
> > >     > > >     >     >         >> > Currently it is not supported by
> > Kafka.
> > > If we
> > >     > > > allow a
> > >     > > >     >     > non-tombstone null
> > >     > > >     >     >         >> > value message to exist after KIP-87.
> > The
> > >     > problem
> > >     > > > is
> > >     > > >     > that such
> > >     > > >     >     > message
> > >     > > >     >     >         >> will
> > >     > > >     >     >         >> > not be supported by the consumers
> > prior
> > > to
> > >     > > KIP-87.
> > >     > > >     > Because a
> > >     > > >     >     > null value
> > >     > > >     >     >         >> > will always be interpreted to a
> > > tombstone.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > One option is that we keep the
> current
> > > way,
> > >     > i.e.
> > >     > > > do not
> > >     > > >     >     > support such
> > >     > > >     >     >         >> > message. It would be good to know if
> > > there is
> > >     > a
> > >     > > >     > concrete use
> > >     > > >     >     > case for
> > >     > > >     >     >         >> such
> > >     > > >     >     >         >> > message. If there is not, we can
> > > probably just
> > >     > > not
> > >     > > >     > support it.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > Thanks,
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > JIangjie (Becket) Qin
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> > > Mayuresh
> > >     > > Gharat <
> > >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> > >     > > >     >     >         >> > > wrote:
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >> > > Hi Ismael,
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > This is something I can think of
> for
> > >     > migration
> > >     > > > plan:
> > >     > > >     >     >         >> > > So the migration plan can look
> > > something
> > >     > like
> > >     > > > this,
> > >     > > >     > with up
> > >     > > >     >     >         >> conversion :
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > 1) Currently lets say we have
> Broker
> > > at
> > >     > > version
> > >     > > > x.
> > >     > > >     >     >         >> > > 2) Currently we have clients at
> > > version x.
> > >     > > >     >     >         >> > > 3) a) We move the version to
> > > Broker(x+1) :
> > >     > > > supports
> > >     > > >     > both
> > >     > > >     >     > tombstone and
> > >     > > >     >     >         >> > null
> > >     > > >     >     >         >> > > for log compaction.
> > >     > > >     >     >         >> > >     b) We upgrade the client to
> > > version
> > >     > > > client(x+1) :
> > >     > > >     > if in
> > >     > > >     >     > the
> > >     > > >     >     >         >> producer
> > >     > > >     >     >         >> > > client(x+1) the value is set to
> > null,
> > > we
> > >     > will
> > >     > > >     > automatically
> > >     > > >     >     > set the
> > >     > > >     >     >         >> > > Tombstone bit internally. If the
> > > producer
> > >     > > > client(x+1)
> > >     > > >     > sets
> > >     > > >     >     > the
> > >     > > >     >     >         >> tombstone
> > >     > > >     >     >         >> > > itself, well and good. For
> producer
> > >     > client(x),
> > >     > > > the
> > >     > > >     > broker
> > >     > > >     >     > will up
> > >     > > >     >     >         >> convert
> > >     > > >     >     >         >> > > to have the tombstone bit.
> > > Broker(x+1) is
> > >     > > > supporting
> > >     > > >     > both.
> > >     > > >     >     > Consumer
> > >     > > >     >     >         >> > > client(x+1) will be aware of this
> > and
> > > should
> > >     > > be
> > >     > > > able
> > >     > > >     > to
> > >     > > >     >     > handle this.
> > >     > > >     >     >         >> For
> > >     > > >     >     >         >> > > consumer client(x) we will down
> > > convert the
> > >     > > > message
> > >     > > >     > on the
> > >     > > >     >     > broker
> > >     > > >     >     >         >> side.
> > >     > > >     >     >         >> > >     c) At this point we will have
> to
> > >     > specify a
> > >     > > >     > warning or
> > >     > > >     >     > clearly
> > >     > > >     >     >         >> specify
> > >     > > >     >     >         >> > > in docs that this behavior is
> about
> > > to be
> > >     > > > changed for
> > >     > > >     > log
> > >     > > >     >     > compaction.
> > >     > > >     >     >         >> > > 4) a) In next release of the
> > > Broker(x+2), we
> > >     > > > say that
> > >     > > >     > only
> > >     > > >     >     > Tombstone
> > >     > > >     >     >         >> is
> > >     > > >     >     >         >> > > used for log compaction on the
> > Broker
> > > side.
> > >     > > >     > Clients(x+1)
> > >     > > >     >     > still is
> > >     > > >     >     >         >> > > supported.
> > >     > > >     >     >         >> > >     b) We upgrade the client to
> > > version
> > >     > > > client(x+2) :
> > >     > > >     > if
> > >     > > >     >     > value is set
> > >     > > >     >     >         >> to
> > >     > > >     >     >         >> > > null, tombstone will not be set
> > >     > automatically.
> > >     > > > The
> > >     > > >     > client
> > >     > > >     >     > will have to
> > >     > > >     >     >         >> > call
> > >     > > >     >     >         >> > > setTombstone() to actually set the
> > >     > tombstone.
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > We should compare this migration
> > plan
> > > with
> > >     > the
> > >     > > >     > migration
> > >     > > >     >     > plan for
> > >     > > >     >     >         >> magic
> > >     > > >     >     >         >> > > byte bump and do whatever looks
> > good.
> > >     > > >     >     >         >> > > I am just worried that if we go
> down
> > > magic
> > >     > > byte
> > >     > > > route,
> > >     > > >     >     > unless I am
> > >     > > >     >     >         >> > missing
> > >     > > >     >     >         >> > > something, it sounds like kafka
> will
> > > be
> > >     > stuck
> > >     > > > with
> > >     > > >     >     > supporting both
> > >     > > >     >     >         >> null
> > >     > > >     >     >         >> > > value and tombstone bit for log
> > > compaction
> > >     > for
> > >     > > > life
> > >     > > >     > long,
> > >     > > >     >     > which does
> > >     > > >     >     >         >> not
> > >     > > >     >     >         >> > > look like a good end state.
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > Thanks,
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > Mayuresh
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
> > > Mayuresh
> > >     > > > Gharat <
> > >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> > >     > > >     >     >         >> > > > wrote:
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > > Hi Ismael,
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > That's a very good point which I
> > > might
> > >     > have
> > >     > > > not
> > >     > > >     >     > considered earlier.
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > Here is a plan that I can think
> > of:
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > Stage 1) The broker from now on,
> > up
> > >     > converts
> > >     > > > the
> > >     > > >     > message
> > >     > > >     >     > to have the
> > >     > > >     >     >         >> > > > tombstone marker. The log
> > compaction
> > >     > thread
> > >     > > > does log
> > >     > > >     >     > compaction
> > >     > > >     >     >         >> based
> > >     > > >     >     >         >> > on
> > >     > > >     >     >         >> > > > both null and tombstone marker.
> > > This is
> > >     > our
> > >     > > >     > transition
> > >     > > >     >     > period.
> > >     > > >     >     >         >> > > > Stage 2) The next release we
> only
> > > say that
> > >     > > log
> > >     > > >     > compaction
> > >     > > >     >     > is based
> > >     > > >     >     >         >> on
> > >     > > >     >     >         >> > > > tombstone marker. (Open source
> > > kafka makes
> > >     > > > this as a
> > >     > > >     >     > policy). By
> > >     > > >     >     >         >> this
> > >     > > >     >     >         >> > > time,
> > >     > > >     >     >         >> > > > the organization which is moving
> > to
> > > this
> > >     > > > release
> > >     > > >     > will be
> > >     > > >     >     > sure that
> > >     > > >     >     >         >> they
> > >     > > >     >     >         >> > > > have gone through the entire
> > > transition
> > >     > > > period.
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > My only goal of doing this is
> that
> > > Kafka
> > >     > > > clearly
> > >     > > >     >     > specifies the end
> > >     > > >     >     >         >> > state
> > >     > > >     >     >         >> > > > about what log compaction means
> > (is
> > > it
> > >     > null
> > >     > > > value
> > >     > > >     > or a
> > >     > > >     >     > tombstone
> > >     > > >     >     >         >> > marker,
> > >     > > >     >     >         >> > > > but not both).
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > What do you think?
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > Thanks,
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > Mayuresh
> > >     > > >     >     >         >> > > > .
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM,
> > > Ismael
> > >     > > Juma <
> > >     > > >     >     > ismael@juma.me.uk>
> > >     > > >     >     >         >> > wrote:
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > >> One comment below.
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08
> PM,
> > > Mayuresh
> > >     > > > Gharat <
> > >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> > >     > > >     >     >         >> > > >> > wrote:
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >> >    - If we don't bump up the
> > > magic
> > >     > byte,
> > >     > > > on the
> > >     > > >     > broker
> > >     > > >     >     > side, the
> > >     > > >     >     >         >> > > broker
> > >     > > >     >     >         >> > > >> >    will always have to look
> at
> > > both
> > >     > > > tombstone
> > >     > > >     > bit and
> > >     > > >     >     > the value
> > >     > > >     >     >         >> when
> > >     > > >     >     >         >> > > do
> > >     > > >     >     >         >> > > >> the
> > >     > > >     >     >         >> > > >> >    compaction. Assuming we do
> > > not bump
> > >     > up
> > >     > > > the
> > >     > > >     > magic
> > >     > > >     >     > byte,
> > >     > > >     >     >         >> > > >> >    imagine the broker sees a
> > > message
> > >     > > which
> > >     > > > does
> > >     > > >     > not
> > >     > > >     >     > have a
> > >     > > >     >     >         >> tombstone
> > >     > > >     >     >         >> > > bit
> > >     > > >     >     >         >> > > >> >    set. The broker does not
> > know
> > > when
> > >     > the
> > >     > > >     > message was
> > >     > > >     >     > produced
> > >     > > >     >     >         >> (i.e.
> > >     > > >     >     >         >> > > >> > whether
> > >     > > >     >     >         >> > > >> >    the message has been up
> > > converted or
> > >     > > > not), it
> > >     > > >     > has
> > >     > > >     >     > to take a
> > >     > > >     >     >         >> > further
> > >     > > >     >     >         >> > > >> > look at
> > >     > > >     >     >         >> > > >> >    the value to see if it is
> > > null or
> > >     > not
> > >     > > in
> > >     > > >     > order to
> > >     > > >     >     > determine
> > >     > > >     >     >         >> if it
> > >     > > >     >     >         >> > > is
> > >     > > >     >     >         >> > > >> a
> > >     > > >     >     >         >> > > >> >    tombstone. The same logic
> > has
> > > to be
> > >     > > put
> > >     > > > on the
> > >     > > >     >     > consumer as
> > >     > > >     >     >         >> well
> > >     > > >     >     >         >> > > >> because
> > >     > > >     >     >         >> > > >> > the
> > >     > > >     >     >         >> > > >> >    consumer does not know if
> > the
> > >     > message
> > >     > > > has
> > >     > > >     > been up
> > >     > > >     >     > converted or
> > >     > > >     >     >         >> > not.
> > >     > > >     >     >         >> > > >> >       - If we upconvert while
> > >     > appending,
> > >     > > > this is
> > >     > > >     > not
> > >     > > >     >     > the case,
> > >     > > >     >     >         >> > right?
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >> If I understand you correctly,
> > > this is
> > >     > not
> > >     > > >     > sufficient
> > >     > > >     >     > because the
> > >     > > >     >     >         >> log
> > >     > > >     >     >         >> > > may
> > >     > > >     >     >         >> > > >> have messages appended before
> it
> > > was
> > >     > > > upgraded to
> > >     > > >     > include
> > >     > > >     >     > KIP-87.
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >> Ismael
> > >     > > >     >     >         >> > > >>
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > > > --
> > >     > > >     >     >         >> > > > -Regards,
> > >     > > >     >     >         >> > > > Mayuresh R. Gharat
> > >     > > >     >     >         >> > > > (862) 250-7125
> > >     > > >     >     >         >> > > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > > --
> > >     > > >     >     >         >> > > -Regards,
> > >     > > >     >     >         >> > > Mayuresh R. Gharat
> > >     > > >     >     >         >> > > (862) 250-7125
> > >     > > >     >     >         >> > >
> > >     > > >     >     >         >> > The information contained in this
> > email
> > > is
> > >     > > > strictly
> > >     > > >     >     > confidential and for
> > >     > > >     >     >         >> > the use of the addressee only,
> unless
> > >     > otherwise
> > >     > > >     > indicated. If
> > >     > > >     >     > you are
> > >     > > >     >     >         >> not
> > >     > > >     >     >         >> > the intended recipient, please do
> not
> > > read,
> > >     > > copy,
> > >     > > > use or
> > >     > > >     >     > disclose to
> > >     > > >     >     >         >> others
> > >     > > >     >     >         >> > this message or any attachment.
> Please
> > > also
> > >     > > > notify the
> > >     > > >     > sender
> > >     > > >     >     > by
> > >     > > >     >     >         >> replying
> > >     > > >     >     >         >> > to this email or by telephone
> (+44(020
> > > 7896
> > >     > > 0011)
> > >     > > > and
> > >     > > >     > then
> > >     > > >     >     > delete the
> > >     > > >     >     >         >> email
> > >     > > >     >     >         >> > and any copies of it. Opinions,
> > > conclusion
> > >     > (etc)
> > >     > > > that
> > >     > > >     > do not
> > >     > > >     >     > relate to
> > >     > > >     >     >         >> the
> > >     > > >     >     >         >> > official business of this company
> > shall
> > > be
> > >     > > > understood as
> > >     > > >     >     > neither given
> > >     > > >     >     >         >> nor
> > >     > > >     >     >         >> > endorsed by it. IG is a trading name
> > of
> > > IG
> > >     > > Markets
> > >     > > >     > Limited (a
> > >     > > >     >     > company
> > >     > > >     >     >         >> > registered in England and Wales,
> > company
> > >     > number
> > >     > > >     > 04008957) and
> > >     > > >     >     > IG Index
> > >     > > >     >     >         >> > Limited (a company registered in
> > > England and
> > >     > > > Wales,
> > >     > > >     > company
> > >     > > >     >     > number
> > >     > > >     >     >         >> > 01190902). Registered address at
> > Cannon
> > > Bridge
> > >     > > > House, 25
> > >     > > >     >     > Dowgate Hill,
> > >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
> > Limited
> > >     > > (register
> > >     > > >     > number
> > >     > > >     >     > 195355) and IG
> > >     > > >     >     >         >> > Index Limited (register number
> 114059)
> > > are
> > >     > > > authorised
> > >     > > >     > and
> > >     > > >     >     > regulated by
> > >     > > >     >     >         >> the
> > >     > > >     >     >         >> > Financial Conduct Authority.
> > >     > > >     >     >         >> >
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >>
> > >     > > >     >     >         >> --
> > >     > > >     >     >         >> -Regards,
> > >     > > >     >     >         >> Mayuresh R. Gharat
> > >     > > >     >     >         >> (862) 250-7125
> > >     > > >     >     >         >> The information contained in this
> email
> > is
> > >     > > strictly
> > >     > > >     >     > confidential and for
> > >     > > >     >     >         >> the use of the addressee only, unless
> > > otherwise
> > >     > > >     > indicated. If
> > >     > > >     >     > you are not
> > >     > > >     >     >         >> the intended recipient, please do not
> > > read,
> > >     > copy,
> > >     > > > use or
> > >     > > >     >     > disclose to others
> > >     > > >     >     >         >> this message or any attachment. Please
> > > also
> > >     > notify
> > >     > > > the
> > >     > > >     > sender
> > >     > > >     >     > by replying
> > >     > > >     >     >         >> to this email or by telephone (+44(020
> > > 7896
> > >     > 0011)
> > >     > > > and then
> > >     > > >     >     > delete the email
> > >     > > >     >     >         >> and any copies of it. Opinions,
> > > conclusion (etc)
> > >     > > > that do
> > >     > > >     > not
> > >     > > >     >     > relate to the
> > >     > > >     >     >         >> official business of this company
> shall
> > be
> > >     > > > understood as
> > >     > > >     >     > neither given nor
> > >     > > >     >     >         >> endorsed by it. IG is a trading name
> of
> > IG
> > >     > Markets
> > >     > > >     > Limited (a
> > >     > > >     >     > company
> > >     > > >     >     >         >> registered in England and Wales,
> company
> > > number
> > >     > > > 04008957)
> > >     > > >     > and
> > >     > > >     >     > IG Index
> > >     > > >     >     >         >> Limited (a company registered in
> England
> > > and
> > >     > > Wales,
> > >     > > >     > company
> > >     > > >     >     > number
> > >     > > >     >     >         >> 01190902). Registered address at
> Cannon
> > > Bridge
> > >     > > > House, 25
> > >     > > >     >     > Dowgate Hill,
> > >     > > >     >     >         >> London EC4R 2YA. Both IG Markets
> Limited
> > >     > (register
> > >     > > > number
> > >     > > >     >     > 195355) and IG
> > >     > > >     >     >         >> Index Limited (register number 114059)
> > are
> > >     > > > authorised and
> > >     > > >     >     > regulated by the
> > >     > > >     >     >         >> Financial Conduct Authority.
> > >     > > >     >     >         >>
> > >     > > >     >     >         >
> > >     > > >     >     >         >
> > >     > > >     >     >         >
> > >     > > >     >     >         > --
> > >     > > >     >     >         > -Regards,
> > >     > > >     >     >         > Mayuresh R. Gharat
> > >     > > >     >     >         > (862) 250-7125
> > >     > > >     >     >         >
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >         --
> > >     > > >     >     >         -Regards,
> > >     > > >     >     >         Mayuresh R. Gharat
> > >     > > >     >     >         (862) 250-7125
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >     The information contained in this email is
> > > strictly
> > >     > > > confidential
> > >     > > >     > and
> > >     > > >     >     > for the use of the addressee only, unless
> otherwise
> > >     > > indicated.
> > >     > > > If
> > >     > > >     > you are
> > >     > > >     >     > not the intended recipient, please do not read,
> > > copy, use
> > >     > or
> > >     > > >     > disclose to
> > >     > > >     >     > others this message or any attachment. Please
> also
> > > notify
> > >     > the
> > >     > > > sender
> > >     > > >     > by
> > >     > > >     >     > replying to this email or by telephone (+44(020
> > 7896
> > > 0011)
> > >     > > and
> > >     > > > then
> > >     > > >     > delete
> > >     > > >     >     > the email and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > >     > > > that do
> > >     > > >     > not
> > >     > > >     >     > relate to the official business of this company
> > > shall be
> > >     > > > understood
> > >     > > >     > as
> > >     > > >     >     > neither given nor endorsed by it. IG is a trading
> > > name of
> > >     > IG
> > >     > > > Markets
> > >     > > >     >     > Limited (a company registered in England and
> Wales,
> > > company
> > >     > > > number
> > >     > > >     >     > 04008957) and IG Index Limited (a company
> > registered
> > > in
> > >     > > > England and
> > >     > > >     > Wales,
> > >     > > >     >     > company number 01190902). Registered address at
> > > Cannon
> > >     > Bridge
> > >     > > > House,
> > >     > > >     > 25
> > >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > > Limited
> > >     > > > (register
> > >     > > >     > number
> > >     > > >     >     > 195355) and IG Index Limited (register number
> > > 114059) are
> > >     > > > authorised
> > >     > > >     > and
> > >     > > >     >     > regulated by the Financial Conduct Authority.
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >     >
> > >     > > >     >
> > >     > > >     >
> > >     > > >     > The information contained in this email is strictly
> > > confidential
> > >     > > and
> > >     > > > for
> > >     > > >     > the use of the addressee only, unless otherwise
> > indicated.
> > > If you
> > >     > > > are not
> > >     > > >     > the intended recipient, please do not read, copy, use
> or
> > > disclose
> > >     > > to
> > >     > > > others
> > >     > > >     > this message or any attachment. Please also notify the
> > > sender by
> > >     > > > replying
> > >     > > >     > to this email or by telephone (+44(020 7896 0011) and
> > then
> > > delete
> > >     > > > the email
> > >     > > >     > and any copies of it. Opinions, conclusion (etc) that
> do
> > > not
> > >     > relate
> > >     > > > to the
> > >     > > >     > official business of this company shall be understood
> as
> > > neither
> > >     > > > given nor
> > >     > > >     > endorsed by it. IG is a trading name of IG Markets
> > Limited
> > > (a
> > >     > > company
> > >     > > >     > registered in England and Wales, company number
> 04008957)
> > > and IG
> > >     > > > Index
> > >     > > >     > Limited (a company registered in England and Wales,
> > company
> > >     > number
> > >     > > >     > 01190902). Registered address at Cannon Bridge House,
> 25
> > > Dowgate
> > >     > > > Hill,
> > >     > > >     > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355)
> > >     > > > and IG
> > >     > > >     > Index Limited (register number 114059) are authorised
> and
> > >     > regulated
> > >     > > > by the
> > >     > > >     > Financial Conduct Authority.
> > >     > > >     >
> > >     > > >
> > >     > > >
> > >     > > > The information contained in this email is strictly
> > confidential
> > > and
> > >     > for
> > >     > > > the use of the addressee only, unless otherwise indicated. If
> > > you are
> > >     > not
> > >     > > > the intended recipient, please do not read, copy, use or
> > > disclose to
> > >     > > others
> > >     > > > this message or any attachment. Please also notify the sender
> > by
> > >     > replying
> > >     > > > to this email or by telephone (+44(020 7896 0011) and then
> > > delete the
> > >     > > email
> > >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to
> > >     > > the
> > >     > > > official business of this company shall be understood as
> > neither
> > > given
> > >     > > nor
> > >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > >     > > > registered in England and Wales, company number 04008957) and
> > IG
> > > Index
> > >     > > > Limited (a company registered in England and Wales, company
> > > number
> > >     > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355) and
> > >     > IG
> > >     > > > Index Limited (register number 114059) are authorised and
> > > regulated by
> > >     > > the
> > >     > > > Financial Conduct Authority.
> > >     > > >
> > >     > > The information contained in this email is strictly
> confidential
> > > and for
> > >     > > the use of the addressee only, unless otherwise indicated. If
> you
> > > are not
> > >     > > the intended recipient, please do not read, copy, use or
> disclose
> > > to
> > >     > others
> > >     > > this message or any attachment. Please also notify the sender
> by
> > > replying
> > >     > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > > the
> > >     > email
> > >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to
> > >     > the
> > >     > > official business of this company shall be understood as
> neither
> > > given
> > >     > nor
> > >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > >     > > registered in England and Wales, company number 04008957) and
> IG
> > > Index
> > >     > > Limited (a company registered in England and Wales, company
> > number
> > >     > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > > Hill,
> > >     > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > and IG
> > >     > > Index Limited (register number 114059) are authorised and
> > > regulated by
> > >     > the
> > >     > > Financial Conduct Authority.
> > >     > >
> > >     > >
> > >     > The information contained in this email is strictly confidential
> > and
> > > for
> > >     > the use of the addressee only, unless otherwise indicated. If you
> > > are not
> > >     > the intended recipient, please do not read, copy, use or disclose
> > to
> > > others
> > >     > this message or any attachment. Please also notify the sender by
> > > replying
> > >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > > the email
> > >     > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > > to the
> > >     > official business of this company shall be understood as neither
> > > given nor
> > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > >     > registered in England and Wales, company number 04008957) and IG
> > > Index
> > >     > Limited (a company registered in England and Wales, company
> number
> > >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > > and IG
> > >     > Index Limited (register number 114059) are authorised and
> regulated
> > > by the
> > >     > Financial Conduct Authority.
> > >     >
> > >     >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Ewen,

I think the easiest way to show this is with code.

As you can see we keep the existing behaviour for code/binaries calling the pre-existing constructors, whereby if the value is null the tombstone is set to true.

Regards
Mike



    /**
     * Creates a record with a specified timestamp to be sent to a specified topic and partition
     *
     * @param topic The topic the record will be appended to
     * @param partition The partition to which the record should be sent
     * @param timestamp The timestamp of the record
     * @param tombstone if the record should be treated as a tombstone if the topic is compacted
     * @param key The key that will be included in the record
     * @param value The record contents
     */
    public ProducerRecord(String topic, Integer partition, Boolean tombstone, Long timestamp, K key, V value) {
        if (topic == null)
            throw new IllegalArgumentException("Topic cannot be null.");
        if (timestamp != null && timestamp < 0)
            throw new IllegalArgumentException(
                    String.format("Invalid timestamp: %d. Timestamp should always be non-negative or null.", timestamp));
        if (partition != null && partition < 0)
            throw new IllegalArgumentException(
                    String.format("Invalid partition: %d. Partition number should always be non-negative or null.", partition));
        if (!tombstone && value == null){
            throw new IllegalArgumentException(
                    String.format("Tombstone must be true if null value"));
        }
        this.topic = topic;
        this.partition = partition;
        this.tombstone = tombstone;
        this.key = key;
        this.value = value;
        this.timestamp = timestamp;
    }

    public ProducerRecord(String topic, Integer partition, Long timestamp, K key, V value) {
        this(topic, partition, value == null, timestamp, key, value);
    }
________________________________________
From: Ewen Cheslack-Postava <ew...@confluent.io>
Sent: Sunday, December 11, 2016 5:45 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

It seemed like this was addressed in the migration section, wasn't it? The
V2 vs V3 requests and the fact that the broker will downgrade the message
format (losing zero copy) if you issues a V2 request to a broker using V3
format handles compatibility, does it not? The existing consumers won't see
the extra metadata in the value, but they will get a null instead and treat
it as a tombstone. Certainly there is a performance impact, but it seemed
compatible.

I'm worried about this though:


   - *NOTE *: With the new version of producer client using ProduceRequest
   V3 (magic byte = 2), a non tombstone (tombstone bit not set) and null value
   should not be allowed as the older version of consumer using FetchRequest
   V2 will think of this as a tombstone when its actually not.


Unless I'm misunderstanding, this ends up breaking binary compatibility for
the Java API. It sounds like this suggests that passing a null value to the
existing ProducerRecord constructors would generate an exception since you
didn't explicitly enable the tombstone (via whatever new constructor is
provided). But that means you can't swap in a newer client jar without
recompiling and get the same behavior if your app deletes keys using the
current approach because your app will start throwing exceptions. Maybe
this is a tradeoff we're ok with, but we've tried pretty hard to avoid
breaking compatibility like this so far -- it makes picking up bug fixes in
the clients harder for users of frameworks that have to pin to earlier
default versions for compatibility.

But then later the KIP says:


   - The old constructors for ProducerRecord without this parameter will be
   kept but updated so that their default behaviour if setting null value will
   be the tombstone will be set to true to keep existing behaviour.


So maybe I am misinterpreting something.

And just a nit re: motivation section, item 6 would be clearer for a union
schema with null (or optional schemas in other formats), e.g. [null,
string], in which case losing the schema truly is losing information
(whereas null is already the only valid value for a pure null schema).

-Ewen


On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jay,
>
> Good point this detail is missing in the KIP write up. Ive added this now.
>
> Essentially simply just upgrading the clients we do not expect any client
> app code change needed.
>
> Cheers
> Mike
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Saturday, December 10, 2016 10:51 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Michael,
>
> The compatibility section goes through the migration path, but isn't the
> bigger compatibility issue with existing apps? There are many (probably
> thousands) of apps in production that use this feature and send null to
> mean delete. It seems like this would break compatibility with them, and
> they would have to be rewritten to right?
>
> -Jay
>
> On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jun,
> >
> > 4) On v3 we honour the tombstone. As such we expect it to be set
> correctly
> > as per the KIP.
> >
> > 4.1) why would we want to produce an error when its v3? This is the exact
> > purpose to support non-null tombstone’s
> > 4.2) again here im unclear on the question on the v3, produce request we
> > expect the tombstone flag to be set correctly.
> >
> > 4.4) compaction only occurs on compacted topics, the bit makes no
> > difference and not looked at on un-compacted (time/size based eviction).
> >
> >
> > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> >
> >     Hi, Michael,
> >
> >     4. Then, I think I misunderstood this point. Could you document the
> >     following points in the wiki?
> >     4.1 If producer V3 sets tombstone, but provides a non-null value,
> does
> > the
> >     send() get an error or does the producer automatically set the value
> to
> >     null?
> >     4.2 If producer V3 doesn't set tombstone, but provides a null value,
> > does
> >     the send() get an error or does the producer automatically sets the
> >     tombstone?
> >     4.3 Does the broker only expect messages that (a) have no tombstone
> and
> >     non-null value; (b) have tombstone and null value and reject the
> > messages
> >     with an error code otherwise?
> >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on
> not?
> >
> >     Thanks,
> >
> >     Jun
> >
> >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> >     wrote:
> >
> >     > Not at all.  This only acts on compacted topics just as what occurs
> > today
> >     >
> >     > Sent using OWA for iPhone
> >     > ________________________________________
> >     > From: Jun Rao <ju...@confluent.io>
> >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> >     > To: dev@kafka.apache.org
> >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     >
> >     > Hi, Michael,
> >     >
> >     > 4. Hmm, does that mean the new client library can never send a null
> > message
> >     > even to a regular topic? This seems like a change of the existing
> > behavior.
> >     >
> >     > Thanks,
> >     >
> >     > Jun
> >     >
> >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > wrote:
> >     >
> >     > > Hi Jun,
> >     > >
> >     > > Re 4) That's because we expect the tombstone value to be set
> > correctly if
> >     > > message bit is 2, as such if an older client sends in on old
> > message the
> >     > > message is upcast and the bit is set correctly. And such no
> longer
> > need
> >     > to
> >     > > check the value. Mayuresh can you confirm my thinking and
> > understanding
> >     > of
> >     > > what we've discussed?
> >     > >
> >     > > The second point I understand what you're getting at now my
> > apologies.
> >     > Yes
> >     > > this makes sense to save on touching the message, if we're the
> > only kip
> >     > > going in, in this release.
> >     > >
> >     > > Cheers
> >     > > Mike
> >     > >
> >     > > Sent using OWA for iPhone
> >     > > ________________________________________
> >     > > From: Jun Rao <ju...@confluent.io>
> >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> >     > > To: dev@kafka.apache.org
> >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > >
> >     > > Hi, Michael,
> >     > >
> >     > > 4. Is this updated in the wiki? The text "If the magic byte on
> > message is
> >     > > 2, the broker should use the tombstone bit for log compaction."
> > doesn't
> >     > > seem to have changed.
> >     > >
> >     > > 2. My point is that if we change the message format just for this
> > KIP, we
> >     > > should consider whether it's worth optimizing the down conversion
> > path
> >     > > (i.e., decide whether a conversion is needed by just looking at
> the
> >     > > tombstone bit in the wrapper message) since tombstone will be
> used
> >     > rarely.
> >     > > However, if the message format change here is combined with other
> > KIPs,
> >     > > then this optimization likely won't be needed. The latter
> probably
> > makes
> >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> >     > >
> >     > > Other than those, +1 from me,
> >     > >
> >     > > Thanks,
> >     > >
> >     > > Jun
> >     > >
> >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > > wrote:
> >     > >
> >     > > > Hi Jun
> >     > > >
> >     > > > do we have your vote on this now?
> >     > > >
> >     > > > Any other concerns?
> >     > > >
> >     > > > Cheers
> >     > > > Mike
> >     > > >
> >     > > > Sent using OWA for iPhone
> >     > > > ________________________________________
> >     > > > From: Michael Pearce <Mi...@ig.com>
> >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> >     > > > To: dev@kafka.apache.org
> >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > >
> >     > > > Hi Jun,
> >     > > >
> >     > > > Have updated. Thanks again for the feedback.
> >     > > >
> >     > > > Agree yes we should align up when it gets to that, I assume
> > you’ve
> >     > > flagged
> >     > > > the same in the other KIP?
> >     > > >
> >     > > > I think for now let’s get this KIP approved, then we can start
> > the work
> >     > > to
> >     > > > get an initial PR, then we can discuss how to align the two if
> > needed.
> >     > > >
> >     > > > Cheers,
> >     > > > Mike
> >     > > >
> >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> >     > > >
> >     > > >     Hi, Michael,
> >     > > >
> >     > > >     For 2), this is fine. Could you update the KIP wiki to make
> > this
> >     > and
> >     > > > other
> >     > > >     points clearer? Other than that, the KIP looks good to me.
> >     > > >
> >     > > >     An orthogonal thing is that there are other KIPs such as
> > KIP-98
> >     > that
> >     > > > also
> >     > > >     intend to change the message format. If they all get
> > approved, we
> >     > > > should
> >     > > >     think about whether it's better to just bump up the magic
> > byte once
> >     > > to
> >     > > >     incorporate multiple format changes like we did in
> > KIP-31/KIP-32.
> >     > > >
> >     > > >     Thanks,
> >     > > >
> >     > > >     Jun
> >     > > >
> >     > > >
> >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> >     > > Michael.Pearce@ig.com>
> >     > > >     wrote:
> >     > > >
> >     > > >     > Hi Jao
> >     > > >     >
> >     > > >     > Thanks for the response. Sorry for slow reply, both with
> > personal
> >     > > > sickness
> >     > > >     > and also battling some critical issues encountered since
> >     > upgrading
> >     > > to
> >     > > >     > 0.10.1.0
> >     > > >     >
> >     > > >     > 1) Thans for spotting, Document error where we branched
> > this KIP
> >     > > from
> >     > > >     > KIP-82, will get that removed.
> >     > > >     > 2) Intent is to do this just at the record message level.
> >     > > >     > 3) Thanks for spotting, Will ensure this is corrected.
> >     > > >     > 4) As per discussion thread we will support tombstone +
> > null
> >     > value,
> >     > > >     > tombstone + non null value, no tombstone + null value.
> >     > > >     > 5) I believe this was in the discussion thread, @Mayuresh
> > is this
> >     > > >     > something we’ve overlooked? I thought we would down
> > convert and
> >     > > > remove the
> >     > > >     > value so the old consumer had existing behavior, or is
> > there
> >     > > > something we
> >     > > >     > haven’t thought about?
> >     > > >     >
> >     > > >     > Cheers
> >     > > >     > Mike
> >     > > >     >
> >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
> wrote:
> >     > > >     >
> >     > > >     >     Hi, Michael,
> >     > > >     >
> >     > > >     >     Thanks for the KIP. A few comments below.
> >     > > >     >
> >     > > >     >     1. The message format change contains "HeadersLength
> >     > Headers".
> >     > > > Is that
> >     > > >     >     intended?
> >     > > >     >
> >     > > >     >     2. For compressed messageset, is the tombstone bit
> > only set
> >     > at
> >     > > > the
> >     > > >     > shallow
> >     > > >     >     level? Do we always leave that bit in the wrapper
> > message
> >     > > unset?
> >     > > > An
> >     > > >     >     alternative is to set the tombstone bit in the
> wrapper
> > if at
> >     > > > least one
> >     > > >     >     inner message has the tombstone bit set. This makes
> > things a
> >     > > bit
> >     > > > more
> >     > > >     >     complicated, but we could potentially exploit that
> for
> >     > > > optimizing down
> >     > > >     >     conversion. For example, we only need to convert
> > messages
> >     > with
> >     > > > magic 2
> >     > > >     > to
> >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > (conversion is
> >     > > > always
> >     > > >     > needed
> >     > > >     >     from magic 2 to magic 0). Not sure if the
> optimization
> > is
> >     > worth
> >     > > > the
> >     > > >     >     complexity though.
> >     > > >     >
> >     > > >     >     3. The referencing of the new version of
> >     > > > ProducerRequest/FetchRequest
> >     > > >     > is
> >     > > >     >     inconsistent (v4 vs v3). Since our convention starts
> at
> >     > version
> >     > > > at 0, I
> >     > > >     >     think the new version would be 3.
> >     > > >     >
> >     > > >     >     4. "If the magic byte on message is 2, the broker
> > should use
> >     > > the
> >     > > >     > tombstone
> >     > > >     >     bit for log compaction." What about null value? My
> >     > > understanding
> >     > > > is
> >     > > >     > that
> >     > > >     >     null value will be treated the same as setting the
> > tombstone
> >     > > bit.
> >     > > >     >
> >     > > >     >     5. For the migration path, it would be useful to
> > describe the
> >     > > > down
> >     > > >     >     conversion path to consumers (i.e., brokers on
> message
> > format
> >     > > > 0.10.2
> >     > > >     > and
> >     > > >     >     consumers on older version).
> >     > > >     >
> >     > > >     >     Thanks,
> >     > > >     >
> >     > > >     >     Jun
> >     > > >     >
> >     > > >     >
> >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> >     > > > Michael.Pearce@ig.com
> >     > > >     > >
> >     > > >     >     wrote:
> >     > > >     >
> >     > > >     >     > Hi All,
> >     > > >     >     >
> >     > > >     >     > We have been discussing in the below thread and
> final
> >     > changes
> >     > > > have
> >     > > >     > been
> >     > > >     >     > made to the KIP wiki based on these discussions.
> >     > > >     >     >
> >     > > >     >     > We would now like to put to the vote the following
> > KIP:
> >     > > >     >     > https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-
> >     > 87+-+
> >     > > >     >     > Add+Compaction+Tombstone+Flag
> >     > > >     >     >
> >     > > >     >     > This kip is for having a distinct compaction
> > attribute
> >     > > > “tombstone”
> >     > > >     > flag
> >     > > >     >     > instead of relying on null value, allowing non-null
> > value
> >     > > > delete
> >     > > >     > messages.
> >     > > >     >     >
> >     > > >     >     > Many thanks,
> >     > > >     >     > Michael
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> >     > > Michael.Pearce@ig.com>
> >     > > >     > wrote:
> >     > > >     >     >
> >     > > >     >     >     Hi Mayuresh,
> >     > > >     >     >
> >     > > >     >     >     LGTM. Ive just made one small adjustment
> > updating the
> >     > > wire
> >     > > >     > protocol to
> >     > > >     >     > show the magic byte bump.
> >     > > >     >     >
> >     > > >     >     >     Do we think we’re good to put to a vote? Is
> > there any
> >     > > > other bits
> >     > > >     >     > needing discussion?
> >     > > >     >     >
> >     > > >     >     >     Cheers
> >     > > >     >     >     Mike
> >     > > >     >     >
> >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> >     > > >     > gharatmayuresh15@gmail.com>
> >     > > >     >     > wrote:
> >     > > >     >     >
> >     > > >     >     >         Hi Michael,
> >     > > >     >     >
> >     > > >     >     >         I have updated the migration section of the
> > KIP.
> >     > Can
> >     > > > you
> >     > > >     > please
> >     > > >     >     > take a look?
> >     > > >     >     >
> >     > > >     >     >         Thanks,
> >     > > >     >     >
> >     > > >     >     >         Mayuresh
> >     > > >     >     >
> >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh
> > Gharat <
> >     > > >     >     > gharatmayuresh15@gmail.com
> >     > > >     >     >         > wrote:
> >     > > >     >     >
> >     > > >     >     >         > Hi Michael,
> >     > > >     >     >         >
> >     > > >     >     >         > That whilst sending tombstone and non
> null
> > value,
> >     > > the
> >     > > >     > consumer
> >     > > >     >     > can expect
> >     > > >     >     >         > only to receive the non-null message only
> > in step
> >     > > > (3) is
> >     > > >     > this
> >     > > >     >     > correct?
> >     > > >     >     >         > ---> I do agree with you here.
> >     > > >     >     >         >
> >     > > >     >     >         > Becket, Ismael : can you guys review the
> >     > migration
> >     > > > plan
> >     > > >     > listed
> >     > > >     >     > above using
> >     > > >     >     >         > magic byte?
> >     > > >     >     >         >
> >     > > >     >     >         > Thanks,
> >     > > >     >     >         >
> >     > > >     >     >         > Mayuresh
> >     > > >     >     >         >
> >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael
> > Pearce <
> >     > > >     >     > Michael.Pearce@ig.com>
> >     > > >     >     >         > wrote:
> >     > > >     >     >         >
> >     > > >     >     >         >> Many thanks for this Mayuresh. I don't
> > have any
> >     > > >     > objections.
> >     > > >     >     >         >>
> >     > > >     >     >         >> I assume we should state:
> >     > > >     >     >         >>
> >     > > >     >     >         >> That whilst sending tombstone and non
> null
> >     > value,
> >     > > > the
> >     > > >     > consumer
> >     > > >     >     > can expect
> >     > > >     >     >         >> only to receive the non-null message
> only
> > in
> >     > step
> >     > > > (3) is
> >     > > >     > this
> >     > > >     >     > correct?
> >     > > >     >     >         >>
> >     > > >     >     >         >> Cheers
> >     > > >     >     >         >> Mike
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >> Sent using OWA for iPhone
> >     > > >     >     >         >> ______________________________
> __________
> >     > > >     >     >         >> From: Mayuresh Gharat <
> >     > gharatmayuresh15@gmail.com
> >     > > >
> >     > > >     >     >         >> Sent: Thursday, November 17, 2016
> 5:18:41
> > PM
> >     > > >     >     >         >> To: dev@kafka.apache.org
> >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
> > Compaction
> >     > > > Tombstone
> >     > > >     > Flag
> >     > > >     >     >         >>
> >     > > >     >     >         >> Hi Ismael,
> >     > > >     >     >         >>
> >     > > >     >     >         >> Thanks for the explanation.
> >     > > >     >     >         >> Specially I like this part where in you
> >     > mentioned
> >     > > > we can
> >     > > >     > get
> >     > > >     >     > rid of the
> >     > > >     >     >         >> older null value support for log
> > compaction
> >     > later
> >     > > > on,
> >     > > >     > here :
> >     > > >     >     >         >> We can't change semantics of the message
> > format
> >     > > > without
> >     > > >     > having
> >     > > >     >     > a long
> >     > > >     >     >         >> transition period. And we can't rely
> >     > > >     >     >         >> on people reading documentation or
> acting
> > on a
> >     > > > warning for
> >     > > >     >     > something so
> >     > > >     >     >         >> fundamental. As such, my take is that we
> > need to
> >     > > > bump the
> >     > > >     > magic
> >     > > >     >     > byte. The
> >     > > >     >     >         >> good news is
> >     > > >     >     >         >> that we don't have to support all
> versions
> >     > > forever.
> >     > > > We
> >     > > >     > have
> >     > > >     >     > said that we
> >     > > >     >     >         >> will support direct upgrades for 2
> years.
> > That
> >     > > > means that
> >     > > >     >     > message format
> >     > > >     >     >         >> version n could, in theory, be removed 2
> > years
> >     > > > after the
> >     > > >     > it's
> >     > > >     >     > introduced.
> >     > > >     >     >         >>
> >     > > >     >     >         >> Just a heads up, I would like to mention
> > that
> >     > even
> >     > > > without
> >     > > >     >     > bumping magic
> >     > > >     >     >         >> byte, we will *NOT* loose zero copy as
> in
> > the
> >     > > > client(x+1)
> >     > > >     > in my
> >     > > >     >     >         >> explanation
> >     > > >     >     >         >> above will convert internally a null
> > value to
> >     > > have a
> >     > > >     > tombstone
> >     > > >     >     > bit set and
> >     > > >     >     >         >> a tombstone bit set to have a null value
> >     > > > automatically
> >     > > >     >     > internally and by
> >     > > >     >     >         >> the time we move to version (x+2), the
> > clients
> >     > > > would have
> >     > > >     >     > upgraded.
> >     > > >     >     >         >> Obviously if we support a request from
> >     > > consumer(x),
> >     > > > we
> >     > > >     > will
> >     > > >     >     > loose zero
> >     > > >     >     >         >> copy
> >     > > >     >     >         >> but that is the same case with magic
> byte.
> >     > > >     >     >         >>
> >     > > >     >     >         >> But if magic byte bump makes life easier
> > for
> >     > > > transition
> >     > > >     > for the
> >     > > >     >     > above
> >     > > >     >     >         >> reasons that you explained, I am OK with
> > it
> >     > since
> >     > > > we are
> >     > > >     > going
> >     > > >     >     > to meet the
> >     > > >     >     >         >> end goal down the road :)
> >     > > >     >     >         >>
> >     > > >     >     >         >> On a side note can we update the doc
> here
> > on
> >     > magic
> >     > > > byte
> >     > > >     > to say
> >     > > >     >     > that "*it
> >     > > >     >     >         >> should be bumped whenever the message
> > format is
> >     > > > changed
> >     > > >     > or the
> >     > > >     >     >         >> interpretation of message format (usage
> > of the
> >     > > > reserved
> >     > > >     > bits as
> >     > > >     >     > well) is
> >     > > >     >     >         >> changed*".
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >> Hi Michael,
> >     > > >     >     >         >>
> >     > > >     >     >         >> Here is the update plan that we
> discussed
> >     > offline
> >     > > >     > yesterday :
> >     > > >     >     >         >>
> >     > > >     >     >         >> Currently the magic-byte which
> > corresponds to
> >     > the
> >     > > >     >     > "message.format.version"
> >     > > >     >     >         >> is set to 1.
> >     > > >     >     >         >>
> >     > > >     >     >         >> 1) On broker it will be set to 1
> > initially.
> >     > > >     >     >         >>
> >     > > >     >     >         >> 2) When a producer client sends a
> message
> > with
> >     > > > magic-byte
> >     > > >     > = 2,
> >     > > >     >     > since the
> >     > > >     >     >         >> broker is on magic-byte = 1, we will
> down
> >     > convert
> >     > > > it,
> >     > > >     > which
> >     > > >     >     > means if the
> >     > > >     >     >         >> tombstone bit is set, the value will be
> > set to
> >     > > > null. A
> >     > > >     > consumer
> >     > > >     >     >         >> understanding magic-byte = 1, will still
> > work
> >     > with
> >     > > > this. A
> >     > > >     >     > consumer
> >     > > >     >     >         >> working
> >     > > >     >     >         >> with magic-byte =2 will also be able to
> >     > understand
> >     > > > this,
> >     > > >     > since
> >     > > >     >     > it
> >     > > >     >     >         >> understands the tombstone.
> >     > > >     >     >         >> Now there is still the question of
> > supporting a
> >     > > >     > non-tombstone
> >     > > >     >     > and null
> >     > > >     >     >         >> value from producer client with
> > magic-byte = 2.*
> >     > > (I
> >     > > > am
> >     > > >     > not sure
> >     > > >     >     > if we
> >     > > >     >     >         >> should support this. Ismael/Becket can
> > comment
> >     > > > here)*
> >     > > >     >     >         >>
> >     > > >     >     >         >> 3) When almost all the clients have
> > upgraded,
> >     > the
> >     > > >     >     > message.format.version
> >     > > >     >     >         >> on
> >     > > >     >     >         >> the broker can be changed to 2, where in
> > the
> >     > down
> >     > > >     > conversion in
> >     > > >     >     > the above
> >     > > >     >     >         >> step will not happen. If at this point
> we
> > get a
> >     > > > consumer
> >     > > >     >     > request from a
> >     > > >     >     >         >> older consumer, we might have to down
> > convert
> >     > > where
> >     > > > in we
> >     > > >     > loose
> >     > > >     >     > zero copy,
> >     > > >     >     >         >> but these cases should be rare.
> >     > > >     >     >         >>
> >     > > >     >     >         >> Becket can you review this plan and add
> > more
> >     > > > details if I
> >     > > >     > have
> >     > > >     >     >         >> missed/wronged something, before we put
> > it on
> >     > KIP.
> >     > > >     >     >         >>
> >     > > >     >     >         >> Thanks,
> >     > > >     >     >         >>
> >     > > >     >     >         >> Mayuresh
> >     > > >     >     >         >>
> >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
> Michael
> >     > Pearce <
> >     > > >     >     > Michael.Pearce@ig.com>
> >     > > >     >     >         >> wrote:
> >     > > >     >     >         >>
> >     > > >     >     >         >> > Thanks guys, for discussing this
> > offline and
> >     > > > getting
> >     > > >     > some
> >     > > >     >     > consensus.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > So its clear for myself and others
> what
> > is
> >     > > > proposed now
> >     > > >     > (i
> >     > > >     >     > think i
> >     > > >     >     >         >> > understand, but want to make sure)
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Could i ask either directly update the
> > kip to
> >     > > > detail the
> >     > > >     >     > migration
> >     > > >     >     >         >> > strategy, or (re-)state your offline
> > discussed
> >     > > and
> >     > > >     > agreed
> >     > > >     >     > migration
> >     > > >     >     >         >> > strategy based on a magic byte is in
> > this
> >     > > thread.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > The main original driver for the KIP
> > was to
> >     > > > support
> >     > > >     >     > compaction where
> >     > > >     >     >         >> value
> >     > > >     >     >         >> > isn't null, based off the discussions
> on
> >     > KIP-82
> >     > > > thread.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > We should be able to support
> > non-tombstone +
> >     > > null
> >     > > > value
> >     > > >     > by the
> >     > > >     >     >         >> completion
> >     > > >     >     >         >> > of the KIP, as we noted when
> discussing
> > this
> >     > > kip,
> >     > > > having
> >     > > >     >     > logic based on
> >     > > >     >     >         >> a
> >     > > >     >     >         >> > null value isn't very clean and also
> > separates
> >     > > the
> >     > > >     > concerns.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > As discussed already though we can
> > split this
> >     > > into
> >     > > >     > KIP-87a
> >     > > >     >     > and KIP-87b
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Where we look to deliver KIP-87a on a
> >     > compacted
> >     > > > topic
> >     > > >     > (to
> >     > > >     >     > address the
> >     > > >     >     >         >> > immediate issues)
> >     > > >     >     >         >> > * tombstone + null value
> >     > > >     >     >         >> > * tombstone + non-null value
> >     > > >     >     >         >> > * non-tombstone + non-null value
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Then we can discuss once KIP-87a is
> > completed
> >     > > > options
> >     > > >     > later
> >     > > >     >     > and how we
> >     > > >     >     >         >> > support the second part KIP-87b to
> > deliver:
> >     > > >     >     >         >> > * non-tombstone + null value
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Cheers
> >     > > >     >     >         >> > Mike
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > ______________________________
> > __________
> >     > > >     >     >         >> > From: Becket Qin <
> becket.qin@gmail.com>
> >     > > >     >     >         >> > Sent: Thursday, November 17, 2016 1:43
> > AM
> >     > > >     >     >         >> > To: dev@kafka.apache.org
> >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add
> > Compaction
> >     > > >     > Tombstone Flag
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Renu, Mayuresh and I had an offline
> >     > discussion,
> >     > > > and
> >     > > >     > following
> >     > > >     >     > is a brief
> >     > > >     >     >         >> > summary.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > 1. We agreed that not bumping up magic
> > value
> >     > may
> >     > > > result
> >     > > >     > in
> >     > > >     >     > losing zero
> >     > > >     >     >         >> copy
> >     > > >     >     >         >> > during migration.
> >     > > >     >     >         >> > 2. Given that bumping up magic value
> is
> > almost
> >     > > > free and
> >     > > >     > has
> >     > > >     >     > benefit of
> >     > > >     >     >         >> > avoiding potential performance issue.
> > It is
> >     > > > probably
> >     > > >     > worth
> >     > > >     >     > doing.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > One issue we still need to think about
> > is
> >     > > whether
> >     > > > we
> >     > > >     > want to
> >     > > >     >     > support a
> >     > > >     >     >         >> > non-tombstone message with null value.
> >     > > >     >     >         >> > Currently it is not supported by
> Kafka.
> > If we
> >     > > > allow a
> >     > > >     >     > non-tombstone null
> >     > > >     >     >         >> > value message to exist after KIP-87.
> The
> >     > problem
> >     > > > is
> >     > > >     > that such
> >     > > >     >     > message
> >     > > >     >     >         >> will
> >     > > >     >     >         >> > not be supported by the consumers
> prior
> > to
> >     > > KIP-87.
> >     > > >     > Because a
> >     > > >     >     > null value
> >     > > >     >     >         >> > will always be interpreted to a
> > tombstone.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > One option is that we keep the current
> > way,
> >     > i.e.
> >     > > > do not
> >     > > >     >     > support such
> >     > > >     >     >         >> > message. It would be good to know if
> > there is
> >     > a
> >     > > >     > concrete use
> >     > > >     >     > case for
> >     > > >     >     >         >> such
> >     > > >     >     >         >> > message. If there is not, we can
> > probably just
> >     > > not
> >     > > >     > support it.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Thanks,
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > JIangjie (Becket) Qin
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> > Mayuresh
> >     > > Gharat <
> >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> >     > > >     >     >         >> > > wrote:
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > > Hi Ismael,
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > This is something I can think of for
> >     > migration
> >     > > > plan:
> >     > > >     >     >         >> > > So the migration plan can look
> > something
> >     > like
> >     > > > this,
> >     > > >     > with up
> >     > > >     >     >         >> conversion :
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > 1) Currently lets say we have Broker
> > at
> >     > > version
> >     > > > x.
> >     > > >     >     >         >> > > 2) Currently we have clients at
> > version x.
> >     > > >     >     >         >> > > 3) a) We move the version to
> > Broker(x+1) :
> >     > > > supports
> >     > > >     > both
> >     > > >     >     > tombstone and
> >     > > >     >     >         >> > null
> >     > > >     >     >         >> > > for log compaction.
> >     > > >     >     >         >> > >     b) We upgrade the client to
> > version
> >     > > > client(x+1) :
> >     > > >     > if in
> >     > > >     >     > the
> >     > > >     >     >         >> producer
> >     > > >     >     >         >> > > client(x+1) the value is set to
> null,
> > we
> >     > will
> >     > > >     > automatically
> >     > > >     >     > set the
> >     > > >     >     >         >> > > Tombstone bit internally. If the
> > producer
> >     > > > client(x+1)
> >     > > >     > sets
> >     > > >     >     > the
> >     > > >     >     >         >> tombstone
> >     > > >     >     >         >> > > itself, well and good. For producer
> >     > client(x),
> >     > > > the
> >     > > >     > broker
> >     > > >     >     > will up
> >     > > >     >     >         >> convert
> >     > > >     >     >         >> > > to have the tombstone bit.
> > Broker(x+1) is
> >     > > > supporting
> >     > > >     > both.
> >     > > >     >     > Consumer
> >     > > >     >     >         >> > > client(x+1) will be aware of this
> and
> > should
> >     > > be
> >     > > > able
> >     > > >     > to
> >     > > >     >     > handle this.
> >     > > >     >     >         >> For
> >     > > >     >     >         >> > > consumer client(x) we will down
> > convert the
> >     > > > message
> >     > > >     > on the
> >     > > >     >     > broker
> >     > > >     >     >         >> side.
> >     > > >     >     >         >> > >     c) At this point we will have to
> >     > specify a
> >     > > >     > warning or
> >     > > >     >     > clearly
> >     > > >     >     >         >> specify
> >     > > >     >     >         >> > > in docs that this behavior is about
> > to be
> >     > > > changed for
> >     > > >     > log
> >     > > >     >     > compaction.
> >     > > >     >     >         >> > > 4) a) In next release of the
> > Broker(x+2), we
> >     > > > say that
> >     > > >     > only
> >     > > >     >     > Tombstone
> >     > > >     >     >         >> is
> >     > > >     >     >         >> > > used for log compaction on the
> Broker
> > side.
> >     > > >     > Clients(x+1)
> >     > > >     >     > still is
> >     > > >     >     >         >> > > supported.
> >     > > >     >     >         >> > >     b) We upgrade the client to
> > version
> >     > > > client(x+2) :
> >     > > >     > if
> >     > > >     >     > value is set
> >     > > >     >     >         >> to
> >     > > >     >     >         >> > > null, tombstone will not be set
> >     > automatically.
> >     > > > The
> >     > > >     > client
> >     > > >     >     > will have to
> >     > > >     >     >         >> > call
> >     > > >     >     >         >> > > setTombstone() to actually set the
> >     > tombstone.
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > We should compare this migration
> plan
> > with
> >     > the
> >     > > >     > migration
> >     > > >     >     > plan for
> >     > > >     >     >         >> magic
> >     > > >     >     >         >> > > byte bump and do whatever looks
> good.
> >     > > >     >     >         >> > > I am just worried that if we go down
> > magic
> >     > > byte
> >     > > > route,
> >     > > >     >     > unless I am
> >     > > >     >     >         >> > missing
> >     > > >     >     >         >> > > something, it sounds like kafka will
> > be
> >     > stuck
> >     > > > with
> >     > > >     >     > supporting both
> >     > > >     >     >         >> null
> >     > > >     >     >         >> > > value and tombstone bit for log
> > compaction
> >     > for
> >     > > > life
> >     > > >     > long,
> >     > > >     >     > which does
> >     > > >     >     >         >> not
> >     > > >     >     >         >> > > look like a good end state.
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > Thanks,
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > Mayuresh
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
> > Mayuresh
> >     > > > Gharat <
> >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> >     > > >     >     >         >> > > > wrote:
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > > Hi Ismael,
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > That's a very good point which I
> > might
> >     > have
> >     > > > not
> >     > > >     >     > considered earlier.
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > Here is a plan that I can think
> of:
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > Stage 1) The broker from now on,
> up
> >     > converts
> >     > > > the
> >     > > >     > message
> >     > > >     >     > to have the
> >     > > >     >     >         >> > > > tombstone marker. The log
> compaction
> >     > thread
> >     > > > does log
> >     > > >     >     > compaction
> >     > > >     >     >         >> based
> >     > > >     >     >         >> > on
> >     > > >     >     >         >> > > > both null and tombstone marker.
> > This is
> >     > our
> >     > > >     > transition
> >     > > >     >     > period.
> >     > > >     >     >         >> > > > Stage 2) The next release we only
> > say that
> >     > > log
> >     > > >     > compaction
> >     > > >     >     > is based
> >     > > >     >     >         >> on
> >     > > >     >     >         >> > > > tombstone marker. (Open source
> > kafka makes
> >     > > > this as a
> >     > > >     >     > policy). By
> >     > > >     >     >         >> this
> >     > > >     >     >         >> > > time,
> >     > > >     >     >         >> > > > the organization which is moving
> to
> > this
> >     > > > release
> >     > > >     > will be
> >     > > >     >     > sure that
> >     > > >     >     >         >> they
> >     > > >     >     >         >> > > > have gone through the entire
> > transition
> >     > > > period.
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > My only goal of doing this is that
> > Kafka
> >     > > > clearly
> >     > > >     >     > specifies the end
> >     > > >     >     >         >> > state
> >     > > >     >     >         >> > > > about what log compaction means
> (is
> > it
> >     > null
> >     > > > value
> >     > > >     > or a
> >     > > >     >     > tombstone
> >     > > >     >     >         >> > marker,
> >     > > >     >     >         >> > > > but not both).
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > What do you think?
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > Thanks,
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > Mayuresh
> >     > > >     >     >         >> > > > .
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM,
> > Ismael
> >     > > Juma <
> >     > > >     >     > ismael@juma.me.uk>
> >     > > >     >     >         >> > wrote:
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > >> One comment below.
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM,
> > Mayuresh
> >     > > > Gharat <
> >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> >     > > >     >     >         >> > > >> > wrote:
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >> >    - If we don't bump up the
> > magic
> >     > byte,
> >     > > > on the
> >     > > >     > broker
> >     > > >     >     > side, the
> >     > > >     >     >         >> > > broker
> >     > > >     >     >         >> > > >> >    will always have to look at
> > both
> >     > > > tombstone
> >     > > >     > bit and
> >     > > >     >     > the value
> >     > > >     >     >         >> when
> >     > > >     >     >         >> > > do
> >     > > >     >     >         >> > > >> the
> >     > > >     >     >         >> > > >> >    compaction. Assuming we do
> > not bump
> >     > up
> >     > > > the
> >     > > >     > magic
> >     > > >     >     > byte,
> >     > > >     >     >         >> > > >> >    imagine the broker sees a
> > message
> >     > > which
> >     > > > does
> >     > > >     > not
> >     > > >     >     > have a
> >     > > >     >     >         >> tombstone
> >     > > >     >     >         >> > > bit
> >     > > >     >     >         >> > > >> >    set. The broker does not
> know
> > when
> >     > the
> >     > > >     > message was
> >     > > >     >     > produced
> >     > > >     >     >         >> (i.e.
> >     > > >     >     >         >> > > >> > whether
> >     > > >     >     >         >> > > >> >    the message has been up
> > converted or
> >     > > > not), it
> >     > > >     > has
> >     > > >     >     > to take a
> >     > > >     >     >         >> > further
> >     > > >     >     >         >> > > >> > look at
> >     > > >     >     >         >> > > >> >    the value to see if it is
> > null or
> >     > not
> >     > > in
> >     > > >     > order to
> >     > > >     >     > determine
> >     > > >     >     >         >> if it
> >     > > >     >     >         >> > > is
> >     > > >     >     >         >> > > >> a
> >     > > >     >     >         >> > > >> >    tombstone. The same logic
> has
> > to be
> >     > > put
> >     > > > on the
> >     > > >     >     > consumer as
> >     > > >     >     >         >> well
> >     > > >     >     >         >> > > >> because
> >     > > >     >     >         >> > > >> > the
> >     > > >     >     >         >> > > >> >    consumer does not know if
> the
> >     > message
> >     > > > has
> >     > > >     > been up
> >     > > >     >     > converted or
> >     > > >     >     >         >> > not.
> >     > > >     >     >         >> > > >> >       - If we upconvert while
> >     > appending,
> >     > > > this is
> >     > > >     > not
> >     > > >     >     > the case,
> >     > > >     >     >         >> > right?
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >> If I understand you correctly,
> > this is
> >     > not
> >     > > >     > sufficient
> >     > > >     >     > because the
> >     > > >     >     >         >> log
> >     > > >     >     >         >> > > may
> >     > > >     >     >         >> > > >> have messages appended before it
> > was
> >     > > > upgraded to
> >     > > >     > include
> >     > > >     >     > KIP-87.
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >> Ismael
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > --
> >     > > >     >     >         >> > > > -Regards,
> >     > > >     >     >         >> > > > Mayuresh R. Gharat
> >     > > >     >     >         >> > > > (862) 250-7125
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > --
> >     > > >     >     >         >> > > -Regards,
> >     > > >     >     >         >> > > Mayuresh R. Gharat
> >     > > >     >     >         >> > > (862) 250-7125
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > The information contained in this
> email
> > is
> >     > > > strictly
> >     > > >     >     > confidential and for
> >     > > >     >     >         >> > the use of the addressee only, unless
> >     > otherwise
> >     > > >     > indicated. If
> >     > > >     >     > you are
> >     > > >     >     >         >> not
> >     > > >     >     >         >> > the intended recipient, please do not
> > read,
> >     > > copy,
> >     > > > use or
> >     > > >     >     > disclose to
> >     > > >     >     >         >> others
> >     > > >     >     >         >> > this message or any attachment. Please
> > also
> >     > > > notify the
> >     > > >     > sender
> >     > > >     >     > by
> >     > > >     >     >         >> replying
> >     > > >     >     >         >> > to this email or by telephone (+44(020
> > 7896
> >     > > 0011)
> >     > > > and
> >     > > >     > then
> >     > > >     >     > delete the
> >     > > >     >     >         >> email
> >     > > >     >     >         >> > and any copies of it. Opinions,
> > conclusion
> >     > (etc)
> >     > > > that
> >     > > >     > do not
> >     > > >     >     > relate to
> >     > > >     >     >         >> the
> >     > > >     >     >         >> > official business of this company
> shall
> > be
> >     > > > understood as
> >     > > >     >     > neither given
> >     > > >     >     >         >> nor
> >     > > >     >     >         >> > endorsed by it. IG is a trading name
> of
> > IG
> >     > > Markets
> >     > > >     > Limited (a
> >     > > >     >     > company
> >     > > >     >     >         >> > registered in England and Wales,
> company
> >     > number
> >     > > >     > 04008957) and
> >     > > >     >     > IG Index
> >     > > >     >     >         >> > Limited (a company registered in
> > England and
> >     > > > Wales,
> >     > > >     > company
> >     > > >     >     > number
> >     > > >     >     >         >> > 01190902). Registered address at
> Cannon
> > Bridge
> >     > > > House, 25
> >     > > >     >     > Dowgate Hill,
> >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
> Limited
> >     > > (register
> >     > > >     > number
> >     > > >     >     > 195355) and IG
> >     > > >     >     >         >> > Index Limited (register number 114059)
> > are
> >     > > > authorised
> >     > > >     > and
> >     > > >     >     > regulated by
> >     > > >     >     >         >> the
> >     > > >     >     >         >> > Financial Conduct Authority.
> >     > > >     >     >         >> >
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >> --
> >     > > >     >     >         >> -Regards,
> >     > > >     >     >         >> Mayuresh R. Gharat
> >     > > >     >     >         >> (862) 250-7125
> >     > > >     >     >         >> The information contained in this email
> is
> >     > > strictly
> >     > > >     >     > confidential and for
> >     > > >     >     >         >> the use of the addressee only, unless
> > otherwise
> >     > > >     > indicated. If
> >     > > >     >     > you are not
> >     > > >     >     >         >> the intended recipient, please do not
> > read,
> >     > copy,
> >     > > > use or
> >     > > >     >     > disclose to others
> >     > > >     >     >         >> this message or any attachment. Please
> > also
> >     > notify
> >     > > > the
> >     > > >     > sender
> >     > > >     >     > by replying
> >     > > >     >     >         >> to this email or by telephone (+44(020
> > 7896
> >     > 0011)
> >     > > > and then
> >     > > >     >     > delete the email
> >     > > >     >     >         >> and any copies of it. Opinions,
> > conclusion (etc)
> >     > > > that do
> >     > > >     > not
> >     > > >     >     > relate to the
> >     > > >     >     >         >> official business of this company shall
> be
> >     > > > understood as
> >     > > >     >     > neither given nor
> >     > > >     >     >         >> endorsed by it. IG is a trading name of
> IG
> >     > Markets
> >     > > >     > Limited (a
> >     > > >     >     > company
> >     > > >     >     >         >> registered in England and Wales, company
> > number
> >     > > > 04008957)
> >     > > >     > and
> >     > > >     >     > IG Index
> >     > > >     >     >         >> Limited (a company registered in England
> > and
> >     > > Wales,
> >     > > >     > company
> >     > > >     >     > number
> >     > > >     >     >         >> 01190902). Registered address at Cannon
> > Bridge
> >     > > > House, 25
> >     > > >     >     > Dowgate Hill,
> >     > > >     >     >         >> London EC4R 2YA. Both IG Markets Limited
> >     > (register
> >     > > > number
> >     > > >     >     > 195355) and IG
> >     > > >     >     >         >> Index Limited (register number 114059)
> are
> >     > > > authorised and
> >     > > >     >     > regulated by the
> >     > > >     >     >         >> Financial Conduct Authority.
> >     > > >     >     >         >>
> >     > > >     >     >         >
> >     > > >     >     >         >
> >     > > >     >     >         >
> >     > > >     >     >         > --
> >     > > >     >     >         > -Regards,
> >     > > >     >     >         > Mayuresh R. Gharat
> >     > > >     >     >         > (862) 250-7125
> >     > > >     >     >         >
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >         --
> >     > > >     >     >         -Regards,
> >     > > >     >     >         Mayuresh R. Gharat
> >     > > >     >     >         (862) 250-7125
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >     The information contained in this email is
> > strictly
> >     > > > confidential
> >     > > >     > and
> >     > > >     >     > for the use of the addressee only, unless otherwise
> >     > > indicated.
> >     > > > If
> >     > > >     > you are
> >     > > >     >     > not the intended recipient, please do not read,
> > copy, use
> >     > or
> >     > > >     > disclose to
> >     > > >     >     > others this message or any attachment. Please also
> > notify
> >     > the
> >     > > > sender
> >     > > >     > by
> >     > > >     >     > replying to this email or by telephone (+44(020
> 7896
> > 0011)
> >     > > and
> >     > > > then
> >     > > >     > delete
> >     > > >     >     > the email and any copies of it. Opinions,
> conclusion
> > (etc)
> >     > > > that do
> >     > > >     > not
> >     > > >     >     > relate to the official business of this company
> > shall be
> >     > > > understood
> >     > > >     > as
> >     > > >     >     > neither given nor endorsed by it. IG is a trading
> > name of
> >     > IG
> >     > > > Markets
> >     > > >     >     > Limited (a company registered in England and Wales,
> > company
> >     > > > number
> >     > > >     >     > 04008957) and IG Index Limited (a company
> registered
> > in
> >     > > > England and
> >     > > >     > Wales,
> >     > > >     >     > company number 01190902). Registered address at
> > Cannon
> >     > Bridge
> >     > > > House,
> >     > > >     > 25
> >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > Limited
> >     > > > (register
> >     > > >     > number
> >     > > >     >     > 195355) and IG Index Limited (register number
> > 114059) are
> >     > > > authorised
> >     > > >     > and
> >     > > >     >     > regulated by the Financial Conduct Authority.
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >
> >     > > >     >
> >     > > >     > The information contained in this email is strictly
> > confidential
> >     > > and
> >     > > > for
> >     > > >     > the use of the addressee only, unless otherwise
> indicated.
> > If you
> >     > > > are not
> >     > > >     > the intended recipient, please do not read, copy, use or
> > disclose
> >     > > to
> >     > > > others
> >     > > >     > this message or any attachment. Please also notify the
> > sender by
> >     > > > replying
> >     > > >     > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete
> >     > > > the email
> >     > > >     > and any copies of it. Opinions, conclusion (etc) that do
> > not
> >     > relate
> >     > > > to the
> >     > > >     > official business of this company shall be understood as
> > neither
> >     > > > given nor
> >     > > >     > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> >     > > company
> >     > > >     > registered in England and Wales, company number 04008957)
> > and IG
> >     > > > Index
> >     > > >     > Limited (a company registered in England and Wales,
> company
> >     > number
> >     > > >     > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > > > Hill,
> >     > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> >     > > > and IG
> >     > > >     > Index Limited (register number 114059) are authorised and
> >     > regulated
> >     > > > by the
> >     > > >     > Financial Conduct Authority.
> >     > > >     >
> >     > > >
> >     > > >
> >     > > > The information contained in this email is strictly
> confidential
> > and
> >     > for
> >     > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > not
> >     > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > others
> >     > > > this message or any attachment. Please also notify the sender
> by
> >     > replying
> >     > > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the
> >     > > email
> >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > > the
> >     > > > official business of this company shall be understood as
> neither
> > given
> >     > > nor
> >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> >     > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > IG
> >     > > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > > the
> >     > > > Financial Conduct Authority.
> >     > > >
> >     > > The information contained in this email is strictly confidential
> > and for
> >     > > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > > the intended recipient, please do not read, copy, use or disclose
> > to
> >     > others
> >     > > this message or any attachment. Please also notify the sender by
> > replying
> >     > > to this email or by telephone (+44(020 7896 0011) and then delete
> > the
> >     > email
> >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > the
> >     > > official business of this company shall be understood as neither
> > given
> >     > nor
> >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > > Limited (a company registered in England and Wales, company
> number
> >     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > the
> >     > > Financial Conduct Authority.
> >     > >
> >     > >
> >     > The information contained in this email is strictly confidential
> and
> > for
> >     > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> >     > this message or any attachment. Please also notify the sender by
> > replying
> >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > the email
> >     > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to the
> >     > official business of this company shall be understood as neither
> > given nor
> >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >     > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > Limited (a company registered in England and Wales, company number
> >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > Index Limited (register number 114059) are authorised and regulated
> > by the
> >     > Financial Conduct Authority.
> >     >
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Ewen Cheslack-Postava <ew...@confluent.io>]

It seemed like this was addressed in the migration section, wasn't it? The
V2 vs V3 requests and the fact that the broker will downgrade the message
format (losing zero copy) if you issues a V2 request to a broker using V3
format handles compatibility, does it not? The existing consumers won't see
the extra metadata in the value, but they will get a null instead and treat
it as a tombstone. Certainly there is a performance impact, but it seemed
compatible.

I'm worried about this though:


   - *NOTE *: With the new version of producer client using ProduceRequest
   V3 (magic byte = 2), a non tombstone (tombstone bit not set) and null value
   should not be allowed as the older version of consumer using FetchRequest
   V2 will think of this as a tombstone when its actually not.


Unless I'm misunderstanding, this ends up breaking binary compatibility for
the Java API. It sounds like this suggests that passing a null value to the
existing ProducerRecord constructors would generate an exception since you
didn't explicitly enable the tombstone (via whatever new constructor is
provided). But that means you can't swap in a newer client jar without
recompiling and get the same behavior if your app deletes keys using the
current approach because your app will start throwing exceptions. Maybe
this is a tradeoff we're ok with, but we've tried pretty hard to avoid
breaking compatibility like this so far -- it makes picking up bug fixes in
the clients harder for users of frameworks that have to pin to earlier
default versions for compatibility.

But then later the KIP says:


   - The old constructors for ProducerRecord without this parameter will be
   kept but updated so that their default behaviour if setting null value will
   be the tombstone will be set to true to keep existing behaviour.


So maybe I am misinterpreting something.

And just a nit re: motivation section, item 6 would be clearer for a union
schema with null (or optional schemas in other formats), e.g. [null,
string], in which case losing the schema truly is losing information
(whereas null is already the only valid value for a pure null schema).

-Ewen


On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jay,
>
> Good point this detail is missing in the KIP write up. Ive added this now.
>
> Essentially simply just upgrading the clients we do not expect any client
> app code change needed.
>
> Cheers
> Mike
> ________________________________________
> From: Jay Kreps <ja...@confluent.io>
> Sent: Saturday, December 10, 2016 10:51 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Michael,
>
> The compatibility section goes through the migration path, but isn't the
> bigger compatibility issue with existing apps? There are many (probably
> thousands) of apps in production that use this feature and send null to
> mean delete. It seems like this would break compatibility with them, and
> they would have to be rewritten to right?
>
> -Jay
>
> On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jun,
> >
> > 4) On v3 we honour the tombstone. As such we expect it to be set
> correctly
> > as per the KIP.
> >
> > 4.1) why would we want to produce an error when its v3? This is the exact
> > purpose to support non-null tombstone’s
> > 4.2) again here im unclear on the question on the v3, produce request we
> > expect the tombstone flag to be set correctly.
> >
> > 4.4) compaction only occurs on compacted topics, the bit makes no
> > difference and not looked at on un-compacted (time/size based eviction).
> >
> >
> > On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
> >
> >     Hi, Michael,
> >
> >     4. Then, I think I misunderstood this point. Could you document the
> >     following points in the wiki?
> >     4.1 If producer V3 sets tombstone, but provides a non-null value,
> does
> > the
> >     send() get an error or does the producer automatically set the value
> to
> >     null?
> >     4.2 If producer V3 doesn't set tombstone, but provides a null value,
> > does
> >     the send() get an error or does the producer automatically sets the
> >     tombstone?
> >     4.3 Does the broker only expect messages that (a) have no tombstone
> and
> >     non-null value; (b) have tombstone and null value and reject the
> > messages
> >     with an error code otherwise?
> >     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on
> not?
> >
> >     Thanks,
> >
> >     Jun
> >
> >     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> >     wrote:
> >
> >     > Not at all.  This only acts on compacted topics just as what occurs
> > today
> >     >
> >     > Sent using OWA for iPhone
> >     > ________________________________________
> >     > From: Jun Rao <ju...@confluent.io>
> >     > Sent: Tuesday, December 6, 2016 6:25:28 PM
> >     > To: dev@kafka.apache.org
> >     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     >
> >     > Hi, Michael,
> >     >
> >     > 4. Hmm, does that mean the new client library can never send a null
> > message
> >     > even to a regular topic? This seems like a change of the existing
> > behavior.
> >     >
> >     > Thanks,
> >     >
> >     > Jun
> >     >
> >     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > wrote:
> >     >
> >     > > Hi Jun,
> >     > >
> >     > > Re 4) That's because we expect the tombstone value to be set
> > correctly if
> >     > > message bit is 2, as such if an older client sends in on old
> > message the
> >     > > message is upcast and the bit is set correctly. And such no
> longer
> > need
> >     > to
> >     > > check the value. Mayuresh can you confirm my thinking and
> > understanding
> >     > of
> >     > > what we've discussed?
> >     > >
> >     > > The second point I understand what you're getting at now my
> > apologies.
> >     > Yes
> >     > > this makes sense to save on touching the message, if we're the
> > only kip
> >     > > going in, in this release.
> >     > >
> >     > > Cheers
> >     > > Mike
> >     > >
> >     > > Sent using OWA for iPhone
> >     > > ________________________________________
> >     > > From: Jun Rao <ju...@confluent.io>
> >     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> >     > > To: dev@kafka.apache.org
> >     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > >
> >     > > Hi, Michael,
> >     > >
> >     > > 4. Is this updated in the wiki? The text "If the magic byte on
> > message is
> >     > > 2, the broker should use the tombstone bit for log compaction."
> > doesn't
> >     > > seem to have changed.
> >     > >
> >     > > 2. My point is that if we change the message format just for this
> > KIP, we
> >     > > should consider whether it's worth optimizing the down conversion
> > path
> >     > > (i.e., decide whether a conversion is needed by just looking at
> the
> >     > > tombstone bit in the wrapper message) since tombstone will be
> used
> >     > rarely.
> >     > > However, if the message format change here is combined with other
> > KIPs,
> >     > > then this optimization likely won't be needed. The latter
> probably
> > makes
> >     > > the code simpler. Jiangjie, Mayuresh, what do you think?
> >     > >
> >     > > Other than those, +1 from me,
> >     > >
> >     > > Thanks,
> >     > >
> >     > > Jun
> >     > >
> >     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > > wrote:
> >     > >
> >     > > > Hi Jun
> >     > > >
> >     > > > do we have your vote on this now?
> >     > > >
> >     > > > Any other concerns?
> >     > > >
> >     > > > Cheers
> >     > > > Mike
> >     > > >
> >     > > > Sent using OWA for iPhone
> >     > > > ________________________________________
> >     > > > From: Michael Pearce <Mi...@ig.com>
> >     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> >     > > > To: dev@kafka.apache.org
> >     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >     > > >
> >     > > > Hi Jun,
> >     > > >
> >     > > > Have updated. Thanks again for the feedback.
> >     > > >
> >     > > > Agree yes we should align up when it gets to that, I assume
> > you’ve
> >     > > flagged
> >     > > > the same in the other KIP?
> >     > > >
> >     > > > I think for now let’s get this KIP approved, then we can start
> > the work
> >     > > to
> >     > > > get an initial PR, then we can discuss how to align the two if
> > needed.
> >     > > >
> >     > > > Cheers,
> >     > > > Mike
> >     > > >
> >     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> >     > > >
> >     > > >     Hi, Michael,
> >     > > >
> >     > > >     For 2), this is fine. Could you update the KIP wiki to make
> > this
> >     > and
> >     > > > other
> >     > > >     points clearer? Other than that, the KIP looks good to me.
> >     > > >
> >     > > >     An orthogonal thing is that there are other KIPs such as
> > KIP-98
> >     > that
> >     > > > also
> >     > > >     intend to change the message format. If they all get
> > approved, we
> >     > > > should
> >     > > >     think about whether it's better to just bump up the magic
> > byte once
> >     > > to
> >     > > >     incorporate multiple format changes like we did in
> > KIP-31/KIP-32.
> >     > > >
> >     > > >     Thanks,
> >     > > >
> >     > > >     Jun
> >     > > >
> >     > > >
> >     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> >     > > Michael.Pearce@ig.com>
> >     > > >     wrote:
> >     > > >
> >     > > >     > Hi Jao
> >     > > >     >
> >     > > >     > Thanks for the response. Sorry for slow reply, both with
> > personal
> >     > > > sickness
> >     > > >     > and also battling some critical issues encountered since
> >     > upgrading
> >     > > to
> >     > > >     > 0.10.1.0
> >     > > >     >
> >     > > >     > 1) Thans for spotting, Document error where we branched
> > this KIP
> >     > > from
> >     > > >     > KIP-82, will get that removed.
> >     > > >     > 2) Intent is to do this just at the record message level.
> >     > > >     > 3) Thanks for spotting, Will ensure this is corrected.
> >     > > >     > 4) As per discussion thread we will support tombstone +
> > null
> >     > value,
> >     > > >     > tombstone + non null value, no tombstone + null value.
> >     > > >     > 5) I believe this was in the discussion thread, @Mayuresh
> > is this
> >     > > >     > something we’ve overlooked? I thought we would down
> > convert and
> >     > > > remove the
> >     > > >     > value so the old consumer had existing behavior, or is
> > there
> >     > > > something we
> >     > > >     > haven’t thought about?
> >     > > >     >
> >     > > >     > Cheers
> >     > > >     > Mike
> >     > > >     >
> >     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io>
> wrote:
> >     > > >     >
> >     > > >     >     Hi, Michael,
> >     > > >     >
> >     > > >     >     Thanks for the KIP. A few comments below.
> >     > > >     >
> >     > > >     >     1. The message format change contains "HeadersLength
> >     > Headers".
> >     > > > Is that
> >     > > >     >     intended?
> >     > > >     >
> >     > > >     >     2. For compressed messageset, is the tombstone bit
> > only set
> >     > at
> >     > > > the
> >     > > >     > shallow
> >     > > >     >     level? Do we always leave that bit in the wrapper
> > message
> >     > > unset?
> >     > > > An
> >     > > >     >     alternative is to set the tombstone bit in the
> wrapper
> > if at
> >     > > > least one
> >     > > >     >     inner message has the tombstone bit set. This makes
> > things a
> >     > > bit
> >     > > > more
> >     > > >     >     complicated, but we could potentially exploit that
> for
> >     > > > optimizing down
> >     > > >     >     conversion. For example, we only need to convert
> > messages
> >     > with
> >     > > > magic 2
> >     > > >     > to
> >     > > >     >     magic 1 if the wrapper's tombstone bit is set
> > (conversion is
> >     > > > always
> >     > > >     > needed
> >     > > >     >     from magic 2 to magic 0). Not sure if the
> optimization
> > is
> >     > worth
> >     > > > the
> >     > > >     >     complexity though.
> >     > > >     >
> >     > > >     >     3. The referencing of the new version of
> >     > > > ProducerRequest/FetchRequest
> >     > > >     > is
> >     > > >     >     inconsistent (v4 vs v3). Since our convention starts
> at
> >     > version
> >     > > > at 0, I
> >     > > >     >     think the new version would be 3.
> >     > > >     >
> >     > > >     >     4. "If the magic byte on message is 2, the broker
> > should use
> >     > > the
> >     > > >     > tombstone
> >     > > >     >     bit for log compaction." What about null value? My
> >     > > understanding
> >     > > > is
> >     > > >     > that
> >     > > >     >     null value will be treated the same as setting the
> > tombstone
> >     > > bit.
> >     > > >     >
> >     > > >     >     5. For the migration path, it would be useful to
> > describe the
> >     > > > down
> >     > > >     >     conversion path to consumers (i.e., brokers on
> message
> > format
> >     > > > 0.10.2
> >     > > >     > and
> >     > > >     >     consumers on older version).
> >     > > >     >
> >     > > >     >     Thanks,
> >     > > >     >
> >     > > >     >     Jun
> >     > > >     >
> >     > > >     >
> >     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> >     > > > Michael.Pearce@ig.com
> >     > > >     > >
> >     > > >     >     wrote:
> >     > > >     >
> >     > > >     >     > Hi All,
> >     > > >     >     >
> >     > > >     >     > We have been discussing in the below thread and
> final
> >     > changes
> >     > > > have
> >     > > >     > been
> >     > > >     >     > made to the KIP wiki based on these discussions.
> >     > > >     >     >
> >     > > >     >     > We would now like to put to the vote the following
> > KIP:
> >     > > >     >     > https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-
> >     > 87+-+
> >     > > >     >     > Add+Compaction+Tombstone+Flag
> >     > > >     >     >
> >     > > >     >     > This kip is for having a distinct compaction
> > attribute
> >     > > > “tombstone”
> >     > > >     > flag
> >     > > >     >     > instead of relying on null value, allowing non-null
> > value
> >     > > > delete
> >     > > >     > messages.
> >     > > >     >     >
> >     > > >     >     > Many thanks,
> >     > > >     >     > Michael
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> >     > > Michael.Pearce@ig.com>
> >     > > >     > wrote:
> >     > > >     >     >
> >     > > >     >     >     Hi Mayuresh,
> >     > > >     >     >
> >     > > >     >     >     LGTM. Ive just made one small adjustment
> > updating the
> >     > > wire
> >     > > >     > protocol to
> >     > > >     >     > show the magic byte bump.
> >     > > >     >     >
> >     > > >     >     >     Do we think we’re good to put to a vote? Is
> > there any
> >     > > > other bits
> >     > > >     >     > needing discussion?
> >     > > >     >     >
> >     > > >     >     >     Cheers
> >     > > >     >     >     Mike
> >     > > >     >     >
> >     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> >     > > >     > gharatmayuresh15@gmail.com>
> >     > > >     >     > wrote:
> >     > > >     >     >
> >     > > >     >     >         Hi Michael,
> >     > > >     >     >
> >     > > >     >     >         I have updated the migration section of the
> > KIP.
> >     > Can
> >     > > > you
> >     > > >     > please
> >     > > >     >     > take a look?
> >     > > >     >     >
> >     > > >     >     >         Thanks,
> >     > > >     >     >
> >     > > >     >     >         Mayuresh
> >     > > >     >     >
> >     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh
> > Gharat <
> >     > > >     >     > gharatmayuresh15@gmail.com
> >     > > >     >     >         > wrote:
> >     > > >     >     >
> >     > > >     >     >         > Hi Michael,
> >     > > >     >     >         >
> >     > > >     >     >         > That whilst sending tombstone and non
> null
> > value,
> >     > > the
> >     > > >     > consumer
> >     > > >     >     > can expect
> >     > > >     >     >         > only to receive the non-null message only
> > in step
> >     > > > (3) is
> >     > > >     > this
> >     > > >     >     > correct?
> >     > > >     >     >         > ---> I do agree with you here.
> >     > > >     >     >         >
> >     > > >     >     >         > Becket, Ismael : can you guys review the
> >     > migration
> >     > > > plan
> >     > > >     > listed
> >     > > >     >     > above using
> >     > > >     >     >         > magic byte?
> >     > > >     >     >         >
> >     > > >     >     >         > Thanks,
> >     > > >     >     >         >
> >     > > >     >     >         > Mayuresh
> >     > > >     >     >         >
> >     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael
> > Pearce <
> >     > > >     >     > Michael.Pearce@ig.com>
> >     > > >     >     >         > wrote:
> >     > > >     >     >         >
> >     > > >     >     >         >> Many thanks for this Mayuresh. I don't
> > have any
> >     > > >     > objections.
> >     > > >     >     >         >>
> >     > > >     >     >         >> I assume we should state:
> >     > > >     >     >         >>
> >     > > >     >     >         >> That whilst sending tombstone and non
> null
> >     > value,
> >     > > > the
> >     > > >     > consumer
> >     > > >     >     > can expect
> >     > > >     >     >         >> only to receive the non-null message
> only
> > in
> >     > step
> >     > > > (3) is
> >     > > >     > this
> >     > > >     >     > correct?
> >     > > >     >     >         >>
> >     > > >     >     >         >> Cheers
> >     > > >     >     >         >> Mike
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >> Sent using OWA for iPhone
> >     > > >     >     >         >> ______________________________
> __________
> >     > > >     >     >         >> From: Mayuresh Gharat <
> >     > gharatmayuresh15@gmail.com
> >     > > >
> >     > > >     >     >         >> Sent: Thursday, November 17, 2016
> 5:18:41
> > PM
> >     > > >     >     >         >> To: dev@kafka.apache.org
> >     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
> > Compaction
> >     > > > Tombstone
> >     > > >     > Flag
> >     > > >     >     >         >>
> >     > > >     >     >         >> Hi Ismael,
> >     > > >     >     >         >>
> >     > > >     >     >         >> Thanks for the explanation.
> >     > > >     >     >         >> Specially I like this part where in you
> >     > mentioned
> >     > > > we can
> >     > > >     > get
> >     > > >     >     > rid of the
> >     > > >     >     >         >> older null value support for log
> > compaction
> >     > later
> >     > > > on,
> >     > > >     > here :
> >     > > >     >     >         >> We can't change semantics of the message
> > format
> >     > > > without
> >     > > >     > having
> >     > > >     >     > a long
> >     > > >     >     >         >> transition period. And we can't rely
> >     > > >     >     >         >> on people reading documentation or
> acting
> > on a
> >     > > > warning for
> >     > > >     >     > something so
> >     > > >     >     >         >> fundamental. As such, my take is that we
> > need to
> >     > > > bump the
> >     > > >     > magic
> >     > > >     >     > byte. The
> >     > > >     >     >         >> good news is
> >     > > >     >     >         >> that we don't have to support all
> versions
> >     > > forever.
> >     > > > We
> >     > > >     > have
> >     > > >     >     > said that we
> >     > > >     >     >         >> will support direct upgrades for 2
> years.
> > That
> >     > > > means that
> >     > > >     >     > message format
> >     > > >     >     >         >> version n could, in theory, be removed 2
> > years
> >     > > > after the
> >     > > >     > it's
> >     > > >     >     > introduced.
> >     > > >     >     >         >>
> >     > > >     >     >         >> Just a heads up, I would like to mention
> > that
> >     > even
> >     > > > without
> >     > > >     >     > bumping magic
> >     > > >     >     >         >> byte, we will *NOT* loose zero copy as
> in
> > the
> >     > > > client(x+1)
> >     > > >     > in my
> >     > > >     >     >         >> explanation
> >     > > >     >     >         >> above will convert internally a null
> > value to
> >     > > have a
> >     > > >     > tombstone
> >     > > >     >     > bit set and
> >     > > >     >     >         >> a tombstone bit set to have a null value
> >     > > > automatically
> >     > > >     >     > internally and by
> >     > > >     >     >         >> the time we move to version (x+2), the
> > clients
> >     > > > would have
> >     > > >     >     > upgraded.
> >     > > >     >     >         >> Obviously if we support a request from
> >     > > consumer(x),
> >     > > > we
> >     > > >     > will
> >     > > >     >     > loose zero
> >     > > >     >     >         >> copy
> >     > > >     >     >         >> but that is the same case with magic
> byte.
> >     > > >     >     >         >>
> >     > > >     >     >         >> But if magic byte bump makes life easier
> > for
> >     > > > transition
> >     > > >     > for the
> >     > > >     >     > above
> >     > > >     >     >         >> reasons that you explained, I am OK with
> > it
> >     > since
> >     > > > we are
> >     > > >     > going
> >     > > >     >     > to meet the
> >     > > >     >     >         >> end goal down the road :)
> >     > > >     >     >         >>
> >     > > >     >     >         >> On a side note can we update the doc
> here
> > on
> >     > magic
> >     > > > byte
> >     > > >     > to say
> >     > > >     >     > that "*it
> >     > > >     >     >         >> should be bumped whenever the message
> > format is
> >     > > > changed
> >     > > >     > or the
> >     > > >     >     >         >> interpretation of message format (usage
> > of the
> >     > > > reserved
> >     > > >     > bits as
> >     > > >     >     > well) is
> >     > > >     >     >         >> changed*".
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >> Hi Michael,
> >     > > >     >     >         >>
> >     > > >     >     >         >> Here is the update plan that we
> discussed
> >     > offline
> >     > > >     > yesterday :
> >     > > >     >     >         >>
> >     > > >     >     >         >> Currently the magic-byte which
> > corresponds to
> >     > the
> >     > > >     >     > "message.format.version"
> >     > > >     >     >         >> is set to 1.
> >     > > >     >     >         >>
> >     > > >     >     >         >> 1) On broker it will be set to 1
> > initially.
> >     > > >     >     >         >>
> >     > > >     >     >         >> 2) When a producer client sends a
> message
> > with
> >     > > > magic-byte
> >     > > >     > = 2,
> >     > > >     >     > since the
> >     > > >     >     >         >> broker is on magic-byte = 1, we will
> down
> >     > convert
> >     > > > it,
> >     > > >     > which
> >     > > >     >     > means if the
> >     > > >     >     >         >> tombstone bit is set, the value will be
> > set to
> >     > > > null. A
> >     > > >     > consumer
> >     > > >     >     >         >> understanding magic-byte = 1, will still
> > work
> >     > with
> >     > > > this. A
> >     > > >     >     > consumer
> >     > > >     >     >         >> working
> >     > > >     >     >         >> with magic-byte =2 will also be able to
> >     > understand
> >     > > > this,
> >     > > >     > since
> >     > > >     >     > it
> >     > > >     >     >         >> understands the tombstone.
> >     > > >     >     >         >> Now there is still the question of
> > supporting a
> >     > > >     > non-tombstone
> >     > > >     >     > and null
> >     > > >     >     >         >> value from producer client with
> > magic-byte = 2.*
> >     > > (I
> >     > > > am
> >     > > >     > not sure
> >     > > >     >     > if we
> >     > > >     >     >         >> should support this. Ismael/Becket can
> > comment
> >     > > > here)*
> >     > > >     >     >         >>
> >     > > >     >     >         >> 3) When almost all the clients have
> > upgraded,
> >     > the
> >     > > >     >     > message.format.version
> >     > > >     >     >         >> on
> >     > > >     >     >         >> the broker can be changed to 2, where in
> > the
> >     > down
> >     > > >     > conversion in
> >     > > >     >     > the above
> >     > > >     >     >         >> step will not happen. If at this point
> we
> > get a
> >     > > > consumer
> >     > > >     >     > request from a
> >     > > >     >     >         >> older consumer, we might have to down
> > convert
> >     > > where
> >     > > > in we
> >     > > >     > loose
> >     > > >     >     > zero copy,
> >     > > >     >     >         >> but these cases should be rare.
> >     > > >     >     >         >>
> >     > > >     >     >         >> Becket can you review this plan and add
> > more
> >     > > > details if I
> >     > > >     > have
> >     > > >     >     >         >> missed/wronged something, before we put
> > it on
> >     > KIP.
> >     > > >     >     >         >>
> >     > > >     >     >         >> Thanks,
> >     > > >     >     >         >>
> >     > > >     >     >         >> Mayuresh
> >     > > >     >     >         >>
> >     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM,
> Michael
> >     > Pearce <
> >     > > >     >     > Michael.Pearce@ig.com>
> >     > > >     >     >         >> wrote:
> >     > > >     >     >         >>
> >     > > >     >     >         >> > Thanks guys, for discussing this
> > offline and
> >     > > > getting
> >     > > >     > some
> >     > > >     >     > consensus.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > So its clear for myself and others
> what
> > is
> >     > > > proposed now
> >     > > >     > (i
> >     > > >     >     > think i
> >     > > >     >     >         >> > understand, but want to make sure)
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Could i ask either directly update the
> > kip to
> >     > > > detail the
> >     > > >     >     > migration
> >     > > >     >     >         >> > strategy, or (re-)state your offline
> > discussed
> >     > > and
> >     > > >     > agreed
> >     > > >     >     > migration
> >     > > >     >     >         >> > strategy based on a magic byte is in
> > this
> >     > > thread.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > The main original driver for the KIP
> > was to
> >     > > > support
> >     > > >     >     > compaction where
> >     > > >     >     >         >> value
> >     > > >     >     >         >> > isn't null, based off the discussions
> on
> >     > KIP-82
> >     > > > thread.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > We should be able to support
> > non-tombstone +
> >     > > null
> >     > > > value
> >     > > >     > by the
> >     > > >     >     >         >> completion
> >     > > >     >     >         >> > of the KIP, as we noted when
> discussing
> > this
> >     > > kip,
> >     > > > having
> >     > > >     >     > logic based on
> >     > > >     >     >         >> a
> >     > > >     >     >         >> > null value isn't very clean and also
> > separates
> >     > > the
> >     > > >     > concerns.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > As discussed already though we can
> > split this
> >     > > into
> >     > > >     > KIP-87a
> >     > > >     >     > and KIP-87b
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Where we look to deliver KIP-87a on a
> >     > compacted
> >     > > > topic
> >     > > >     > (to
> >     > > >     >     > address the
> >     > > >     >     >         >> > immediate issues)
> >     > > >     >     >         >> > * tombstone + null value
> >     > > >     >     >         >> > * tombstone + non-null value
> >     > > >     >     >         >> > * non-tombstone + non-null value
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Then we can discuss once KIP-87a is
> > completed
> >     > > > options
> >     > > >     > later
> >     > > >     >     > and how we
> >     > > >     >     >         >> > support the second part KIP-87b to
> > deliver:
> >     > > >     >     >         >> > * non-tombstone + null value
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Cheers
> >     > > >     >     >         >> > Mike
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > ______________________________
> > __________
> >     > > >     >     >         >> > From: Becket Qin <
> becket.qin@gmail.com>
> >     > > >     >     >         >> > Sent: Thursday, November 17, 2016 1:43
> > AM
> >     > > >     >     >         >> > To: dev@kafka.apache.org
> >     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add
> > Compaction
> >     > > >     > Tombstone Flag
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Renu, Mayuresh and I had an offline
> >     > discussion,
> >     > > > and
> >     > > >     > following
> >     > > >     >     > is a brief
> >     > > >     >     >         >> > summary.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > 1. We agreed that not bumping up magic
> > value
> >     > may
> >     > > > result
> >     > > >     > in
> >     > > >     >     > losing zero
> >     > > >     >     >         >> copy
> >     > > >     >     >         >> > during migration.
> >     > > >     >     >         >> > 2. Given that bumping up magic value
> is
> > almost
> >     > > > free and
> >     > > >     > has
> >     > > >     >     > benefit of
> >     > > >     >     >         >> > avoiding potential performance issue.
> > It is
> >     > > > probably
> >     > > >     > worth
> >     > > >     >     > doing.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > One issue we still need to think about
> > is
> >     > > whether
> >     > > > we
> >     > > >     > want to
> >     > > >     >     > support a
> >     > > >     >     >         >> > non-tombstone message with null value.
> >     > > >     >     >         >> > Currently it is not supported by
> Kafka.
> > If we
> >     > > > allow a
> >     > > >     >     > non-tombstone null
> >     > > >     >     >         >> > value message to exist after KIP-87.
> The
> >     > problem
> >     > > > is
> >     > > >     > that such
> >     > > >     >     > message
> >     > > >     >     >         >> will
> >     > > >     >     >         >> > not be supported by the consumers
> prior
> > to
> >     > > KIP-87.
> >     > > >     > Because a
> >     > > >     >     > null value
> >     > > >     >     >         >> > will always be interpreted to a
> > tombstone.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > One option is that we keep the current
> > way,
> >     > i.e.
> >     > > > do not
> >     > > >     >     > support such
> >     > > >     >     >         >> > message. It would be good to know if
> > there is
> >     > a
> >     > > >     > concrete use
> >     > > >     >     > case for
> >     > > >     >     >         >> such
> >     > > >     >     >         >> > message. If there is not, we can
> > probably just
> >     > > not
> >     > > >     > support it.
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > Thanks,
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > JIangjie (Becket) Qin
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> > Mayuresh
> >     > > Gharat <
> >     > > >     >     >         >> > gharatmayuresh15@gmail.com
> >     > > >     >     >         >> > > wrote:
> >     > > >     >     >         >> >
> >     > > >     >     >         >> > > Hi Ismael,
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > This is something I can think of for
> >     > migration
> >     > > > plan:
> >     > > >     >     >         >> > > So the migration plan can look
> > something
> >     > like
> >     > > > this,
> >     > > >     > with up
> >     > > >     >     >         >> conversion :
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > 1) Currently lets say we have Broker
> > at
> >     > > version
> >     > > > x.
> >     > > >     >     >         >> > > 2) Currently we have clients at
> > version x.
> >     > > >     >     >         >> > > 3) a) We move the version to
> > Broker(x+1) :
> >     > > > supports
> >     > > >     > both
> >     > > >     >     > tombstone and
> >     > > >     >     >         >> > null
> >     > > >     >     >         >> > > for log compaction.
> >     > > >     >     >         >> > >     b) We upgrade the client to
> > version
> >     > > > client(x+1) :
> >     > > >     > if in
> >     > > >     >     > the
> >     > > >     >     >         >> producer
> >     > > >     >     >         >> > > client(x+1) the value is set to
> null,
> > we
> >     > will
> >     > > >     > automatically
> >     > > >     >     > set the
> >     > > >     >     >         >> > > Tombstone bit internally. If the
> > producer
> >     > > > client(x+1)
> >     > > >     > sets
> >     > > >     >     > the
> >     > > >     >     >         >> tombstone
> >     > > >     >     >         >> > > itself, well and good. For producer
> >     > client(x),
> >     > > > the
> >     > > >     > broker
> >     > > >     >     > will up
> >     > > >     >     >         >> convert
> >     > > >     >     >         >> > > to have the tombstone bit.
> > Broker(x+1) is
> >     > > > supporting
> >     > > >     > both.
> >     > > >     >     > Consumer
> >     > > >     >     >         >> > > client(x+1) will be aware of this
> and
> > should
> >     > > be
> >     > > > able
> >     > > >     > to
> >     > > >     >     > handle this.
> >     > > >     >     >         >> For
> >     > > >     >     >         >> > > consumer client(x) we will down
> > convert the
> >     > > > message
> >     > > >     > on the
> >     > > >     >     > broker
> >     > > >     >     >         >> side.
> >     > > >     >     >         >> > >     c) At this point we will have to
> >     > specify a
> >     > > >     > warning or
> >     > > >     >     > clearly
> >     > > >     >     >         >> specify
> >     > > >     >     >         >> > > in docs that this behavior is about
> > to be
> >     > > > changed for
> >     > > >     > log
> >     > > >     >     > compaction.
> >     > > >     >     >         >> > > 4) a) In next release of the
> > Broker(x+2), we
> >     > > > say that
> >     > > >     > only
> >     > > >     >     > Tombstone
> >     > > >     >     >         >> is
> >     > > >     >     >         >> > > used for log compaction on the
> Broker
> > side.
> >     > > >     > Clients(x+1)
> >     > > >     >     > still is
> >     > > >     >     >         >> > > supported.
> >     > > >     >     >         >> > >     b) We upgrade the client to
> > version
> >     > > > client(x+2) :
> >     > > >     > if
> >     > > >     >     > value is set
> >     > > >     >     >         >> to
> >     > > >     >     >         >> > > null, tombstone will not be set
> >     > automatically.
> >     > > > The
> >     > > >     > client
> >     > > >     >     > will have to
> >     > > >     >     >         >> > call
> >     > > >     >     >         >> > > setTombstone() to actually set the
> >     > tombstone.
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > We should compare this migration
> plan
> > with
> >     > the
> >     > > >     > migration
> >     > > >     >     > plan for
> >     > > >     >     >         >> magic
> >     > > >     >     >         >> > > byte bump and do whatever looks
> good.
> >     > > >     >     >         >> > > I am just worried that if we go down
> > magic
> >     > > byte
> >     > > > route,
> >     > > >     >     > unless I am
> >     > > >     >     >         >> > missing
> >     > > >     >     >         >> > > something, it sounds like kafka will
> > be
> >     > stuck
> >     > > > with
> >     > > >     >     > supporting both
> >     > > >     >     >         >> null
> >     > > >     >     >         >> > > value and tombstone bit for log
> > compaction
> >     > for
> >     > > > life
> >     > > >     > long,
> >     > > >     >     > which does
> >     > > >     >     >         >> not
> >     > > >     >     >         >> > > look like a good end state.
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > Thanks,
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > Mayuresh
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
> > Mayuresh
> >     > > > Gharat <
> >     > > >     >     >         >> > > gharatmayuresh15@gmail.com
> >     > > >     >     >         >> > > > wrote:
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > > Hi Ismael,
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > That's a very good point which I
> > might
> >     > have
> >     > > > not
> >     > > >     >     > considered earlier.
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > Here is a plan that I can think
> of:
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > Stage 1) The broker from now on,
> up
> >     > converts
> >     > > > the
> >     > > >     > message
> >     > > >     >     > to have the
> >     > > >     >     >         >> > > > tombstone marker. The log
> compaction
> >     > thread
> >     > > > does log
> >     > > >     >     > compaction
> >     > > >     >     >         >> based
> >     > > >     >     >         >> > on
> >     > > >     >     >         >> > > > both null and tombstone marker.
> > This is
> >     > our
> >     > > >     > transition
> >     > > >     >     > period.
> >     > > >     >     >         >> > > > Stage 2) The next release we only
> > say that
> >     > > log
> >     > > >     > compaction
> >     > > >     >     > is based
> >     > > >     >     >         >> on
> >     > > >     >     >         >> > > > tombstone marker. (Open source
> > kafka makes
> >     > > > this as a
> >     > > >     >     > policy). By
> >     > > >     >     >         >> this
> >     > > >     >     >         >> > > time,
> >     > > >     >     >         >> > > > the organization which is moving
> to
> > this
> >     > > > release
> >     > > >     > will be
> >     > > >     >     > sure that
> >     > > >     >     >         >> they
> >     > > >     >     >         >> > > > have gone through the entire
> > transition
> >     > > > period.
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > My only goal of doing this is that
> > Kafka
> >     > > > clearly
> >     > > >     >     > specifies the end
> >     > > >     >     >         >> > state
> >     > > >     >     >         >> > > > about what log compaction means
> (is
> > it
> >     > null
> >     > > > value
> >     > > >     > or a
> >     > > >     >     > tombstone
> >     > > >     >     >         >> > marker,
> >     > > >     >     >         >> > > > but not both).
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > What do you think?
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > Thanks,
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > Mayuresh
> >     > > >     >     >         >> > > > .
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM,
> > Ismael
> >     > > Juma <
> >     > > >     >     > ismael@juma.me.uk>
> >     > > >     >     >         >> > wrote:
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > >> One comment below.
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM,
> > Mayuresh
> >     > > > Gharat <
> >     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> >     > > >     >     >         >> > > >> > wrote:
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >> >    - If we don't bump up the
> > magic
> >     > byte,
> >     > > > on the
> >     > > >     > broker
> >     > > >     >     > side, the
> >     > > >     >     >         >> > > broker
> >     > > >     >     >         >> > > >> >    will always have to look at
> > both
> >     > > > tombstone
> >     > > >     > bit and
> >     > > >     >     > the value
> >     > > >     >     >         >> when
> >     > > >     >     >         >> > > do
> >     > > >     >     >         >> > > >> the
> >     > > >     >     >         >> > > >> >    compaction. Assuming we do
> > not bump
> >     > up
> >     > > > the
> >     > > >     > magic
> >     > > >     >     > byte,
> >     > > >     >     >         >> > > >> >    imagine the broker sees a
> > message
> >     > > which
> >     > > > does
> >     > > >     > not
> >     > > >     >     > have a
> >     > > >     >     >         >> tombstone
> >     > > >     >     >         >> > > bit
> >     > > >     >     >         >> > > >> >    set. The broker does not
> know
> > when
> >     > the
> >     > > >     > message was
> >     > > >     >     > produced
> >     > > >     >     >         >> (i.e.
> >     > > >     >     >         >> > > >> > whether
> >     > > >     >     >         >> > > >> >    the message has been up
> > converted or
> >     > > > not), it
> >     > > >     > has
> >     > > >     >     > to take a
> >     > > >     >     >         >> > further
> >     > > >     >     >         >> > > >> > look at
> >     > > >     >     >         >> > > >> >    the value to see if it is
> > null or
> >     > not
> >     > > in
> >     > > >     > order to
> >     > > >     >     > determine
> >     > > >     >     >         >> if it
> >     > > >     >     >         >> > > is
> >     > > >     >     >         >> > > >> a
> >     > > >     >     >         >> > > >> >    tombstone. The same logic
> has
> > to be
> >     > > put
> >     > > > on the
> >     > > >     >     > consumer as
> >     > > >     >     >         >> well
> >     > > >     >     >         >> > > >> because
> >     > > >     >     >         >> > > >> > the
> >     > > >     >     >         >> > > >> >    consumer does not know if
> the
> >     > message
> >     > > > has
> >     > > >     > been up
> >     > > >     >     > converted or
> >     > > >     >     >         >> > not.
> >     > > >     >     >         >> > > >> >       - If we upconvert while
> >     > appending,
> >     > > > this is
> >     > > >     > not
> >     > > >     >     > the case,
> >     > > >     >     >         >> > right?
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >> If I understand you correctly,
> > this is
> >     > not
> >     > > >     > sufficient
> >     > > >     >     > because the
> >     > > >     >     >         >> log
> >     > > >     >     >         >> > > may
> >     > > >     >     >         >> > > >> have messages appended before it
> > was
> >     > > > upgraded to
> >     > > >     > include
> >     > > >     >     > KIP-87.
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >> Ismael
> >     > > >     >     >         >> > > >>
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > > > --
> >     > > >     >     >         >> > > > -Regards,
> >     > > >     >     >         >> > > > Mayuresh R. Gharat
> >     > > >     >     >         >> > > > (862) 250-7125
> >     > > >     >     >         >> > > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > > --
> >     > > >     >     >         >> > > -Regards,
> >     > > >     >     >         >> > > Mayuresh R. Gharat
> >     > > >     >     >         >> > > (862) 250-7125
> >     > > >     >     >         >> > >
> >     > > >     >     >         >> > The information contained in this
> email
> > is
> >     > > > strictly
> >     > > >     >     > confidential and for
> >     > > >     >     >         >> > the use of the addressee only, unless
> >     > otherwise
> >     > > >     > indicated. If
> >     > > >     >     > you are
> >     > > >     >     >         >> not
> >     > > >     >     >         >> > the intended recipient, please do not
> > read,
> >     > > copy,
> >     > > > use or
> >     > > >     >     > disclose to
> >     > > >     >     >         >> others
> >     > > >     >     >         >> > this message or any attachment. Please
> > also
> >     > > > notify the
> >     > > >     > sender
> >     > > >     >     > by
> >     > > >     >     >         >> replying
> >     > > >     >     >         >> > to this email or by telephone (+44(020
> > 7896
> >     > > 0011)
> >     > > > and
> >     > > >     > then
> >     > > >     >     > delete the
> >     > > >     >     >         >> email
> >     > > >     >     >         >> > and any copies of it. Opinions,
> > conclusion
> >     > (etc)
> >     > > > that
> >     > > >     > do not
> >     > > >     >     > relate to
> >     > > >     >     >         >> the
> >     > > >     >     >         >> > official business of this company
> shall
> > be
> >     > > > understood as
> >     > > >     >     > neither given
> >     > > >     >     >         >> nor
> >     > > >     >     >         >> > endorsed by it. IG is a trading name
> of
> > IG
> >     > > Markets
> >     > > >     > Limited (a
> >     > > >     >     > company
> >     > > >     >     >         >> > registered in England and Wales,
> company
> >     > number
> >     > > >     > 04008957) and
> >     > > >     >     > IG Index
> >     > > >     >     >         >> > Limited (a company registered in
> > England and
> >     > > > Wales,
> >     > > >     > company
> >     > > >     >     > number
> >     > > >     >     >         >> > 01190902). Registered address at
> Cannon
> > Bridge
> >     > > > House, 25
> >     > > >     >     > Dowgate Hill,
> >     > > >     >     >         >> > London EC4R 2YA. Both IG Markets
> Limited
> >     > > (register
> >     > > >     > number
> >     > > >     >     > 195355) and IG
> >     > > >     >     >         >> > Index Limited (register number 114059)
> > are
> >     > > > authorised
> >     > > >     > and
> >     > > >     >     > regulated by
> >     > > >     >     >         >> the
> >     > > >     >     >         >> > Financial Conduct Authority.
> >     > > >     >     >         >> >
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >>
> >     > > >     >     >         >> --
> >     > > >     >     >         >> -Regards,
> >     > > >     >     >         >> Mayuresh R. Gharat
> >     > > >     >     >         >> (862) 250-7125
> >     > > >     >     >         >> The information contained in this email
> is
> >     > > strictly
> >     > > >     >     > confidential and for
> >     > > >     >     >         >> the use of the addressee only, unless
> > otherwise
> >     > > >     > indicated. If
> >     > > >     >     > you are not
> >     > > >     >     >         >> the intended recipient, please do not
> > read,
> >     > copy,
> >     > > > use or
> >     > > >     >     > disclose to others
> >     > > >     >     >         >> this message or any attachment. Please
> > also
> >     > notify
> >     > > > the
> >     > > >     > sender
> >     > > >     >     > by replying
> >     > > >     >     >         >> to this email or by telephone (+44(020
> > 7896
> >     > 0011)
> >     > > > and then
> >     > > >     >     > delete the email
> >     > > >     >     >         >> and any copies of it. Opinions,
> > conclusion (etc)
> >     > > > that do
> >     > > >     > not
> >     > > >     >     > relate to the
> >     > > >     >     >         >> official business of this company shall
> be
> >     > > > understood as
> >     > > >     >     > neither given nor
> >     > > >     >     >         >> endorsed by it. IG is a trading name of
> IG
> >     > Markets
> >     > > >     > Limited (a
> >     > > >     >     > company
> >     > > >     >     >         >> registered in England and Wales, company
> > number
> >     > > > 04008957)
> >     > > >     > and
> >     > > >     >     > IG Index
> >     > > >     >     >         >> Limited (a company registered in England
> > and
> >     > > Wales,
> >     > > >     > company
> >     > > >     >     > number
> >     > > >     >     >         >> 01190902). Registered address at Cannon
> > Bridge
> >     > > > House, 25
> >     > > >     >     > Dowgate Hill,
> >     > > >     >     >         >> London EC4R 2YA. Both IG Markets Limited
> >     > (register
> >     > > > number
> >     > > >     >     > 195355) and IG
> >     > > >     >     >         >> Index Limited (register number 114059)
> are
> >     > > > authorised and
> >     > > >     >     > regulated by the
> >     > > >     >     >         >> Financial Conduct Authority.
> >     > > >     >     >         >>
> >     > > >     >     >         >
> >     > > >     >     >         >
> >     > > >     >     >         >
> >     > > >     >     >         > --
> >     > > >     >     >         > -Regards,
> >     > > >     >     >         > Mayuresh R. Gharat
> >     > > >     >     >         > (862) 250-7125
> >     > > >     >     >         >
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >         --
> >     > > >     >     >         -Regards,
> >     > > >     >     >         Mayuresh R. Gharat
> >     > > >     >     >         (862) 250-7125
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >     The information contained in this email is
> > strictly
> >     > > > confidential
> >     > > >     > and
> >     > > >     >     > for the use of the addressee only, unless otherwise
> >     > > indicated.
> >     > > > If
> >     > > >     > you are
> >     > > >     >     > not the intended recipient, please do not read,
> > copy, use
> >     > or
> >     > > >     > disclose to
> >     > > >     >     > others this message or any attachment. Please also
> > notify
> >     > the
> >     > > > sender
> >     > > >     > by
> >     > > >     >     > replying to this email or by telephone (+44(020
> 7896
> > 0011)
> >     > > and
> >     > > > then
> >     > > >     > delete
> >     > > >     >     > the email and any copies of it. Opinions,
> conclusion
> > (etc)
> >     > > > that do
> >     > > >     > not
> >     > > >     >     > relate to the official business of this company
> > shall be
> >     > > > understood
> >     > > >     > as
> >     > > >     >     > neither given nor endorsed by it. IG is a trading
> > name of
> >     > IG
> >     > > > Markets
> >     > > >     >     > Limited (a company registered in England and Wales,
> > company
> >     > > > number
> >     > > >     >     > 04008957) and IG Index Limited (a company
> registered
> > in
> >     > > > England and
> >     > > >     > Wales,
> >     > > >     >     > company number 01190902). Registered address at
> > Cannon
> >     > Bridge
> >     > > > House,
> >     > > >     > 25
> >     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > Limited
> >     > > > (register
> >     > > >     > number
> >     > > >     >     > 195355) and IG Index Limited (register number
> > 114059) are
> >     > > > authorised
> >     > > >     > and
> >     > > >     >     > regulated by the Financial Conduct Authority.
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >     >
> >     > > >     >
> >     > > >     >
> >     > > >     > The information contained in this email is strictly
> > confidential
> >     > > and
> >     > > > for
> >     > > >     > the use of the addressee only, unless otherwise
> indicated.
> > If you
> >     > > > are not
> >     > > >     > the intended recipient, please do not read, copy, use or
> > disclose
> >     > > to
> >     > > > others
> >     > > >     > this message or any attachment. Please also notify the
> > sender by
> >     > > > replying
> >     > > >     > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete
> >     > > > the email
> >     > > >     > and any copies of it. Opinions, conclusion (etc) that do
> > not
> >     > relate
> >     > > > to the
> >     > > >     > official business of this company shall be understood as
> > neither
> >     > > > given nor
> >     > > >     > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> >     > > company
> >     > > >     > registered in England and Wales, company number 04008957)
> > and IG
> >     > > > Index
> >     > > >     > Limited (a company registered in England and Wales,
> company
> >     > number
> >     > > >     > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> >     > > > Hill,
> >     > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> >     > > > and IG
> >     > > >     > Index Limited (register number 114059) are authorised and
> >     > regulated
> >     > > > by the
> >     > > >     > Financial Conduct Authority.
> >     > > >     >
> >     > > >
> >     > > >
> >     > > > The information contained in this email is strictly
> confidential
> > and
> >     > for
> >     > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > not
> >     > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > others
> >     > > > this message or any attachment. Please also notify the sender
> by
> >     > replying
> >     > > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the
> >     > > email
> >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > > the
> >     > > > official business of this company shall be understood as
> neither
> > given
> >     > > nor
> >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> >     > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > IG
> >     > > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > > the
> >     > > > Financial Conduct Authority.
> >     > > >
> >     > > The information contained in this email is strictly confidential
> > and for
> >     > > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > > the intended recipient, please do not read, copy, use or disclose
> > to
> >     > others
> >     > > this message or any attachment. Please also notify the sender by
> > replying
> >     > > to this email or by telephone (+44(020 7896 0011) and then delete
> > the
> >     > email
> >     > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > the
> >     > > official business of this company shall be understood as neither
> > given
> >     > nor
> >     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > > Limited (a company registered in England and Wales, company
> number
> >     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > the
> >     > > Financial Conduct Authority.
> >     > >
> >     > >
> >     > The information contained in this email is strictly confidential
> and
> > for
> >     > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> >     > this message or any attachment. Please also notify the sender by
> > replying
> >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > the email
> >     > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to the
> >     > official business of this company shall be understood as neither
> > given nor
> >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >     > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > Limited (a company registered in England and Wales, company number
> >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > Index Limited (register number 114059) are authorised and regulated
> > by the
> >     > Financial Conduct Authority.
> >     >
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jay,

Good point this detail is missing in the KIP write up. Ive added this now.

Essentially simply just upgrading the clients we do not expect any client app code change needed.

Cheers
Mike
________________________________________
From: Jay Kreps <ja...@confluent.io>
Sent: Saturday, December 10, 2016 10:51 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Michael,

The compatibility section goes through the migration path, but isn't the
bigger compatibility issue with existing apps? There are many (probably
thousands) of apps in production that use this feature and send null to
mean delete. It seems like this would break compatibility with them, and
they would have to be rewritten to right?

-Jay

On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jun,
>
> 4) On v3 we honour the tombstone. As such we expect it to be set correctly
> as per the KIP.
>
> 4.1) why would we want to produce an error when its v3? This is the exact
> purpose to support non-null tombstone’s
> 4.2) again here im unclear on the question on the v3, produce request we
> expect the tombstone flag to be set correctly.
>
> 4.4) compaction only occurs on compacted topics, the bit makes no
> difference and not looked at on un-compacted (time/size based eviction).
>
>
> On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
>
>     Hi, Michael,
>
>     4. Then, I think I misunderstood this point. Could you document the
>     following points in the wiki?
>     4.1 If producer V3 sets tombstone, but provides a non-null value, does
> the
>     send() get an error or does the producer automatically set the value to
>     null?
>     4.2 If producer V3 doesn't set tombstone, but provides a null value,
> does
>     the send() get an error or does the producer automatically sets the
>     tombstone?
>     4.3 Does the broker only expect messages that (a) have no tombstone and
>     non-null value; (b) have tombstone and null value and reject the
> messages
>     with an error code otherwise?
>     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on not?
>
>     Thanks,
>
>     Jun
>
>     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <Michael.Pearce@ig.com
> >
>     wrote:
>
>     > Not at all.  This only acts on compacted topics just as what occurs
> today
>     >
>     > Sent using OWA for iPhone
>     > ________________________________________
>     > From: Jun Rao <ju...@confluent.io>
>     > Sent: Tuesday, December 6, 2016 6:25:28 PM
>     > To: dev@kafka.apache.org
>     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     >
>     > Hi, Michael,
>     >
>     > 4. Hmm, does that mean the new client library can never send a null
> message
>     > even to a regular topic? This seems like a change of the existing
> behavior.
>     >
>     > Thanks,
>     >
>     > Jun
>     >
>     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > wrote:
>     >
>     > > Hi Jun,
>     > >
>     > > Re 4) That's because we expect the tombstone value to be set
> correctly if
>     > > message bit is 2, as such if an older client sends in on old
> message the
>     > > message is upcast and the bit is set correctly. And such no longer
> need
>     > to
>     > > check the value. Mayuresh can you confirm my thinking and
> understanding
>     > of
>     > > what we've discussed?
>     > >
>     > > The second point I understand what you're getting at now my
> apologies.
>     > Yes
>     > > this makes sense to save on touching the message, if we're the
> only kip
>     > > going in, in this release.
>     > >
>     > > Cheers
>     > > Mike
>     > >
>     > > Sent using OWA for iPhone
>     > > ________________________________________
>     > > From: Jun Rao <ju...@confluent.io>
>     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>     > > To: dev@kafka.apache.org
>     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > >
>     > > Hi, Michael,
>     > >
>     > > 4. Is this updated in the wiki? The text "If the magic byte on
> message is
>     > > 2, the broker should use the tombstone bit for log compaction."
> doesn't
>     > > seem to have changed.
>     > >
>     > > 2. My point is that if we change the message format just for this
> KIP, we
>     > > should consider whether it's worth optimizing the down conversion
> path
>     > > (i.e., decide whether a conversion is needed by just looking at the
>     > > tombstone bit in the wrapper message) since tombstone will be used
>     > rarely.
>     > > However, if the message format change here is combined with other
> KIPs,
>     > > then this optimization likely won't be needed. The latter probably
> makes
>     > > the code simpler. Jiangjie, Mayuresh, what do you think?
>     > >
>     > > Other than those, +1 from me,
>     > >
>     > > Thanks,
>     > >
>     > > Jun
>     > >
>     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > > wrote:
>     > >
>     > > > Hi Jun
>     > > >
>     > > > do we have your vote on this now?
>     > > >
>     > > > Any other concerns?
>     > > >
>     > > > Cheers
>     > > > Mike
>     > > >
>     > > > Sent using OWA for iPhone
>     > > > ________________________________________
>     > > > From: Michael Pearce <Mi...@ig.com>
>     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>     > > > To: dev@kafka.apache.org
>     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > >
>     > > > Hi Jun,
>     > > >
>     > > > Have updated. Thanks again for the feedback.
>     > > >
>     > > > Agree yes we should align up when it gets to that, I assume
> you’ve
>     > > flagged
>     > > > the same in the other KIP?
>     > > >
>     > > > I think for now let’s get this KIP approved, then we can start
> the work
>     > > to
>     > > > get an initial PR, then we can discuss how to align the two if
> needed.
>     > > >
>     > > > Cheers,
>     > > > Mike
>     > > >
>     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
>     > > >
>     > > >     Hi, Michael,
>     > > >
>     > > >     For 2), this is fine. Could you update the KIP wiki to make
> this
>     > and
>     > > > other
>     > > >     points clearer? Other than that, the KIP looks good to me.
>     > > >
>     > > >     An orthogonal thing is that there are other KIPs such as
> KIP-98
>     > that
>     > > > also
>     > > >     intend to change the message format. If they all get
> approved, we
>     > > > should
>     > > >     think about whether it's better to just bump up the magic
> byte once
>     > > to
>     > > >     incorporate multiple format changes like we did in
> KIP-31/KIP-32.
>     > > >
>     > > >     Thanks,
>     > > >
>     > > >     Jun
>     > > >
>     > > >
>     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
>     > > Michael.Pearce@ig.com>
>     > > >     wrote:
>     > > >
>     > > >     > Hi Jao
>     > > >     >
>     > > >     > Thanks for the response. Sorry for slow reply, both with
> personal
>     > > > sickness
>     > > >     > and also battling some critical issues encountered since
>     > upgrading
>     > > to
>     > > >     > 0.10.1.0
>     > > >     >
>     > > >     > 1) Thans for spotting, Document error where we branched
> this KIP
>     > > from
>     > > >     > KIP-82, will get that removed.
>     > > >     > 2) Intent is to do this just at the record message level.
>     > > >     > 3) Thanks for spotting, Will ensure this is corrected.
>     > > >     > 4) As per discussion thread we will support tombstone +
> null
>     > value,
>     > > >     > tombstone + non null value, no tombstone + null value.
>     > > >     > 5) I believe this was in the discussion thread, @Mayuresh
> is this
>     > > >     > something we’ve overlooked? I thought we would down
> convert and
>     > > > remove the
>     > > >     > value so the old consumer had existing behavior, or is
> there
>     > > > something we
>     > > >     > haven’t thought about?
>     > > >     >
>     > > >     > Cheers
>     > > >     > Mike
>     > > >     >
>     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
>     > > >     >
>     > > >     >     Hi, Michael,
>     > > >     >
>     > > >     >     Thanks for the KIP. A few comments below.
>     > > >     >
>     > > >     >     1. The message format change contains "HeadersLength
>     > Headers".
>     > > > Is that
>     > > >     >     intended?
>     > > >     >
>     > > >     >     2. For compressed messageset, is the tombstone bit
> only set
>     > at
>     > > > the
>     > > >     > shallow
>     > > >     >     level? Do we always leave that bit in the wrapper
> message
>     > > unset?
>     > > > An
>     > > >     >     alternative is to set the tombstone bit in the wrapper
> if at
>     > > > least one
>     > > >     >     inner message has the tombstone bit set. This makes
> things a
>     > > bit
>     > > > more
>     > > >     >     complicated, but we could potentially exploit that for
>     > > > optimizing down
>     > > >     >     conversion. For example, we only need to convert
> messages
>     > with
>     > > > magic 2
>     > > >     > to
>     > > >     >     magic 1 if the wrapper's tombstone bit is set
> (conversion is
>     > > > always
>     > > >     > needed
>     > > >     >     from magic 2 to magic 0). Not sure if the optimization
> is
>     > worth
>     > > > the
>     > > >     >     complexity though.
>     > > >     >
>     > > >     >     3. The referencing of the new version of
>     > > > ProducerRequest/FetchRequest
>     > > >     > is
>     > > >     >     inconsistent (v4 vs v3). Since our convention starts at
>     > version
>     > > > at 0, I
>     > > >     >     think the new version would be 3.
>     > > >     >
>     > > >     >     4. "If the magic byte on message is 2, the broker
> should use
>     > > the
>     > > >     > tombstone
>     > > >     >     bit for log compaction." What about null value? My
>     > > understanding
>     > > > is
>     > > >     > that
>     > > >     >     null value will be treated the same as setting the
> tombstone
>     > > bit.
>     > > >     >
>     > > >     >     5. For the migration path, it would be useful to
> describe the
>     > > > down
>     > > >     >     conversion path to consumers (i.e., brokers on message
> format
>     > > > 0.10.2
>     > > >     > and
>     > > >     >     consumers on older version).
>     > > >     >
>     > > >     >     Thanks,
>     > > >     >
>     > > >     >     Jun
>     > > >     >
>     > > >     >
>     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
>     > > > Michael.Pearce@ig.com
>     > > >     > >
>     > > >     >     wrote:
>     > > >     >
>     > > >     >     > Hi All,
>     > > >     >     >
>     > > >     >     > We have been discussing in the below thread and final
>     > changes
>     > > > have
>     > > >     > been
>     > > >     >     > made to the KIP wiki based on these discussions.
>     > > >     >     >
>     > > >     >     > We would now like to put to the vote the following
> KIP:
>     > > >     >     > https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
>     > 87+-+
>     > > >     >     > Add+Compaction+Tombstone+Flag
>     > > >     >     >
>     > > >     >     > This kip is for having a distinct compaction
> attribute
>     > > > “tombstone”
>     > > >     > flag
>     > > >     >     > instead of relying on null value, allowing non-null
> value
>     > > > delete
>     > > >     > messages.
>     > > >     >     >
>     > > >     >     > Many thanks,
>     > > >     >     > Michael
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >
>     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
>     > > Michael.Pearce@ig.com>
>     > > >     > wrote:
>     > > >     >     >
>     > > >     >     >     Hi Mayuresh,
>     > > >     >     >
>     > > >     >     >     LGTM. Ive just made one small adjustment
> updating the
>     > > wire
>     > > >     > protocol to
>     > > >     >     > show the magic byte bump.
>     > > >     >     >
>     > > >     >     >     Do we think we’re good to put to a vote? Is
> there any
>     > > > other bits
>     > > >     >     > needing discussion?
>     > > >     >     >
>     > > >     >     >     Cheers
>     > > >     >     >     Mike
>     > > >     >     >
>     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
>     > > >     > gharatmayuresh15@gmail.com>
>     > > >     >     > wrote:
>     > > >     >     >
>     > > >     >     >         Hi Michael,
>     > > >     >     >
>     > > >     >     >         I have updated the migration section of the
> KIP.
>     > Can
>     > > > you
>     > > >     > please
>     > > >     >     > take a look?
>     > > >     >     >
>     > > >     >     >         Thanks,
>     > > >     >     >
>     > > >     >     >         Mayuresh
>     > > >     >     >
>     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh
> Gharat <
>     > > >     >     > gharatmayuresh15@gmail.com
>     > > >     >     >         > wrote:
>     > > >     >     >
>     > > >     >     >         > Hi Michael,
>     > > >     >     >         >
>     > > >     >     >         > That whilst sending tombstone and non null
> value,
>     > > the
>     > > >     > consumer
>     > > >     >     > can expect
>     > > >     >     >         > only to receive the non-null message only
> in step
>     > > > (3) is
>     > > >     > this
>     > > >     >     > correct?
>     > > >     >     >         > ---> I do agree with you here.
>     > > >     >     >         >
>     > > >     >     >         > Becket, Ismael : can you guys review the
>     > migration
>     > > > plan
>     > > >     > listed
>     > > >     >     > above using
>     > > >     >     >         > magic byte?
>     > > >     >     >         >
>     > > >     >     >         > Thanks,
>     > > >     >     >         >
>     > > >     >     >         > Mayuresh
>     > > >     >     >         >
>     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael
> Pearce <
>     > > >     >     > Michael.Pearce@ig.com>
>     > > >     >     >         > wrote:
>     > > >     >     >         >
>     > > >     >     >         >> Many thanks for this Mayuresh. I don't
> have any
>     > > >     > objections.
>     > > >     >     >         >>
>     > > >     >     >         >> I assume we should state:
>     > > >     >     >         >>
>     > > >     >     >         >> That whilst sending tombstone and non null
>     > value,
>     > > > the
>     > > >     > consumer
>     > > >     >     > can expect
>     > > >     >     >         >> only to receive the non-null message only
> in
>     > step
>     > > > (3) is
>     > > >     > this
>     > > >     >     > correct?
>     > > >     >     >         >>
>     > > >     >     >         >> Cheers
>     > > >     >     >         >> Mike
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >> Sent using OWA for iPhone
>     > > >     >     >         >> ________________________________________
>     > > >     >     >         >> From: Mayuresh Gharat <
>     > gharatmayuresh15@gmail.com
>     > > >
>     > > >     >     >         >> Sent: Thursday, November 17, 2016 5:18:41
> PM
>     > > >     >     >         >> To: dev@kafka.apache.org
>     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
> Compaction
>     > > > Tombstone
>     > > >     > Flag
>     > > >     >     >         >>
>     > > >     >     >         >> Hi Ismael,
>     > > >     >     >         >>
>     > > >     >     >         >> Thanks for the explanation.
>     > > >     >     >         >> Specially I like this part where in you
>     > mentioned
>     > > > we can
>     > > >     > get
>     > > >     >     > rid of the
>     > > >     >     >         >> older null value support for log
> compaction
>     > later
>     > > > on,
>     > > >     > here :
>     > > >     >     >         >> We can't change semantics of the message
> format
>     > > > without
>     > > >     > having
>     > > >     >     > a long
>     > > >     >     >         >> transition period. And we can't rely
>     > > >     >     >         >> on people reading documentation or acting
> on a
>     > > > warning for
>     > > >     >     > something so
>     > > >     >     >         >> fundamental. As such, my take is that we
> need to
>     > > > bump the
>     > > >     > magic
>     > > >     >     > byte. The
>     > > >     >     >         >> good news is
>     > > >     >     >         >> that we don't have to support all versions
>     > > forever.
>     > > > We
>     > > >     > have
>     > > >     >     > said that we
>     > > >     >     >         >> will support direct upgrades for 2 years.
> That
>     > > > means that
>     > > >     >     > message format
>     > > >     >     >         >> version n could, in theory, be removed 2
> years
>     > > > after the
>     > > >     > it's
>     > > >     >     > introduced.
>     > > >     >     >         >>
>     > > >     >     >         >> Just a heads up, I would like to mention
> that
>     > even
>     > > > without
>     > > >     >     > bumping magic
>     > > >     >     >         >> byte, we will *NOT* loose zero copy as in
> the
>     > > > client(x+1)
>     > > >     > in my
>     > > >     >     >         >> explanation
>     > > >     >     >         >> above will convert internally a null
> value to
>     > > have a
>     > > >     > tombstone
>     > > >     >     > bit set and
>     > > >     >     >         >> a tombstone bit set to have a null value
>     > > > automatically
>     > > >     >     > internally and by
>     > > >     >     >         >> the time we move to version (x+2), the
> clients
>     > > > would have
>     > > >     >     > upgraded.
>     > > >     >     >         >> Obviously if we support a request from
>     > > consumer(x),
>     > > > we
>     > > >     > will
>     > > >     >     > loose zero
>     > > >     >     >         >> copy
>     > > >     >     >         >> but that is the same case with magic byte.
>     > > >     >     >         >>
>     > > >     >     >         >> But if magic byte bump makes life easier
> for
>     > > > transition
>     > > >     > for the
>     > > >     >     > above
>     > > >     >     >         >> reasons that you explained, I am OK with
> it
>     > since
>     > > > we are
>     > > >     > going
>     > > >     >     > to meet the
>     > > >     >     >         >> end goal down the road :)
>     > > >     >     >         >>
>     > > >     >     >         >> On a side note can we update the doc here
> on
>     > magic
>     > > > byte
>     > > >     > to say
>     > > >     >     > that "*it
>     > > >     >     >         >> should be bumped whenever the message
> format is
>     > > > changed
>     > > >     > or the
>     > > >     >     >         >> interpretation of message format (usage
> of the
>     > > > reserved
>     > > >     > bits as
>     > > >     >     > well) is
>     > > >     >     >         >> changed*".
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >> Hi Michael,
>     > > >     >     >         >>
>     > > >     >     >         >> Here is the update plan that we discussed
>     > offline
>     > > >     > yesterday :
>     > > >     >     >         >>
>     > > >     >     >         >> Currently the magic-byte which
> corresponds to
>     > the
>     > > >     >     > "message.format.version"
>     > > >     >     >         >> is set to 1.
>     > > >     >     >         >>
>     > > >     >     >         >> 1) On broker it will be set to 1
> initially.
>     > > >     >     >         >>
>     > > >     >     >         >> 2) When a producer client sends a message
> with
>     > > > magic-byte
>     > > >     > = 2,
>     > > >     >     > since the
>     > > >     >     >         >> broker is on magic-byte = 1, we will down
>     > convert
>     > > > it,
>     > > >     > which
>     > > >     >     > means if the
>     > > >     >     >         >> tombstone bit is set, the value will be
> set to
>     > > > null. A
>     > > >     > consumer
>     > > >     >     >         >> understanding magic-byte = 1, will still
> work
>     > with
>     > > > this. A
>     > > >     >     > consumer
>     > > >     >     >         >> working
>     > > >     >     >         >> with magic-byte =2 will also be able to
>     > understand
>     > > > this,
>     > > >     > since
>     > > >     >     > it
>     > > >     >     >         >> understands the tombstone.
>     > > >     >     >         >> Now there is still the question of
> supporting a
>     > > >     > non-tombstone
>     > > >     >     > and null
>     > > >     >     >         >> value from producer client with
> magic-byte = 2.*
>     > > (I
>     > > > am
>     > > >     > not sure
>     > > >     >     > if we
>     > > >     >     >         >> should support this. Ismael/Becket can
> comment
>     > > > here)*
>     > > >     >     >         >>
>     > > >     >     >         >> 3) When almost all the clients have
> upgraded,
>     > the
>     > > >     >     > message.format.version
>     > > >     >     >         >> on
>     > > >     >     >         >> the broker can be changed to 2, where in
> the
>     > down
>     > > >     > conversion in
>     > > >     >     > the above
>     > > >     >     >         >> step will not happen. If at this point we
> get a
>     > > > consumer
>     > > >     >     > request from a
>     > > >     >     >         >> older consumer, we might have to down
> convert
>     > > where
>     > > > in we
>     > > >     > loose
>     > > >     >     > zero copy,
>     > > >     >     >         >> but these cases should be rare.
>     > > >     >     >         >>
>     > > >     >     >         >> Becket can you review this plan and add
> more
>     > > > details if I
>     > > >     > have
>     > > >     >     >         >> missed/wronged something, before we put
> it on
>     > KIP.
>     > > >     >     >         >>
>     > > >     >     >         >> Thanks,
>     > > >     >     >         >>
>     > > >     >     >         >> Mayuresh
>     > > >     >     >         >>
>     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael
>     > Pearce <
>     > > >     >     > Michael.Pearce@ig.com>
>     > > >     >     >         >> wrote:
>     > > >     >     >         >>
>     > > >     >     >         >> > Thanks guys, for discussing this
> offline and
>     > > > getting
>     > > >     > some
>     > > >     >     > consensus.
>     > > >     >     >         >> >
>     > > >     >     >         >> > So its clear for myself and others what
> is
>     > > > proposed now
>     > > >     > (i
>     > > >     >     > think i
>     > > >     >     >         >> > understand, but want to make sure)
>     > > >     >     >         >> >
>     > > >     >     >         >> > Could i ask either directly update the
> kip to
>     > > > detail the
>     > > >     >     > migration
>     > > >     >     >         >> > strategy, or (re-)state your offline
> discussed
>     > > and
>     > > >     > agreed
>     > > >     >     > migration
>     > > >     >     >         >> > strategy based on a magic byte is in
> this
>     > > thread.
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> > The main original driver for the KIP
> was to
>     > > > support
>     > > >     >     > compaction where
>     > > >     >     >         >> value
>     > > >     >     >         >> > isn't null, based off the discussions on
>     > KIP-82
>     > > > thread.
>     > > >     >     >         >> >
>     > > >     >     >         >> > We should be able to support
> non-tombstone +
>     > > null
>     > > > value
>     > > >     > by the
>     > > >     >     >         >> completion
>     > > >     >     >         >> > of the KIP, as we noted when discussing
> this
>     > > kip,
>     > > > having
>     > > >     >     > logic based on
>     > > >     >     >         >> a
>     > > >     >     >         >> > null value isn't very clean and also
> separates
>     > > the
>     > > >     > concerns.
>     > > >     >     >         >> >
>     > > >     >     >         >> > As discussed already though we can
> split this
>     > > into
>     > > >     > KIP-87a
>     > > >     >     > and KIP-87b
>     > > >     >     >         >> >
>     > > >     >     >         >> > Where we look to deliver KIP-87a on a
>     > compacted
>     > > > topic
>     > > >     > (to
>     > > >     >     > address the
>     > > >     >     >         >> > immediate issues)
>     > > >     >     >         >> > * tombstone + null value
>     > > >     >     >         >> > * tombstone + non-null value
>     > > >     >     >         >> > * non-tombstone + non-null value
>     > > >     >     >         >> >
>     > > >     >     >         >> > Then we can discuss once KIP-87a is
> completed
>     > > > options
>     > > >     > later
>     > > >     >     > and how we
>     > > >     >     >         >> > support the second part KIP-87b to
> deliver:
>     > > >     >     >         >> > * non-tombstone + null value
>     > > >     >     >         >> >
>     > > >     >     >         >> > Cheers
>     > > >     >     >         >> > Mike
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> > ______________________________
> __________
>     > > >     >     >         >> > From: Becket Qin <be...@gmail.com>
>     > > >     >     >         >> > Sent: Thursday, November 17, 2016 1:43
> AM
>     > > >     >     >         >> > To: dev@kafka.apache.org
>     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add
> Compaction
>     > > >     > Tombstone Flag
>     > > >     >     >         >> >
>     > > >     >     >         >> > Renu, Mayuresh and I had an offline
>     > discussion,
>     > > > and
>     > > >     > following
>     > > >     >     > is a brief
>     > > >     >     >         >> > summary.
>     > > >     >     >         >> >
>     > > >     >     >         >> > 1. We agreed that not bumping up magic
> value
>     > may
>     > > > result
>     > > >     > in
>     > > >     >     > losing zero
>     > > >     >     >         >> copy
>     > > >     >     >         >> > during migration.
>     > > >     >     >         >> > 2. Given that bumping up magic value is
> almost
>     > > > free and
>     > > >     > has
>     > > >     >     > benefit of
>     > > >     >     >         >> > avoiding potential performance issue.
> It is
>     > > > probably
>     > > >     > worth
>     > > >     >     > doing.
>     > > >     >     >         >> >
>     > > >     >     >         >> > One issue we still need to think about
> is
>     > > whether
>     > > > we
>     > > >     > want to
>     > > >     >     > support a
>     > > >     >     >         >> > non-tombstone message with null value.
>     > > >     >     >         >> > Currently it is not supported by Kafka.
> If we
>     > > > allow a
>     > > >     >     > non-tombstone null
>     > > >     >     >         >> > value message to exist after KIP-87. The
>     > problem
>     > > > is
>     > > >     > that such
>     > > >     >     > message
>     > > >     >     >         >> will
>     > > >     >     >         >> > not be supported by the consumers prior
> to
>     > > KIP-87.
>     > > >     > Because a
>     > > >     >     > null value
>     > > >     >     >         >> > will always be interpreted to a
> tombstone.
>     > > >     >     >         >> >
>     > > >     >     >         >> > One option is that we keep the current
> way,
>     > i.e.
>     > > > do not
>     > > >     >     > support such
>     > > >     >     >         >> > message. It would be good to know if
> there is
>     > a
>     > > >     > concrete use
>     > > >     >     > case for
>     > > >     >     >         >> such
>     > > >     >     >         >> > message. If there is not, we can
> probably just
>     > > not
>     > > >     > support it.
>     > > >     >     >         >> >
>     > > >     >     >         >> > Thanks,
>     > > >     >     >         >> >
>     > > >     >     >         >> > JIangjie (Becket) Qin
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> Mayuresh
>     > > Gharat <
>     > > >     >     >         >> > gharatmayuresh15@gmail.com
>     > > >     >     >         >> > > wrote:
>     > > >     >     >         >> >
>     > > >     >     >         >> > > Hi Ismael,
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > This is something I can think of for
>     > migration
>     > > > plan:
>     > > >     >     >         >> > > So the migration plan can look
> something
>     > like
>     > > > this,
>     > > >     > with up
>     > > >     >     >         >> conversion :
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > 1) Currently lets say we have Broker
> at
>     > > version
>     > > > x.
>     > > >     >     >         >> > > 2) Currently we have clients at
> version x.
>     > > >     >     >         >> > > 3) a) We move the version to
> Broker(x+1) :
>     > > > supports
>     > > >     > both
>     > > >     >     > tombstone and
>     > > >     >     >         >> > null
>     > > >     >     >         >> > > for log compaction.
>     > > >     >     >         >> > >     b) We upgrade the client to
> version
>     > > > client(x+1) :
>     > > >     > if in
>     > > >     >     > the
>     > > >     >     >         >> producer
>     > > >     >     >         >> > > client(x+1) the value is set to null,
> we
>     > will
>     > > >     > automatically
>     > > >     >     > set the
>     > > >     >     >         >> > > Tombstone bit internally. If the
> producer
>     > > > client(x+1)
>     > > >     > sets
>     > > >     >     > the
>     > > >     >     >         >> tombstone
>     > > >     >     >         >> > > itself, well and good. For producer
>     > client(x),
>     > > > the
>     > > >     > broker
>     > > >     >     > will up
>     > > >     >     >         >> convert
>     > > >     >     >         >> > > to have the tombstone bit.
> Broker(x+1) is
>     > > > supporting
>     > > >     > both.
>     > > >     >     > Consumer
>     > > >     >     >         >> > > client(x+1) will be aware of this and
> should
>     > > be
>     > > > able
>     > > >     > to
>     > > >     >     > handle this.
>     > > >     >     >         >> For
>     > > >     >     >         >> > > consumer client(x) we will down
> convert the
>     > > > message
>     > > >     > on the
>     > > >     >     > broker
>     > > >     >     >         >> side.
>     > > >     >     >         >> > >     c) At this point we will have to
>     > specify a
>     > > >     > warning or
>     > > >     >     > clearly
>     > > >     >     >         >> specify
>     > > >     >     >         >> > > in docs that this behavior is about
> to be
>     > > > changed for
>     > > >     > log
>     > > >     >     > compaction.
>     > > >     >     >         >> > > 4) a) In next release of the
> Broker(x+2), we
>     > > > say that
>     > > >     > only
>     > > >     >     > Tombstone
>     > > >     >     >         >> is
>     > > >     >     >         >> > > used for log compaction on the Broker
> side.
>     > > >     > Clients(x+1)
>     > > >     >     > still is
>     > > >     >     >         >> > > supported.
>     > > >     >     >         >> > >     b) We upgrade the client to
> version
>     > > > client(x+2) :
>     > > >     > if
>     > > >     >     > value is set
>     > > >     >     >         >> to
>     > > >     >     >         >> > > null, tombstone will not be set
>     > automatically.
>     > > > The
>     > > >     > client
>     > > >     >     > will have to
>     > > >     >     >         >> > call
>     > > >     >     >         >> > > setTombstone() to actually set the
>     > tombstone.
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > We should compare this migration plan
> with
>     > the
>     > > >     > migration
>     > > >     >     > plan for
>     > > >     >     >         >> magic
>     > > >     >     >         >> > > byte bump and do whatever looks good.
>     > > >     >     >         >> > > I am just worried that if we go down
> magic
>     > > byte
>     > > > route,
>     > > >     >     > unless I am
>     > > >     >     >         >> > missing
>     > > >     >     >         >> > > something, it sounds like kafka will
> be
>     > stuck
>     > > > with
>     > > >     >     > supporting both
>     > > >     >     >         >> null
>     > > >     >     >         >> > > value and tombstone bit for log
> compaction
>     > for
>     > > > life
>     > > >     > long,
>     > > >     >     > which does
>     > > >     >     >         >> not
>     > > >     >     >         >> > > look like a good end state.
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > Thanks,
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > Mayuresh
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
> Mayuresh
>     > > > Gharat <
>     > > >     >     >         >> > > gharatmayuresh15@gmail.com
>     > > >     >     >         >> > > > wrote:
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > > Hi Ismael,
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > That's a very good point which I
> might
>     > have
>     > > > not
>     > > >     >     > considered earlier.
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > Here is a plan that I can think of:
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > Stage 1) The broker from now on, up
>     > converts
>     > > > the
>     > > >     > message
>     > > >     >     > to have the
>     > > >     >     >         >> > > > tombstone marker. The log compaction
>     > thread
>     > > > does log
>     > > >     >     > compaction
>     > > >     >     >         >> based
>     > > >     >     >         >> > on
>     > > >     >     >         >> > > > both null and tombstone marker.
> This is
>     > our
>     > > >     > transition
>     > > >     >     > period.
>     > > >     >     >         >> > > > Stage 2) The next release we only
> say that
>     > > log
>     > > >     > compaction
>     > > >     >     > is based
>     > > >     >     >         >> on
>     > > >     >     >         >> > > > tombstone marker. (Open source
> kafka makes
>     > > > this as a
>     > > >     >     > policy). By
>     > > >     >     >         >> this
>     > > >     >     >         >> > > time,
>     > > >     >     >         >> > > > the organization which is moving to
> this
>     > > > release
>     > > >     > will be
>     > > >     >     > sure that
>     > > >     >     >         >> they
>     > > >     >     >         >> > > > have gone through the entire
> transition
>     > > > period.
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > My only goal of doing this is that
> Kafka
>     > > > clearly
>     > > >     >     > specifies the end
>     > > >     >     >         >> > state
>     > > >     >     >         >> > > > about what log compaction means (is
> it
>     > null
>     > > > value
>     > > >     > or a
>     > > >     >     > tombstone
>     > > >     >     >         >> > marker,
>     > > >     >     >         >> > > > but not both).
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > What do you think?
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > Thanks,
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > Mayuresh
>     > > >     >     >         >> > > > .
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM,
> Ismael
>     > > Juma <
>     > > >     >     > ismael@juma.me.uk>
>     > > >     >     >         >> > wrote:
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > >> One comment below.
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM,
> Mayuresh
>     > > > Gharat <
>     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
>     > > >     >     >         >> > > >> > wrote:
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >> >    - If we don't bump up the
> magic
>     > byte,
>     > > > on the
>     > > >     > broker
>     > > >     >     > side, the
>     > > >     >     >         >> > > broker
>     > > >     >     >         >> > > >> >    will always have to look at
> both
>     > > > tombstone
>     > > >     > bit and
>     > > >     >     > the value
>     > > >     >     >         >> when
>     > > >     >     >         >> > > do
>     > > >     >     >         >> > > >> the
>     > > >     >     >         >> > > >> >    compaction. Assuming we do
> not bump
>     > up
>     > > > the
>     > > >     > magic
>     > > >     >     > byte,
>     > > >     >     >         >> > > >> >    imagine the broker sees a
> message
>     > > which
>     > > > does
>     > > >     > not
>     > > >     >     > have a
>     > > >     >     >         >> tombstone
>     > > >     >     >         >> > > bit
>     > > >     >     >         >> > > >> >    set. The broker does not know
> when
>     > the
>     > > >     > message was
>     > > >     >     > produced
>     > > >     >     >         >> (i.e.
>     > > >     >     >         >> > > >> > whether
>     > > >     >     >         >> > > >> >    the message has been up
> converted or
>     > > > not), it
>     > > >     > has
>     > > >     >     > to take a
>     > > >     >     >         >> > further
>     > > >     >     >         >> > > >> > look at
>     > > >     >     >         >> > > >> >    the value to see if it is
> null or
>     > not
>     > > in
>     > > >     > order to
>     > > >     >     > determine
>     > > >     >     >         >> if it
>     > > >     >     >         >> > > is
>     > > >     >     >         >> > > >> a
>     > > >     >     >         >> > > >> >    tombstone. The same logic has
> to be
>     > > put
>     > > > on the
>     > > >     >     > consumer as
>     > > >     >     >         >> well
>     > > >     >     >         >> > > >> because
>     > > >     >     >         >> > > >> > the
>     > > >     >     >         >> > > >> >    consumer does not know if the
>     > message
>     > > > has
>     > > >     > been up
>     > > >     >     > converted or
>     > > >     >     >         >> > not.
>     > > >     >     >         >> > > >> >       - If we upconvert while
>     > appending,
>     > > > this is
>     > > >     > not
>     > > >     >     > the case,
>     > > >     >     >         >> > right?
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >> If I understand you correctly,
> this is
>     > not
>     > > >     > sufficient
>     > > >     >     > because the
>     > > >     >     >         >> log
>     > > >     >     >         >> > > may
>     > > >     >     >         >> > > >> have messages appended before it
> was
>     > > > upgraded to
>     > > >     > include
>     > > >     >     > KIP-87.
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >> Ismael
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > --
>     > > >     >     >         >> > > > -Regards,
>     > > >     >     >         >> > > > Mayuresh R. Gharat
>     > > >     >     >         >> > > > (862) 250-7125
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > --
>     > > >     >     >         >> > > -Regards,
>     > > >     >     >         >> > > Mayuresh R. Gharat
>     > > >     >     >         >> > > (862) 250-7125
>     > > >     >     >         >> > >
>     > > >     >     >         >> > The information contained in this email
> is
>     > > > strictly
>     > > >     >     > confidential and for
>     > > >     >     >         >> > the use of the addressee only, unless
>     > otherwise
>     > > >     > indicated. If
>     > > >     >     > you are
>     > > >     >     >         >> not
>     > > >     >     >         >> > the intended recipient, please do not
> read,
>     > > copy,
>     > > > use or
>     > > >     >     > disclose to
>     > > >     >     >         >> others
>     > > >     >     >         >> > this message or any attachment. Please
> also
>     > > > notify the
>     > > >     > sender
>     > > >     >     > by
>     > > >     >     >         >> replying
>     > > >     >     >         >> > to this email or by telephone (+44(020
> 7896
>     > > 0011)
>     > > > and
>     > > >     > then
>     > > >     >     > delete the
>     > > >     >     >         >> email
>     > > >     >     >         >> > and any copies of it. Opinions,
> conclusion
>     > (etc)
>     > > > that
>     > > >     > do not
>     > > >     >     > relate to
>     > > >     >     >         >> the
>     > > >     >     >         >> > official business of this company shall
> be
>     > > > understood as
>     > > >     >     > neither given
>     > > >     >     >         >> nor
>     > > >     >     >         >> > endorsed by it. IG is a trading name of
> IG
>     > > Markets
>     > > >     > Limited (a
>     > > >     >     > company
>     > > >     >     >         >> > registered in England and Wales, company
>     > number
>     > > >     > 04008957) and
>     > > >     >     > IG Index
>     > > >     >     >         >> > Limited (a company registered in
> England and
>     > > > Wales,
>     > > >     > company
>     > > >     >     > number
>     > > >     >     >         >> > 01190902). Registered address at Cannon
> Bridge
>     > > > House, 25
>     > > >     >     > Dowgate Hill,
>     > > >     >     >         >> > London EC4R 2YA. Both IG Markets Limited
>     > > (register
>     > > >     > number
>     > > >     >     > 195355) and IG
>     > > >     >     >         >> > Index Limited (register number 114059)
> are
>     > > > authorised
>     > > >     > and
>     > > >     >     > regulated by
>     > > >     >     >         >> the
>     > > >     >     >         >> > Financial Conduct Authority.
>     > > >     >     >         >> >
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >> --
>     > > >     >     >         >> -Regards,
>     > > >     >     >         >> Mayuresh R. Gharat
>     > > >     >     >         >> (862) 250-7125
>     > > >     >     >         >> The information contained in this email is
>     > > strictly
>     > > >     >     > confidential and for
>     > > >     >     >         >> the use of the addressee only, unless
> otherwise
>     > > >     > indicated. If
>     > > >     >     > you are not
>     > > >     >     >         >> the intended recipient, please do not
> read,
>     > copy,
>     > > > use or
>     > > >     >     > disclose to others
>     > > >     >     >         >> this message or any attachment. Please
> also
>     > notify
>     > > > the
>     > > >     > sender
>     > > >     >     > by replying
>     > > >     >     >         >> to this email or by telephone (+44(020
> 7896
>     > 0011)
>     > > > and then
>     > > >     >     > delete the email
>     > > >     >     >         >> and any copies of it. Opinions,
> conclusion (etc)
>     > > > that do
>     > > >     > not
>     > > >     >     > relate to the
>     > > >     >     >         >> official business of this company shall be
>     > > > understood as
>     > > >     >     > neither given nor
>     > > >     >     >         >> endorsed by it. IG is a trading name of IG
>     > Markets
>     > > >     > Limited (a
>     > > >     >     > company
>     > > >     >     >         >> registered in England and Wales, company
> number
>     > > > 04008957)
>     > > >     > and
>     > > >     >     > IG Index
>     > > >     >     >         >> Limited (a company registered in England
> and
>     > > Wales,
>     > > >     > company
>     > > >     >     > number
>     > > >     >     >         >> 01190902). Registered address at Cannon
> Bridge
>     > > > House, 25
>     > > >     >     > Dowgate Hill,
>     > > >     >     >         >> London EC4R 2YA. Both IG Markets Limited
>     > (register
>     > > > number
>     > > >     >     > 195355) and IG
>     > > >     >     >         >> Index Limited (register number 114059) are
>     > > > authorised and
>     > > >     >     > regulated by the
>     > > >     >     >         >> Financial Conduct Authority.
>     > > >     >     >         >>
>     > > >     >     >         >
>     > > >     >     >         >
>     > > >     >     >         >
>     > > >     >     >         > --
>     > > >     >     >         > -Regards,
>     > > >     >     >         > Mayuresh R. Gharat
>     > > >     >     >         > (862) 250-7125
>     > > >     >     >         >
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >         --
>     > > >     >     >         -Regards,
>     > > >     >     >         Mayuresh R. Gharat
>     > > >     >     >         (862) 250-7125
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >     The information contained in this email is
> strictly
>     > > > confidential
>     > > >     > and
>     > > >     >     > for the use of the addressee only, unless otherwise
>     > > indicated.
>     > > > If
>     > > >     > you are
>     > > >     >     > not the intended recipient, please do not read,
> copy, use
>     > or
>     > > >     > disclose to
>     > > >     >     > others this message or any attachment. Please also
> notify
>     > the
>     > > > sender
>     > > >     > by
>     > > >     >     > replying to this email or by telephone (+44(020 7896
> 0011)
>     > > and
>     > > > then
>     > > >     > delete
>     > > >     >     > the email and any copies of it. Opinions, conclusion
> (etc)
>     > > > that do
>     > > >     > not
>     > > >     >     > relate to the official business of this company
> shall be
>     > > > understood
>     > > >     > as
>     > > >     >     > neither given nor endorsed by it. IG is a trading
> name of
>     > IG
>     > > > Markets
>     > > >     >     > Limited (a company registered in England and Wales,
> company
>     > > > number
>     > > >     >     > 04008957) and IG Index Limited (a company registered
> in
>     > > > England and
>     > > >     > Wales,
>     > > >     >     > company number 01190902). Registered address at
> Cannon
>     > Bridge
>     > > > House,
>     > > >     > 25
>     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
> Limited
>     > > > (register
>     > > >     > number
>     > > >     >     > 195355) and IG Index Limited (register number
> 114059) are
>     > > > authorised
>     > > >     > and
>     > > >     >     > regulated by the Financial Conduct Authority.
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >
>     > > >     >
>     > > >     >
>     > > >     > The information contained in this email is strictly
> confidential
>     > > and
>     > > > for
>     > > >     > the use of the addressee only, unless otherwise indicated.
> If you
>     > > > are not
>     > > >     > the intended recipient, please do not read, copy, use or
> disclose
>     > > to
>     > > > others
>     > > >     > this message or any attachment. Please also notify the
> sender by
>     > > > replying
>     > > >     > to this email or by telephone (+44(020 7896 0011) and then
> delete
>     > > > the email
>     > > >     > and any copies of it. Opinions, conclusion (etc) that do
> not
>     > relate
>     > > > to the
>     > > >     > official business of this company shall be understood as
> neither
>     > > > given nor
>     > > >     > endorsed by it. IG is a trading name of IG Markets Limited
> (a
>     > > company
>     > > >     > registered in England and Wales, company number 04008957)
> and IG
>     > > > Index
>     > > >     > Limited (a company registered in England and Wales, company
>     > number
>     > > >     > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > > > Hill,
>     > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
>     > > > and IG
>     > > >     > Index Limited (register number 114059) are authorised and
>     > regulated
>     > > > by the
>     > > >     > Financial Conduct Authority.
>     > > >     >
>     > > >
>     > > >
>     > > > The information contained in this email is strictly confidential
> and
>     > for
>     > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > not
>     > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > others
>     > > > this message or any attachment. Please also notify the sender by
>     > replying
>     > > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
>     > > email
>     > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > > the
>     > > > official business of this company shall be understood as neither
> given
>     > > nor
>     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > > Limited (a company registered in England and Wales, company
> number
>     > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > IG
>     > > > Index Limited (register number 114059) are authorised and
> regulated by
>     > > the
>     > > > Financial Conduct Authority.
>     > > >
>     > > The information contained in this email is strictly confidential
> and for
>     > > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > > the intended recipient, please do not read, copy, use or disclose
> to
>     > others
>     > > this message or any attachment. Please also notify the sender by
> replying
>     > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
>     > email
>     > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > the
>     > > official business of this company shall be understood as neither
> given
>     > nor
>     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > Limited (a company registered in England and Wales, company number
>     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > > Index Limited (register number 114059) are authorised and
> regulated by
>     > the
>     > > Financial Conduct Authority.
>     > >
>     > >
>     > The information contained in this email is strictly confidential and
> for
>     > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > the intended recipient, please do not read, copy, use or disclose to
> others
>     > this message or any attachment. Please also notify the sender by
> replying
>     > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
>     > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
>     > official business of this company shall be understood as neither
> given nor
>     > endorsed by it. IG is a trading name of IG Markets Limited (a company
>     > registered in England and Wales, company number 04008957) and IG
> Index
>     > Limited (a company registered in England and Wales, company number
>     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > Index Limited (register number 114059) are authorised and regulated
> by the
>     > Financial Conduct Authority.
>     >
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jay Kreps <ja...@confluent.io>]

Michael,

The compatibility section goes through the migration path, but isn't the
bigger compatibility issue with existing apps? There are many (probably
thousands) of apps in production that use this feature and send null to
mean delete. It seems like this would break compatibility with them, and
they would have to be rewritten to right?

-Jay

On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jun,
>
> 4) On v3 we honour the tombstone. As such we expect it to be set correctly
> as per the KIP.
>
> 4.1) why would we want to produce an error when its v3? This is the exact
> purpose to support non-null tombstone’s
> 4.2) again here im unclear on the question on the v3, produce request we
> expect the tombstone flag to be set correctly.
>
> 4.4) compaction only occurs on compacted topics, the bit makes no
> difference and not looked at on un-compacted (time/size based eviction).
>
>
> On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:
>
>     Hi, Michael,
>
>     4. Then, I think I misunderstood this point. Could you document the
>     following points in the wiki?
>     4.1 If producer V3 sets tombstone, but provides a non-null value, does
> the
>     send() get an error or does the producer automatically set the value to
>     null?
>     4.2 If producer V3 doesn't set tombstone, but provides a null value,
> does
>     the send() get an error or does the producer automatically sets the
>     tombstone?
>     4.3 Does the broker only expect messages that (a) have no tombstone and
>     non-null value; (b) have tombstone and null value and reject the
> messages
>     with an error code otherwise?
>     4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on not?
>
>     Thanks,
>
>     Jun
>
>     On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <Michael.Pearce@ig.com
> >
>     wrote:
>
>     > Not at all.  This only acts on compacted topics just as what occurs
> today
>     >
>     > Sent using OWA for iPhone
>     > ________________________________________
>     > From: Jun Rao <ju...@confluent.io>
>     > Sent: Tuesday, December 6, 2016 6:25:28 PM
>     > To: dev@kafka.apache.org
>     > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     >
>     > Hi, Michael,
>     >
>     > 4. Hmm, does that mean the new client library can never send a null
> message
>     > even to a regular topic? This seems like a change of the existing
> behavior.
>     >
>     > Thanks,
>     >
>     > Jun
>     >
>     > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > wrote:
>     >
>     > > Hi Jun,
>     > >
>     > > Re 4) That's because we expect the tombstone value to be set
> correctly if
>     > > message bit is 2, as such if an older client sends in on old
> message the
>     > > message is upcast and the bit is set correctly. And such no longer
> need
>     > to
>     > > check the value. Mayuresh can you confirm my thinking and
> understanding
>     > of
>     > > what we've discussed?
>     > >
>     > > The second point I understand what you're getting at now my
> apologies.
>     > Yes
>     > > this makes sense to save on touching the message, if we're the
> only kip
>     > > going in, in this release.
>     > >
>     > > Cheers
>     > > Mike
>     > >
>     > > Sent using OWA for iPhone
>     > > ________________________________________
>     > > From: Jun Rao <ju...@confluent.io>
>     > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>     > > To: dev@kafka.apache.org
>     > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > >
>     > > Hi, Michael,
>     > >
>     > > 4. Is this updated in the wiki? The text "If the magic byte on
> message is
>     > > 2, the broker should use the tombstone bit for log compaction."
> doesn't
>     > > seem to have changed.
>     > >
>     > > 2. My point is that if we change the message format just for this
> KIP, we
>     > > should consider whether it's worth optimizing the down conversion
> path
>     > > (i.e., decide whether a conversion is needed by just looking at the
>     > > tombstone bit in the wrapper message) since tombstone will be used
>     > rarely.
>     > > However, if the message format change here is combined with other
> KIPs,
>     > > then this optimization likely won't be needed. The latter probably
> makes
>     > > the code simpler. Jiangjie, Mayuresh, what do you think?
>     > >
>     > > Other than those, +1 from me,
>     > >
>     > > Thanks,
>     > >
>     > > Jun
>     > >
>     > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>     > > wrote:
>     > >
>     > > > Hi Jun
>     > > >
>     > > > do we have your vote on this now?
>     > > >
>     > > > Any other concerns?
>     > > >
>     > > > Cheers
>     > > > Mike
>     > > >
>     > > > Sent using OWA for iPhone
>     > > > ________________________________________
>     > > > From: Michael Pearce <Mi...@ig.com>
>     > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>     > > > To: dev@kafka.apache.org
>     > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>     > > >
>     > > > Hi Jun,
>     > > >
>     > > > Have updated. Thanks again for the feedback.
>     > > >
>     > > > Agree yes we should align up when it gets to that, I assume
> you’ve
>     > > flagged
>     > > > the same in the other KIP?
>     > > >
>     > > > I think for now let’s get this KIP approved, then we can start
> the work
>     > > to
>     > > > get an initial PR, then we can discuss how to align the two if
> needed.
>     > > >
>     > > > Cheers,
>     > > > Mike
>     > > >
>     > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
>     > > >
>     > > >     Hi, Michael,
>     > > >
>     > > >     For 2), this is fine. Could you update the KIP wiki to make
> this
>     > and
>     > > > other
>     > > >     points clearer? Other than that, the KIP looks good to me.
>     > > >
>     > > >     An orthogonal thing is that there are other KIPs such as
> KIP-98
>     > that
>     > > > also
>     > > >     intend to change the message format. If they all get
> approved, we
>     > > > should
>     > > >     think about whether it's better to just bump up the magic
> byte once
>     > > to
>     > > >     incorporate multiple format changes like we did in
> KIP-31/KIP-32.
>     > > >
>     > > >     Thanks,
>     > > >
>     > > >     Jun
>     > > >
>     > > >
>     > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
>     > > Michael.Pearce@ig.com>
>     > > >     wrote:
>     > > >
>     > > >     > Hi Jao
>     > > >     >
>     > > >     > Thanks for the response. Sorry for slow reply, both with
> personal
>     > > > sickness
>     > > >     > and also battling some critical issues encountered since
>     > upgrading
>     > > to
>     > > >     > 0.10.1.0
>     > > >     >
>     > > >     > 1) Thans for spotting, Document error where we branched
> this KIP
>     > > from
>     > > >     > KIP-82, will get that removed.
>     > > >     > 2) Intent is to do this just at the record message level.
>     > > >     > 3) Thanks for spotting, Will ensure this is corrected.
>     > > >     > 4) As per discussion thread we will support tombstone +
> null
>     > value,
>     > > >     > tombstone + non null value, no tombstone + null value.
>     > > >     > 5) I believe this was in the discussion thread, @Mayuresh
> is this
>     > > >     > something we’ve overlooked? I thought we would down
> convert and
>     > > > remove the
>     > > >     > value so the old consumer had existing behavior, or is
> there
>     > > > something we
>     > > >     > haven’t thought about?
>     > > >     >
>     > > >     > Cheers
>     > > >     > Mike
>     > > >     >
>     > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
>     > > >     >
>     > > >     >     Hi, Michael,
>     > > >     >
>     > > >     >     Thanks for the KIP. A few comments below.
>     > > >     >
>     > > >     >     1. The message format change contains "HeadersLength
>     > Headers".
>     > > > Is that
>     > > >     >     intended?
>     > > >     >
>     > > >     >     2. For compressed messageset, is the tombstone bit
> only set
>     > at
>     > > > the
>     > > >     > shallow
>     > > >     >     level? Do we always leave that bit in the wrapper
> message
>     > > unset?
>     > > > An
>     > > >     >     alternative is to set the tombstone bit in the wrapper
> if at
>     > > > least one
>     > > >     >     inner message has the tombstone bit set. This makes
> things a
>     > > bit
>     > > > more
>     > > >     >     complicated, but we could potentially exploit that for
>     > > > optimizing down
>     > > >     >     conversion. For example, we only need to convert
> messages
>     > with
>     > > > magic 2
>     > > >     > to
>     > > >     >     magic 1 if the wrapper's tombstone bit is set
> (conversion is
>     > > > always
>     > > >     > needed
>     > > >     >     from magic 2 to magic 0). Not sure if the optimization
> is
>     > worth
>     > > > the
>     > > >     >     complexity though.
>     > > >     >
>     > > >     >     3. The referencing of the new version of
>     > > > ProducerRequest/FetchRequest
>     > > >     > is
>     > > >     >     inconsistent (v4 vs v3). Since our convention starts at
>     > version
>     > > > at 0, I
>     > > >     >     think the new version would be 3.
>     > > >     >
>     > > >     >     4. "If the magic byte on message is 2, the broker
> should use
>     > > the
>     > > >     > tombstone
>     > > >     >     bit for log compaction." What about null value? My
>     > > understanding
>     > > > is
>     > > >     > that
>     > > >     >     null value will be treated the same as setting the
> tombstone
>     > > bit.
>     > > >     >
>     > > >     >     5. For the migration path, it would be useful to
> describe the
>     > > > down
>     > > >     >     conversion path to consumers (i.e., brokers on message
> format
>     > > > 0.10.2
>     > > >     > and
>     > > >     >     consumers on older version).
>     > > >     >
>     > > >     >     Thanks,
>     > > >     >
>     > > >     >     Jun
>     > > >     >
>     > > >     >
>     > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
>     > > > Michael.Pearce@ig.com
>     > > >     > >
>     > > >     >     wrote:
>     > > >     >
>     > > >     >     > Hi All,
>     > > >     >     >
>     > > >     >     > We have been discussing in the below thread and final
>     > changes
>     > > > have
>     > > >     > been
>     > > >     >     > made to the KIP wiki based on these discussions.
>     > > >     >     >
>     > > >     >     > We would now like to put to the vote the following
> KIP:
>     > > >     >     > https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
>     > 87+-+
>     > > >     >     > Add+Compaction+Tombstone+Flag
>     > > >     >     >
>     > > >     >     > This kip is for having a distinct compaction
> attribute
>     > > > “tombstone”
>     > > >     > flag
>     > > >     >     > instead of relying on null value, allowing non-null
> value
>     > > > delete
>     > > >     > messages.
>     > > >     >     >
>     > > >     >     > Many thanks,
>     > > >     >     > Michael
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >
>     > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
>     > > Michael.Pearce@ig.com>
>     > > >     > wrote:
>     > > >     >     >
>     > > >     >     >     Hi Mayuresh,
>     > > >     >     >
>     > > >     >     >     LGTM. Ive just made one small adjustment
> updating the
>     > > wire
>     > > >     > protocol to
>     > > >     >     > show the magic byte bump.
>     > > >     >     >
>     > > >     >     >     Do we think we’re good to put to a vote? Is
> there any
>     > > > other bits
>     > > >     >     > needing discussion?
>     > > >     >     >
>     > > >     >     >     Cheers
>     > > >     >     >     Mike
>     > > >     >     >
>     > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
>     > > >     > gharatmayuresh15@gmail.com>
>     > > >     >     > wrote:
>     > > >     >     >
>     > > >     >     >         Hi Michael,
>     > > >     >     >
>     > > >     >     >         I have updated the migration section of the
> KIP.
>     > Can
>     > > > you
>     > > >     > please
>     > > >     >     > take a look?
>     > > >     >     >
>     > > >     >     >         Thanks,
>     > > >     >     >
>     > > >     >     >         Mayuresh
>     > > >     >     >
>     > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh
> Gharat <
>     > > >     >     > gharatmayuresh15@gmail.com
>     > > >     >     >         > wrote:
>     > > >     >     >
>     > > >     >     >         > Hi Michael,
>     > > >     >     >         >
>     > > >     >     >         > That whilst sending tombstone and non null
> value,
>     > > the
>     > > >     > consumer
>     > > >     >     > can expect
>     > > >     >     >         > only to receive the non-null message only
> in step
>     > > > (3) is
>     > > >     > this
>     > > >     >     > correct?
>     > > >     >     >         > ---> I do agree with you here.
>     > > >     >     >         >
>     > > >     >     >         > Becket, Ismael : can you guys review the
>     > migration
>     > > > plan
>     > > >     > listed
>     > > >     >     > above using
>     > > >     >     >         > magic byte?
>     > > >     >     >         >
>     > > >     >     >         > Thanks,
>     > > >     >     >         >
>     > > >     >     >         > Mayuresh
>     > > >     >     >         >
>     > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael
> Pearce <
>     > > >     >     > Michael.Pearce@ig.com>
>     > > >     >     >         > wrote:
>     > > >     >     >         >
>     > > >     >     >         >> Many thanks for this Mayuresh. I don't
> have any
>     > > >     > objections.
>     > > >     >     >         >>
>     > > >     >     >         >> I assume we should state:
>     > > >     >     >         >>
>     > > >     >     >         >> That whilst sending tombstone and non null
>     > value,
>     > > > the
>     > > >     > consumer
>     > > >     >     > can expect
>     > > >     >     >         >> only to receive the non-null message only
> in
>     > step
>     > > > (3) is
>     > > >     > this
>     > > >     >     > correct?
>     > > >     >     >         >>
>     > > >     >     >         >> Cheers
>     > > >     >     >         >> Mike
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >> Sent using OWA for iPhone
>     > > >     >     >         >> ________________________________________
>     > > >     >     >         >> From: Mayuresh Gharat <
>     > gharatmayuresh15@gmail.com
>     > > >
>     > > >     >     >         >> Sent: Thursday, November 17, 2016 5:18:41
> PM
>     > > >     >     >         >> To: dev@kafka.apache.org
>     > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add
> Compaction
>     > > > Tombstone
>     > > >     > Flag
>     > > >     >     >         >>
>     > > >     >     >         >> Hi Ismael,
>     > > >     >     >         >>
>     > > >     >     >         >> Thanks for the explanation.
>     > > >     >     >         >> Specially I like this part where in you
>     > mentioned
>     > > > we can
>     > > >     > get
>     > > >     >     > rid of the
>     > > >     >     >         >> older null value support for log
> compaction
>     > later
>     > > > on,
>     > > >     > here :
>     > > >     >     >         >> We can't change semantics of the message
> format
>     > > > without
>     > > >     > having
>     > > >     >     > a long
>     > > >     >     >         >> transition period. And we can't rely
>     > > >     >     >         >> on people reading documentation or acting
> on a
>     > > > warning for
>     > > >     >     > something so
>     > > >     >     >         >> fundamental. As such, my take is that we
> need to
>     > > > bump the
>     > > >     > magic
>     > > >     >     > byte. The
>     > > >     >     >         >> good news is
>     > > >     >     >         >> that we don't have to support all versions
>     > > forever.
>     > > > We
>     > > >     > have
>     > > >     >     > said that we
>     > > >     >     >         >> will support direct upgrades for 2 years.
> That
>     > > > means that
>     > > >     >     > message format
>     > > >     >     >         >> version n could, in theory, be removed 2
> years
>     > > > after the
>     > > >     > it's
>     > > >     >     > introduced.
>     > > >     >     >         >>
>     > > >     >     >         >> Just a heads up, I would like to mention
> that
>     > even
>     > > > without
>     > > >     >     > bumping magic
>     > > >     >     >         >> byte, we will *NOT* loose zero copy as in
> the
>     > > > client(x+1)
>     > > >     > in my
>     > > >     >     >         >> explanation
>     > > >     >     >         >> above will convert internally a null
> value to
>     > > have a
>     > > >     > tombstone
>     > > >     >     > bit set and
>     > > >     >     >         >> a tombstone bit set to have a null value
>     > > > automatically
>     > > >     >     > internally and by
>     > > >     >     >         >> the time we move to version (x+2), the
> clients
>     > > > would have
>     > > >     >     > upgraded.
>     > > >     >     >         >> Obviously if we support a request from
>     > > consumer(x),
>     > > > we
>     > > >     > will
>     > > >     >     > loose zero
>     > > >     >     >         >> copy
>     > > >     >     >         >> but that is the same case with magic byte.
>     > > >     >     >         >>
>     > > >     >     >         >> But if magic byte bump makes life easier
> for
>     > > > transition
>     > > >     > for the
>     > > >     >     > above
>     > > >     >     >         >> reasons that you explained, I am OK with
> it
>     > since
>     > > > we are
>     > > >     > going
>     > > >     >     > to meet the
>     > > >     >     >         >> end goal down the road :)
>     > > >     >     >         >>
>     > > >     >     >         >> On a side note can we update the doc here
> on
>     > magic
>     > > > byte
>     > > >     > to say
>     > > >     >     > that "*it
>     > > >     >     >         >> should be bumped whenever the message
> format is
>     > > > changed
>     > > >     > or the
>     > > >     >     >         >> interpretation of message format (usage
> of the
>     > > > reserved
>     > > >     > bits as
>     > > >     >     > well) is
>     > > >     >     >         >> changed*".
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >> Hi Michael,
>     > > >     >     >         >>
>     > > >     >     >         >> Here is the update plan that we discussed
>     > offline
>     > > >     > yesterday :
>     > > >     >     >         >>
>     > > >     >     >         >> Currently the magic-byte which
> corresponds to
>     > the
>     > > >     >     > "message.format.version"
>     > > >     >     >         >> is set to 1.
>     > > >     >     >         >>
>     > > >     >     >         >> 1) On broker it will be set to 1
> initially.
>     > > >     >     >         >>
>     > > >     >     >         >> 2) When a producer client sends a message
> with
>     > > > magic-byte
>     > > >     > = 2,
>     > > >     >     > since the
>     > > >     >     >         >> broker is on magic-byte = 1, we will down
>     > convert
>     > > > it,
>     > > >     > which
>     > > >     >     > means if the
>     > > >     >     >         >> tombstone bit is set, the value will be
> set to
>     > > > null. A
>     > > >     > consumer
>     > > >     >     >         >> understanding magic-byte = 1, will still
> work
>     > with
>     > > > this. A
>     > > >     >     > consumer
>     > > >     >     >         >> working
>     > > >     >     >         >> with magic-byte =2 will also be able to
>     > understand
>     > > > this,
>     > > >     > since
>     > > >     >     > it
>     > > >     >     >         >> understands the tombstone.
>     > > >     >     >         >> Now there is still the question of
> supporting a
>     > > >     > non-tombstone
>     > > >     >     > and null
>     > > >     >     >         >> value from producer client with
> magic-byte = 2.*
>     > > (I
>     > > > am
>     > > >     > not sure
>     > > >     >     > if we
>     > > >     >     >         >> should support this. Ismael/Becket can
> comment
>     > > > here)*
>     > > >     >     >         >>
>     > > >     >     >         >> 3) When almost all the clients have
> upgraded,
>     > the
>     > > >     >     > message.format.version
>     > > >     >     >         >> on
>     > > >     >     >         >> the broker can be changed to 2, where in
> the
>     > down
>     > > >     > conversion in
>     > > >     >     > the above
>     > > >     >     >         >> step will not happen. If at this point we
> get a
>     > > > consumer
>     > > >     >     > request from a
>     > > >     >     >         >> older consumer, we might have to down
> convert
>     > > where
>     > > > in we
>     > > >     > loose
>     > > >     >     > zero copy,
>     > > >     >     >         >> but these cases should be rare.
>     > > >     >     >         >>
>     > > >     >     >         >> Becket can you review this plan and add
> more
>     > > > details if I
>     > > >     > have
>     > > >     >     >         >> missed/wronged something, before we put
> it on
>     > KIP.
>     > > >     >     >         >>
>     > > >     >     >         >> Thanks,
>     > > >     >     >         >>
>     > > >     >     >         >> Mayuresh
>     > > >     >     >         >>
>     > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael
>     > Pearce <
>     > > >     >     > Michael.Pearce@ig.com>
>     > > >     >     >         >> wrote:
>     > > >     >     >         >>
>     > > >     >     >         >> > Thanks guys, for discussing this
> offline and
>     > > > getting
>     > > >     > some
>     > > >     >     > consensus.
>     > > >     >     >         >> >
>     > > >     >     >         >> > So its clear for myself and others what
> is
>     > > > proposed now
>     > > >     > (i
>     > > >     >     > think i
>     > > >     >     >         >> > understand, but want to make sure)
>     > > >     >     >         >> >
>     > > >     >     >         >> > Could i ask either directly update the
> kip to
>     > > > detail the
>     > > >     >     > migration
>     > > >     >     >         >> > strategy, or (re-)state your offline
> discussed
>     > > and
>     > > >     > agreed
>     > > >     >     > migration
>     > > >     >     >         >> > strategy based on a magic byte is in
> this
>     > > thread.
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> > The main original driver for the KIP
> was to
>     > > > support
>     > > >     >     > compaction where
>     > > >     >     >         >> value
>     > > >     >     >         >> > isn't null, based off the discussions on
>     > KIP-82
>     > > > thread.
>     > > >     >     >         >> >
>     > > >     >     >         >> > We should be able to support
> non-tombstone +
>     > > null
>     > > > value
>     > > >     > by the
>     > > >     >     >         >> completion
>     > > >     >     >         >> > of the KIP, as we noted when discussing
> this
>     > > kip,
>     > > > having
>     > > >     >     > logic based on
>     > > >     >     >         >> a
>     > > >     >     >         >> > null value isn't very clean and also
> separates
>     > > the
>     > > >     > concerns.
>     > > >     >     >         >> >
>     > > >     >     >         >> > As discussed already though we can
> split this
>     > > into
>     > > >     > KIP-87a
>     > > >     >     > and KIP-87b
>     > > >     >     >         >> >
>     > > >     >     >         >> > Where we look to deliver KIP-87a on a
>     > compacted
>     > > > topic
>     > > >     > (to
>     > > >     >     > address the
>     > > >     >     >         >> > immediate issues)
>     > > >     >     >         >> > * tombstone + null value
>     > > >     >     >         >> > * tombstone + non-null value
>     > > >     >     >         >> > * non-tombstone + non-null value
>     > > >     >     >         >> >
>     > > >     >     >         >> > Then we can discuss once KIP-87a is
> completed
>     > > > options
>     > > >     > later
>     > > >     >     > and how we
>     > > >     >     >         >> > support the second part KIP-87b to
> deliver:
>     > > >     >     >         >> > * non-tombstone + null value
>     > > >     >     >         >> >
>     > > >     >     >         >> > Cheers
>     > > >     >     >         >> > Mike
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> > ______________________________
> __________
>     > > >     >     >         >> > From: Becket Qin <be...@gmail.com>
>     > > >     >     >         >> > Sent: Thursday, November 17, 2016 1:43
> AM
>     > > >     >     >         >> > To: dev@kafka.apache.org
>     > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add
> Compaction
>     > > >     > Tombstone Flag
>     > > >     >     >         >> >
>     > > >     >     >         >> > Renu, Mayuresh and I had an offline
>     > discussion,
>     > > > and
>     > > >     > following
>     > > >     >     > is a brief
>     > > >     >     >         >> > summary.
>     > > >     >     >         >> >
>     > > >     >     >         >> > 1. We agreed that not bumping up magic
> value
>     > may
>     > > > result
>     > > >     > in
>     > > >     >     > losing zero
>     > > >     >     >         >> copy
>     > > >     >     >         >> > during migration.
>     > > >     >     >         >> > 2. Given that bumping up magic value is
> almost
>     > > > free and
>     > > >     > has
>     > > >     >     > benefit of
>     > > >     >     >         >> > avoiding potential performance issue.
> It is
>     > > > probably
>     > > >     > worth
>     > > >     >     > doing.
>     > > >     >     >         >> >
>     > > >     >     >         >> > One issue we still need to think about
> is
>     > > whether
>     > > > we
>     > > >     > want to
>     > > >     >     > support a
>     > > >     >     >         >> > non-tombstone message with null value.
>     > > >     >     >         >> > Currently it is not supported by Kafka.
> If we
>     > > > allow a
>     > > >     >     > non-tombstone null
>     > > >     >     >         >> > value message to exist after KIP-87. The
>     > problem
>     > > > is
>     > > >     > that such
>     > > >     >     > message
>     > > >     >     >         >> will
>     > > >     >     >         >> > not be supported by the consumers prior
> to
>     > > KIP-87.
>     > > >     > Because a
>     > > >     >     > null value
>     > > >     >     >         >> > will always be interpreted to a
> tombstone.
>     > > >     >     >         >> >
>     > > >     >     >         >> > One option is that we keep the current
> way,
>     > i.e.
>     > > > do not
>     > > >     >     > support such
>     > > >     >     >         >> > message. It would be good to know if
> there is
>     > a
>     > > >     > concrete use
>     > > >     >     > case for
>     > > >     >     >         >> such
>     > > >     >     >         >> > message. If there is not, we can
> probably just
>     > > not
>     > > >     > support it.
>     > > >     >     >         >> >
>     > > >     >     >         >> > Thanks,
>     > > >     >     >         >> >
>     > > >     >     >         >> > JIangjie (Becket) Qin
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> >
>     > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM,
> Mayuresh
>     > > Gharat <
>     > > >     >     >         >> > gharatmayuresh15@gmail.com
>     > > >     >     >         >> > > wrote:
>     > > >     >     >         >> >
>     > > >     >     >         >> > > Hi Ismael,
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > This is something I can think of for
>     > migration
>     > > > plan:
>     > > >     >     >         >> > > So the migration plan can look
> something
>     > like
>     > > > this,
>     > > >     > with up
>     > > >     >     >         >> conversion :
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > 1) Currently lets say we have Broker
> at
>     > > version
>     > > > x.
>     > > >     >     >         >> > > 2) Currently we have clients at
> version x.
>     > > >     >     >         >> > > 3) a) We move the version to
> Broker(x+1) :
>     > > > supports
>     > > >     > both
>     > > >     >     > tombstone and
>     > > >     >     >         >> > null
>     > > >     >     >         >> > > for log compaction.
>     > > >     >     >         >> > >     b) We upgrade the client to
> version
>     > > > client(x+1) :
>     > > >     > if in
>     > > >     >     > the
>     > > >     >     >         >> producer
>     > > >     >     >         >> > > client(x+1) the value is set to null,
> we
>     > will
>     > > >     > automatically
>     > > >     >     > set the
>     > > >     >     >         >> > > Tombstone bit internally. If the
> producer
>     > > > client(x+1)
>     > > >     > sets
>     > > >     >     > the
>     > > >     >     >         >> tombstone
>     > > >     >     >         >> > > itself, well and good. For producer
>     > client(x),
>     > > > the
>     > > >     > broker
>     > > >     >     > will up
>     > > >     >     >         >> convert
>     > > >     >     >         >> > > to have the tombstone bit.
> Broker(x+1) is
>     > > > supporting
>     > > >     > both.
>     > > >     >     > Consumer
>     > > >     >     >         >> > > client(x+1) will be aware of this and
> should
>     > > be
>     > > > able
>     > > >     > to
>     > > >     >     > handle this.
>     > > >     >     >         >> For
>     > > >     >     >         >> > > consumer client(x) we will down
> convert the
>     > > > message
>     > > >     > on the
>     > > >     >     > broker
>     > > >     >     >         >> side.
>     > > >     >     >         >> > >     c) At this point we will have to
>     > specify a
>     > > >     > warning or
>     > > >     >     > clearly
>     > > >     >     >         >> specify
>     > > >     >     >         >> > > in docs that this behavior is about
> to be
>     > > > changed for
>     > > >     > log
>     > > >     >     > compaction.
>     > > >     >     >         >> > > 4) a) In next release of the
> Broker(x+2), we
>     > > > say that
>     > > >     > only
>     > > >     >     > Tombstone
>     > > >     >     >         >> is
>     > > >     >     >         >> > > used for log compaction on the Broker
> side.
>     > > >     > Clients(x+1)
>     > > >     >     > still is
>     > > >     >     >         >> > > supported.
>     > > >     >     >         >> > >     b) We upgrade the client to
> version
>     > > > client(x+2) :
>     > > >     > if
>     > > >     >     > value is set
>     > > >     >     >         >> to
>     > > >     >     >         >> > > null, tombstone will not be set
>     > automatically.
>     > > > The
>     > > >     > client
>     > > >     >     > will have to
>     > > >     >     >         >> > call
>     > > >     >     >         >> > > setTombstone() to actually set the
>     > tombstone.
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > We should compare this migration plan
> with
>     > the
>     > > >     > migration
>     > > >     >     > plan for
>     > > >     >     >         >> magic
>     > > >     >     >         >> > > byte bump and do whatever looks good.
>     > > >     >     >         >> > > I am just worried that if we go down
> magic
>     > > byte
>     > > > route,
>     > > >     >     > unless I am
>     > > >     >     >         >> > missing
>     > > >     >     >         >> > > something, it sounds like kafka will
> be
>     > stuck
>     > > > with
>     > > >     >     > supporting both
>     > > >     >     >         >> null
>     > > >     >     >         >> > > value and tombstone bit for log
> compaction
>     > for
>     > > > life
>     > > >     > long,
>     > > >     >     > which does
>     > > >     >     >         >> not
>     > > >     >     >         >> > > look like a good end state.
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > Thanks,
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > Mayuresh
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM,
> Mayuresh
>     > > > Gharat <
>     > > >     >     >         >> > > gharatmayuresh15@gmail.com
>     > > >     >     >         >> > > > wrote:
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > > Hi Ismael,
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > That's a very good point which I
> might
>     > have
>     > > > not
>     > > >     >     > considered earlier.
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > Here is a plan that I can think of:
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > Stage 1) The broker from now on, up
>     > converts
>     > > > the
>     > > >     > message
>     > > >     >     > to have the
>     > > >     >     >         >> > > > tombstone marker. The log compaction
>     > thread
>     > > > does log
>     > > >     >     > compaction
>     > > >     >     >         >> based
>     > > >     >     >         >> > on
>     > > >     >     >         >> > > > both null and tombstone marker.
> This is
>     > our
>     > > >     > transition
>     > > >     >     > period.
>     > > >     >     >         >> > > > Stage 2) The next release we only
> say that
>     > > log
>     > > >     > compaction
>     > > >     >     > is based
>     > > >     >     >         >> on
>     > > >     >     >         >> > > > tombstone marker. (Open source
> kafka makes
>     > > > this as a
>     > > >     >     > policy). By
>     > > >     >     >         >> this
>     > > >     >     >         >> > > time,
>     > > >     >     >         >> > > > the organization which is moving to
> this
>     > > > release
>     > > >     > will be
>     > > >     >     > sure that
>     > > >     >     >         >> they
>     > > >     >     >         >> > > > have gone through the entire
> transition
>     > > > period.
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > My only goal of doing this is that
> Kafka
>     > > > clearly
>     > > >     >     > specifies the end
>     > > >     >     >         >> > state
>     > > >     >     >         >> > > > about what log compaction means (is
> it
>     > null
>     > > > value
>     > > >     > or a
>     > > >     >     > tombstone
>     > > >     >     >         >> > marker,
>     > > >     >     >         >> > > > but not both).
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > What do you think?
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > Thanks,
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > Mayuresh
>     > > >     >     >         >> > > > .
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM,
> Ismael
>     > > Juma <
>     > > >     >     > ismael@juma.me.uk>
>     > > >     >     >         >> > wrote:
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > >> One comment below.
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM,
> Mayuresh
>     > > > Gharat <
>     > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
>     > > >     >     >         >> > > >> > wrote:
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >> >    - If we don't bump up the
> magic
>     > byte,
>     > > > on the
>     > > >     > broker
>     > > >     >     > side, the
>     > > >     >     >         >> > > broker
>     > > >     >     >         >> > > >> >    will always have to look at
> both
>     > > > tombstone
>     > > >     > bit and
>     > > >     >     > the value
>     > > >     >     >         >> when
>     > > >     >     >         >> > > do
>     > > >     >     >         >> > > >> the
>     > > >     >     >         >> > > >> >    compaction. Assuming we do
> not bump
>     > up
>     > > > the
>     > > >     > magic
>     > > >     >     > byte,
>     > > >     >     >         >> > > >> >    imagine the broker sees a
> message
>     > > which
>     > > > does
>     > > >     > not
>     > > >     >     > have a
>     > > >     >     >         >> tombstone
>     > > >     >     >         >> > > bit
>     > > >     >     >         >> > > >> >    set. The broker does not know
> when
>     > the
>     > > >     > message was
>     > > >     >     > produced
>     > > >     >     >         >> (i.e.
>     > > >     >     >         >> > > >> > whether
>     > > >     >     >         >> > > >> >    the message has been up
> converted or
>     > > > not), it
>     > > >     > has
>     > > >     >     > to take a
>     > > >     >     >         >> > further
>     > > >     >     >         >> > > >> > look at
>     > > >     >     >         >> > > >> >    the value to see if it is
> null or
>     > not
>     > > in
>     > > >     > order to
>     > > >     >     > determine
>     > > >     >     >         >> if it
>     > > >     >     >         >> > > is
>     > > >     >     >         >> > > >> a
>     > > >     >     >         >> > > >> >    tombstone. The same logic has
> to be
>     > > put
>     > > > on the
>     > > >     >     > consumer as
>     > > >     >     >         >> well
>     > > >     >     >         >> > > >> because
>     > > >     >     >         >> > > >> > the
>     > > >     >     >         >> > > >> >    consumer does not know if the
>     > message
>     > > > has
>     > > >     > been up
>     > > >     >     > converted or
>     > > >     >     >         >> > not.
>     > > >     >     >         >> > > >> >       - If we upconvert while
>     > appending,
>     > > > this is
>     > > >     > not
>     > > >     >     > the case,
>     > > >     >     >         >> > right?
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >> If I understand you correctly,
> this is
>     > not
>     > > >     > sufficient
>     > > >     >     > because the
>     > > >     >     >         >> log
>     > > >     >     >         >> > > may
>     > > >     >     >         >> > > >> have messages appended before it
> was
>     > > > upgraded to
>     > > >     > include
>     > > >     >     > KIP-87.
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >> Ismael
>     > > >     >     >         >> > > >>
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > > > --
>     > > >     >     >         >> > > > -Regards,
>     > > >     >     >         >> > > > Mayuresh R. Gharat
>     > > >     >     >         >> > > > (862) 250-7125
>     > > >     >     >         >> > > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > >
>     > > >     >     >         >> > > --
>     > > >     >     >         >> > > -Regards,
>     > > >     >     >         >> > > Mayuresh R. Gharat
>     > > >     >     >         >> > > (862) 250-7125
>     > > >     >     >         >> > >
>     > > >     >     >         >> > The information contained in this email
> is
>     > > > strictly
>     > > >     >     > confidential and for
>     > > >     >     >         >> > the use of the addressee only, unless
>     > otherwise
>     > > >     > indicated. If
>     > > >     >     > you are
>     > > >     >     >         >> not
>     > > >     >     >         >> > the intended recipient, please do not
> read,
>     > > copy,
>     > > > use or
>     > > >     >     > disclose to
>     > > >     >     >         >> others
>     > > >     >     >         >> > this message or any attachment. Please
> also
>     > > > notify the
>     > > >     > sender
>     > > >     >     > by
>     > > >     >     >         >> replying
>     > > >     >     >         >> > to this email or by telephone (+44(020
> 7896
>     > > 0011)
>     > > > and
>     > > >     > then
>     > > >     >     > delete the
>     > > >     >     >         >> email
>     > > >     >     >         >> > and any copies of it. Opinions,
> conclusion
>     > (etc)
>     > > > that
>     > > >     > do not
>     > > >     >     > relate to
>     > > >     >     >         >> the
>     > > >     >     >         >> > official business of this company shall
> be
>     > > > understood as
>     > > >     >     > neither given
>     > > >     >     >         >> nor
>     > > >     >     >         >> > endorsed by it. IG is a trading name of
> IG
>     > > Markets
>     > > >     > Limited (a
>     > > >     >     > company
>     > > >     >     >         >> > registered in England and Wales, company
>     > number
>     > > >     > 04008957) and
>     > > >     >     > IG Index
>     > > >     >     >         >> > Limited (a company registered in
> England and
>     > > > Wales,
>     > > >     > company
>     > > >     >     > number
>     > > >     >     >         >> > 01190902). Registered address at Cannon
> Bridge
>     > > > House, 25
>     > > >     >     > Dowgate Hill,
>     > > >     >     >         >> > London EC4R 2YA. Both IG Markets Limited
>     > > (register
>     > > >     > number
>     > > >     >     > 195355) and IG
>     > > >     >     >         >> > Index Limited (register number 114059)
> are
>     > > > authorised
>     > > >     > and
>     > > >     >     > regulated by
>     > > >     >     >         >> the
>     > > >     >     >         >> > Financial Conduct Authority.
>     > > >     >     >         >> >
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >>
>     > > >     >     >         >> --
>     > > >     >     >         >> -Regards,
>     > > >     >     >         >> Mayuresh R. Gharat
>     > > >     >     >         >> (862) 250-7125
>     > > >     >     >         >> The information contained in this email is
>     > > strictly
>     > > >     >     > confidential and for
>     > > >     >     >         >> the use of the addressee only, unless
> otherwise
>     > > >     > indicated. If
>     > > >     >     > you are not
>     > > >     >     >         >> the intended recipient, please do not
> read,
>     > copy,
>     > > > use or
>     > > >     >     > disclose to others
>     > > >     >     >         >> this message or any attachment. Please
> also
>     > notify
>     > > > the
>     > > >     > sender
>     > > >     >     > by replying
>     > > >     >     >         >> to this email or by telephone (+44(020
> 7896
>     > 0011)
>     > > > and then
>     > > >     >     > delete the email
>     > > >     >     >         >> and any copies of it. Opinions,
> conclusion (etc)
>     > > > that do
>     > > >     > not
>     > > >     >     > relate to the
>     > > >     >     >         >> official business of this company shall be
>     > > > understood as
>     > > >     >     > neither given nor
>     > > >     >     >         >> endorsed by it. IG is a trading name of IG
>     > Markets
>     > > >     > Limited (a
>     > > >     >     > company
>     > > >     >     >         >> registered in England and Wales, company
> number
>     > > > 04008957)
>     > > >     > and
>     > > >     >     > IG Index
>     > > >     >     >         >> Limited (a company registered in England
> and
>     > > Wales,
>     > > >     > company
>     > > >     >     > number
>     > > >     >     >         >> 01190902). Registered address at Cannon
> Bridge
>     > > > House, 25
>     > > >     >     > Dowgate Hill,
>     > > >     >     >         >> London EC4R 2YA. Both IG Markets Limited
>     > (register
>     > > > number
>     > > >     >     > 195355) and IG
>     > > >     >     >         >> Index Limited (register number 114059) are
>     > > > authorised and
>     > > >     >     > regulated by the
>     > > >     >     >         >> Financial Conduct Authority.
>     > > >     >     >         >>
>     > > >     >     >         >
>     > > >     >     >         >
>     > > >     >     >         >
>     > > >     >     >         > --
>     > > >     >     >         > -Regards,
>     > > >     >     >         > Mayuresh R. Gharat
>     > > >     >     >         > (862) 250-7125
>     > > >     >     >         >
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >         --
>     > > >     >     >         -Regards,
>     > > >     >     >         Mayuresh R. Gharat
>     > > >     >     >         (862) 250-7125
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >     The information contained in this email is
> strictly
>     > > > confidential
>     > > >     > and
>     > > >     >     > for the use of the addressee only, unless otherwise
>     > > indicated.
>     > > > If
>     > > >     > you are
>     > > >     >     > not the intended recipient, please do not read,
> copy, use
>     > or
>     > > >     > disclose to
>     > > >     >     > others this message or any attachment. Please also
> notify
>     > the
>     > > > sender
>     > > >     > by
>     > > >     >     > replying to this email or by telephone (+44(020 7896
> 0011)
>     > > and
>     > > > then
>     > > >     > delete
>     > > >     >     > the email and any copies of it. Opinions, conclusion
> (etc)
>     > > > that do
>     > > >     > not
>     > > >     >     > relate to the official business of this company
> shall be
>     > > > understood
>     > > >     > as
>     > > >     >     > neither given nor endorsed by it. IG is a trading
> name of
>     > IG
>     > > > Markets
>     > > >     >     > Limited (a company registered in England and Wales,
> company
>     > > > number
>     > > >     >     > 04008957) and IG Index Limited (a company registered
> in
>     > > > England and
>     > > >     > Wales,
>     > > >     >     > company number 01190902). Registered address at
> Cannon
>     > Bridge
>     > > > House,
>     > > >     > 25
>     > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets
> Limited
>     > > > (register
>     > > >     > number
>     > > >     >     > 195355) and IG Index Limited (register number
> 114059) are
>     > > > authorised
>     > > >     > and
>     > > >     >     > regulated by the Financial Conduct Authority.
>     > > >     >     >
>     > > >     >     >
>     > > >     >     >
>     > > >     >
>     > > >     >
>     > > >     > The information contained in this email is strictly
> confidential
>     > > and
>     > > > for
>     > > >     > the use of the addressee only, unless otherwise indicated.
> If you
>     > > > are not
>     > > >     > the intended recipient, please do not read, copy, use or
> disclose
>     > > to
>     > > > others
>     > > >     > this message or any attachment. Please also notify the
> sender by
>     > > > replying
>     > > >     > to this email or by telephone (+44(020 7896 0011) and then
> delete
>     > > > the email
>     > > >     > and any copies of it. Opinions, conclusion (etc) that do
> not
>     > relate
>     > > > to the
>     > > >     > official business of this company shall be understood as
> neither
>     > > > given nor
>     > > >     > endorsed by it. IG is a trading name of IG Markets Limited
> (a
>     > > company
>     > > >     > registered in England and Wales, company number 04008957)
> and IG
>     > > > Index
>     > > >     > Limited (a company registered in England and Wales, company
>     > number
>     > > >     > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
>     > > > Hill,
>     > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
>     > > > and IG
>     > > >     > Index Limited (register number 114059) are authorised and
>     > regulated
>     > > > by the
>     > > >     > Financial Conduct Authority.
>     > > >     >
>     > > >
>     > > >
>     > > > The information contained in this email is strictly confidential
> and
>     > for
>     > > > the use of the addressee only, unless otherwise indicated. If
> you are
>     > not
>     > > > the intended recipient, please do not read, copy, use or
> disclose to
>     > > others
>     > > > this message or any attachment. Please also notify the sender by
>     > replying
>     > > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
>     > > email
>     > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > > the
>     > > > official business of this company shall be understood as neither
> given
>     > > nor
>     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > > Limited (a company registered in England and Wales, company
> number
>     > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>     > IG
>     > > > Index Limited (register number 114059) are authorised and
> regulated by
>     > > the
>     > > > Financial Conduct Authority.
>     > > >
>     > > The information contained in this email is strictly confidential
> and for
>     > > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > > the intended recipient, please do not read, copy, use or disclose
> to
>     > others
>     > > this message or any attachment. Please also notify the sender by
> replying
>     > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
>     > email
>     > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>     > the
>     > > official business of this company shall be understood as neither
> given
>     > nor
>     > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>     > > registered in England and Wales, company number 04008957) and IG
> Index
>     > > Limited (a company registered in England and Wales, company number
>     > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > > Index Limited (register number 114059) are authorised and
> regulated by
>     > the
>     > > Financial Conduct Authority.
>     > >
>     > >
>     > The information contained in this email is strictly confidential and
> for
>     > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > the intended recipient, please do not read, copy, use or disclose to
> others
>     > this message or any attachment. Please also notify the sender by
> replying
>     > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
>     > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
>     > official business of this company shall be understood as neither
> given nor
>     > endorsed by it. IG is a trading name of IG Markets Limited (a company
>     > registered in England and Wales, company number 04008957) and IG
> Index
>     > Limited (a company registered in England and Wales, company number
>     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > Index Limited (register number 114059) are authorised and regulated
> by the
>     > Financial Conduct Authority.
>     >
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jun,

4) On v3 we honour the tombstone. As such we expect it to be set correctly as per the KIP.

4.1) why would we want to produce an error when its v3? This is the exact purpose to support non-null tombstone’s
4.2) again here im unclear on the question on the v3, produce request we expect the tombstone flag to be set correctly.

4.4) compaction only occurs on compacted topics, the bit makes no difference and not looked at on un-compacted (time/size based eviction).


On 06/12/2016, 20:08, "Jun Rao" <ju...@confluent.io> wrote:

    Hi, Michael,

    4. Then, I think I misunderstood this point. Could you document the
    following points in the wiki?
    4.1 If producer V3 sets tombstone, but provides a non-null value, does the
    send() get an error or does the producer automatically set the value to
    null?
    4.2 If producer V3 doesn't set tombstone, but provides a null value, does
    the send() get an error or does the producer automatically sets the
    tombstone?
    4.3 Does the broker only expect messages that (a) have no tombstone and
    non-null value; (b) have tombstone and null value and reject the messages
    with an error code otherwise?
    4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on not?

    Thanks,

    Jun

    On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <Mi...@ig.com>
    wrote:

    > Not at all.  This only acts on compacted topics just as what occurs today
    >
    > Sent using OWA for iPhone
    > ________________________________________
    > From: Jun Rao <ju...@confluent.io>
    > Sent: Tuesday, December 6, 2016 6:25:28 PM
    > To: dev@kafka.apache.org
    > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    >
    > Hi, Michael,
    >
    > 4. Hmm, does that mean the new client library can never send a null message
    > even to a regular topic? This seems like a change of the existing behavior.
    >
    > Thanks,
    >
    > Jun
    >
    > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <Mi...@ig.com>
    > wrote:
    >
    > > Hi Jun,
    > >
    > > Re 4) That's because we expect the tombstone value to be set correctly if
    > > message bit is 2, as such if an older client sends in on old message the
    > > message is upcast and the bit is set correctly. And such no longer need
    > to
    > > check the value. Mayuresh can you confirm my thinking and understanding
    > of
    > > what we've discussed?
    > >
    > > The second point I understand what you're getting at now my apologies.
    > Yes
    > > this makes sense to save on touching the message, if we're the only kip
    > > going in, in this release.
    > >
    > > Cheers
    > > Mike
    > >
    > > Sent using OWA for iPhone
    > > ________________________________________
    > > From: Jun Rao <ju...@confluent.io>
    > > Sent: Tuesday, December 6, 2016 5:22:13 PM
    > > To: dev@kafka.apache.org
    > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > >
    > > Hi, Michael,
    > >
    > > 4. Is this updated in the wiki? The text "If the magic byte on message is
    > > 2, the broker should use the tombstone bit for log compaction." doesn't
    > > seem to have changed.
    > >
    > > 2. My point is that if we change the message format just for this KIP, we
    > > should consider whether it's worth optimizing the down conversion path
    > > (i.e., decide whether a conversion is needed by just looking at the
    > > tombstone bit in the wrapper message) since tombstone will be used
    > rarely.
    > > However, if the message format change here is combined with other KIPs,
    > > then this optimization likely won't be needed. The latter probably makes
    > > the code simpler. Jiangjie, Mayuresh, what do you think?
    > >
    > > Other than those, +1 from me,
    > >
    > > Thanks,
    > >
    > > Jun
    > >
    > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <Mi...@ig.com>
    > > wrote:
    > >
    > > > Hi Jun
    > > >
    > > > do we have your vote on this now?
    > > >
    > > > Any other concerns?
    > > >
    > > > Cheers
    > > > Mike
    > > >
    > > > Sent using OWA for iPhone
    > > > ________________________________________
    > > > From: Michael Pearce <Mi...@ig.com>
    > > > Sent: Saturday, December 3, 2016 1:37:45 AM
    > > > To: dev@kafka.apache.org
    > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
    > > >
    > > > Hi Jun,
    > > >
    > > > Have updated. Thanks again for the feedback.
    > > >
    > > > Agree yes we should align up when it gets to that, I assume you’ve
    > > flagged
    > > > the same in the other KIP?
    > > >
    > > > I think for now let’s get this KIP approved, then we can start the work
    > > to
    > > > get an initial PR, then we can discuss how to align the two if needed.
    > > >
    > > > Cheers,
    > > > Mike
    > > >
    > > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
    > > >
    > > >     Hi, Michael,
    > > >
    > > >     For 2), this is fine. Could you update the KIP wiki to make this
    > and
    > > > other
    > > >     points clearer? Other than that, the KIP looks good to me.
    > > >
    > > >     An orthogonal thing is that there are other KIPs such as KIP-98
    > that
    > > > also
    > > >     intend to change the message format. If they all get approved, we
    > > > should
    > > >     think about whether it's better to just bump up the magic byte once
    > > to
    > > >     incorporate multiple format changes like we did in KIP-31/KIP-32.
    > > >
    > > >     Thanks,
    > > >
    > > >     Jun
    > > >
    > > >
    > > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
    > > Michael.Pearce@ig.com>
    > > >     wrote:
    > > >
    > > >     > Hi Jao
    > > >     >
    > > >     > Thanks for the response. Sorry for slow reply, both with personal
    > > > sickness
    > > >     > and also battling some critical issues encountered since
    > upgrading
    > > to
    > > >     > 0.10.1.0
    > > >     >
    > > >     > 1) Thans for spotting, Document error where we branched this KIP
    > > from
    > > >     > KIP-82, will get that removed.
    > > >     > 2) Intent is to do this just at the record message level.
    > > >     > 3) Thanks for spotting, Will ensure this is corrected.
    > > >     > 4) As per discussion thread we will support tombstone + null
    > value,
    > > >     > tombstone + non null value, no tombstone + null value.
    > > >     > 5) I believe this was in the discussion thread, @Mayuresh is this
    > > >     > something we’ve overlooked? I thought we would down convert and
    > > > remove the
    > > >     > value so the old consumer had existing behavior, or is there
    > > > something we
    > > >     > haven’t thought about?
    > > >     >
    > > >     > Cheers
    > > >     > Mike
    > > >     >
    > > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
    > > >     >
    > > >     >     Hi, Michael,
    > > >     >
    > > >     >     Thanks for the KIP. A few comments below.
    > > >     >
    > > >     >     1. The message format change contains "HeadersLength
    > Headers".
    > > > Is that
    > > >     >     intended?
    > > >     >
    > > >     >     2. For compressed messageset, is the tombstone bit only set
    > at
    > > > the
    > > >     > shallow
    > > >     >     level? Do we always leave that bit in the wrapper message
    > > unset?
    > > > An
    > > >     >     alternative is to set the tombstone bit in the wrapper if at
    > > > least one
    > > >     >     inner message has the tombstone bit set. This makes things a
    > > bit
    > > > more
    > > >     >     complicated, but we could potentially exploit that for
    > > > optimizing down
    > > >     >     conversion. For example, we only need to convert messages
    > with
    > > > magic 2
    > > >     > to
    > > >     >     magic 1 if the wrapper's tombstone bit is set (conversion is
    > > > always
    > > >     > needed
    > > >     >     from magic 2 to magic 0). Not sure if the optimization is
    > worth
    > > > the
    > > >     >     complexity though.
    > > >     >
    > > >     >     3. The referencing of the new version of
    > > > ProducerRequest/FetchRequest
    > > >     > is
    > > >     >     inconsistent (v4 vs v3). Since our convention starts at
    > version
    > > > at 0, I
    > > >     >     think the new version would be 3.
    > > >     >
    > > >     >     4. "If the magic byte on message is 2, the broker should use
    > > the
    > > >     > tombstone
    > > >     >     bit for log compaction." What about null value? My
    > > understanding
    > > > is
    > > >     > that
    > > >     >     null value will be treated the same as setting the tombstone
    > > bit.
    > > >     >
    > > >     >     5. For the migration path, it would be useful to describe the
    > > > down
    > > >     >     conversion path to consumers (i.e., brokers on message format
    > > > 0.10.2
    > > >     > and
    > > >     >     consumers on older version).
    > > >     >
    > > >     >     Thanks,
    > > >     >
    > > >     >     Jun
    > > >     >
    > > >     >
    > > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
    > > > Michael.Pearce@ig.com
    > > >     > >
    > > >     >     wrote:
    > > >     >
    > > >     >     > Hi All,
    > > >     >     >
    > > >     >     > We have been discussing in the below thread and final
    > changes
    > > > have
    > > >     > been
    > > >     >     > made to the KIP wiki based on these discussions.
    > > >     >     >
    > > >     >     > We would now like to put to the vote the following KIP:
    > > >     >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
    > 87+-+
    > > >     >     > Add+Compaction+Tombstone+Flag
    > > >     >     >
    > > >     >     > This kip is for having a distinct compaction attribute
    > > > “tombstone”
    > > >     > flag
    > > >     >     > instead of relying on null value, allowing non-null value
    > > > delete
    > > >     > messages.
    > > >     >     >
    > > >     >     > Many thanks,
    > > >     >     > Michael
    > > >     >     >
    > > >     >     >
    > > >     >     >
    > > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
    > > Michael.Pearce@ig.com>
    > > >     > wrote:
    > > >     >     >
    > > >     >     >     Hi Mayuresh,
    > > >     >     >
    > > >     >     >     LGTM. Ive just made one small adjustment updating the
    > > wire
    > > >     > protocol to
    > > >     >     > show the magic byte bump.
    > > >     >     >
    > > >     >     >     Do we think we’re good to put to a vote? Is there any
    > > > other bits
    > > >     >     > needing discussion?
    > > >     >     >
    > > >     >     >     Cheers
    > > >     >     >     Mike
    > > >     >     >
    > > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
    > > >     > gharatmayuresh15@gmail.com>
    > > >     >     > wrote:
    > > >     >     >
    > > >     >     >         Hi Michael,
    > > >     >     >
    > > >     >     >         I have updated the migration section of the KIP.
    > Can
    > > > you
    > > >     > please
    > > >     >     > take a look?
    > > >     >     >
    > > >     >     >         Thanks,
    > > >     >     >
    > > >     >     >         Mayuresh
    > > >     >     >
    > > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
    > > >     >     > gharatmayuresh15@gmail.com
    > > >     >     >         > wrote:
    > > >     >     >
    > > >     >     >         > Hi Michael,
    > > >     >     >         >
    > > >     >     >         > That whilst sending tombstone and non null value,
    > > the
    > > >     > consumer
    > > >     >     > can expect
    > > >     >     >         > only to receive the non-null message only in step
    > > > (3) is
    > > >     > this
    > > >     >     > correct?
    > > >     >     >         > ---> I do agree with you here.
    > > >     >     >         >
    > > >     >     >         > Becket, Ismael : can you guys review the
    > migration
    > > > plan
    > > >     > listed
    > > >     >     > above using
    > > >     >     >         > magic byte?
    > > >     >     >         >
    > > >     >     >         > Thanks,
    > > >     >     >         >
    > > >     >     >         > Mayuresh
    > > >     >     >         >
    > > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
    > > >     >     > Michael.Pearce@ig.com>
    > > >     >     >         > wrote:
    > > >     >     >         >
    > > >     >     >         >> Many thanks for this Mayuresh. I don't have any
    > > >     > objections.
    > > >     >     >         >>
    > > >     >     >         >> I assume we should state:
    > > >     >     >         >>
    > > >     >     >         >> That whilst sending tombstone and non null
    > value,
    > > > the
    > > >     > consumer
    > > >     >     > can expect
    > > >     >     >         >> only to receive the non-null message only in
    > step
    > > > (3) is
    > > >     > this
    > > >     >     > correct?
    > > >     >     >         >>
    > > >     >     >         >> Cheers
    > > >     >     >         >> Mike
    > > >     >     >         >>
    > > >     >     >         >>
    > > >     >     >         >>
    > > >     >     >         >> Sent using OWA for iPhone
    > > >     >     >         >> ________________________________________
    > > >     >     >         >> From: Mayuresh Gharat <
    > gharatmayuresh15@gmail.com
    > > >
    > > >     >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
    > > >     >     >         >> To: dev@kafka.apache.org
    > > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
    > > > Tombstone
    > > >     > Flag
    > > >     >     >         >>
    > > >     >     >         >> Hi Ismael,
    > > >     >     >         >>
    > > >     >     >         >> Thanks for the explanation.
    > > >     >     >         >> Specially I like this part where in you
    > mentioned
    > > > we can
    > > >     > get
    > > >     >     > rid of the
    > > >     >     >         >> older null value support for log compaction
    > later
    > > > on,
    > > >     > here :
    > > >     >     >         >> We can't change semantics of the message format
    > > > without
    > > >     > having
    > > >     >     > a long
    > > >     >     >         >> transition period. And we can't rely
    > > >     >     >         >> on people reading documentation or acting on a
    > > > warning for
    > > >     >     > something so
    > > >     >     >         >> fundamental. As such, my take is that we need to
    > > > bump the
    > > >     > magic
    > > >     >     > byte. The
    > > >     >     >         >> good news is
    > > >     >     >         >> that we don't have to support all versions
    > > forever.
    > > > We
    > > >     > have
    > > >     >     > said that we
    > > >     >     >         >> will support direct upgrades for 2 years. That
    > > > means that
    > > >     >     > message format
    > > >     >     >         >> version n could, in theory, be removed 2 years
    > > > after the
    > > >     > it's
    > > >     >     > introduced.
    > > >     >     >         >>
    > > >     >     >         >> Just a heads up, I would like to mention that
    > even
    > > > without
    > > >     >     > bumping magic
    > > >     >     >         >> byte, we will *NOT* loose zero copy as in the
    > > > client(x+1)
    > > >     > in my
    > > >     >     >         >> explanation
    > > >     >     >         >> above will convert internally a null value to
    > > have a
    > > >     > tombstone
    > > >     >     > bit set and
    > > >     >     >         >> a tombstone bit set to have a null value
    > > > automatically
    > > >     >     > internally and by
    > > >     >     >         >> the time we move to version (x+2), the clients
    > > > would have
    > > >     >     > upgraded.
    > > >     >     >         >> Obviously if we support a request from
    > > consumer(x),
    > > > we
    > > >     > will
    > > >     >     > loose zero
    > > >     >     >         >> copy
    > > >     >     >         >> but that is the same case with magic byte.
    > > >     >     >         >>
    > > >     >     >         >> But if magic byte bump makes life easier for
    > > > transition
    > > >     > for the
    > > >     >     > above
    > > >     >     >         >> reasons that you explained, I am OK with it
    > since
    > > > we are
    > > >     > going
    > > >     >     > to meet the
    > > >     >     >         >> end goal down the road :)
    > > >     >     >         >>
    > > >     >     >         >> On a side note can we update the doc here on
    > magic
    > > > byte
    > > >     > to say
    > > >     >     > that "*it
    > > >     >     >         >> should be bumped whenever the message format is
    > > > changed
    > > >     > or the
    > > >     >     >         >> interpretation of message format (usage of the
    > > > reserved
    > > >     > bits as
    > > >     >     > well) is
    > > >     >     >         >> changed*".
    > > >     >     >         >>
    > > >     >     >         >>
    > > >     >     >         >> Hi Michael,
    > > >     >     >         >>
    > > >     >     >         >> Here is the update plan that we discussed
    > offline
    > > >     > yesterday :
    > > >     >     >         >>
    > > >     >     >         >> Currently the magic-byte which corresponds to
    > the
    > > >     >     > "message.format.version"
    > > >     >     >         >> is set to 1.
    > > >     >     >         >>
    > > >     >     >         >> 1) On broker it will be set to 1 initially.
    > > >     >     >         >>
    > > >     >     >         >> 2) When a producer client sends a message with
    > > > magic-byte
    > > >     > = 2,
    > > >     >     > since the
    > > >     >     >         >> broker is on magic-byte = 1, we will down
    > convert
    > > > it,
    > > >     > which
    > > >     >     > means if the
    > > >     >     >         >> tombstone bit is set, the value will be set to
    > > > null. A
    > > >     > consumer
    > > >     >     >         >> understanding magic-byte = 1, will still work
    > with
    > > > this. A
    > > >     >     > consumer
    > > >     >     >         >> working
    > > >     >     >         >> with magic-byte =2 will also be able to
    > understand
    > > > this,
    > > >     > since
    > > >     >     > it
    > > >     >     >         >> understands the tombstone.
    > > >     >     >         >> Now there is still the question of supporting a
    > > >     > non-tombstone
    > > >     >     > and null
    > > >     >     >         >> value from producer client with magic-byte = 2.*
    > > (I
    > > > am
    > > >     > not sure
    > > >     >     > if we
    > > >     >     >         >> should support this. Ismael/Becket can comment
    > > > here)*
    > > >     >     >         >>
    > > >     >     >         >> 3) When almost all the clients have upgraded,
    > the
    > > >     >     > message.format.version
    > > >     >     >         >> on
    > > >     >     >         >> the broker can be changed to 2, where in the
    > down
    > > >     > conversion in
    > > >     >     > the above
    > > >     >     >         >> step will not happen. If at this point we get a
    > > > consumer
    > > >     >     > request from a
    > > >     >     >         >> older consumer, we might have to down convert
    > > where
    > > > in we
    > > >     > loose
    > > >     >     > zero copy,
    > > >     >     >         >> but these cases should be rare.
    > > >     >     >         >>
    > > >     >     >         >> Becket can you review this plan and add more
    > > > details if I
    > > >     > have
    > > >     >     >         >> missed/wronged something, before we put it on
    > KIP.
    > > >     >     >         >>
    > > >     >     >         >> Thanks,
    > > >     >     >         >>
    > > >     >     >         >> Mayuresh
    > > >     >     >         >>
    > > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael
    > Pearce <
    > > >     >     > Michael.Pearce@ig.com>
    > > >     >     >         >> wrote:
    > > >     >     >         >>
    > > >     >     >         >> > Thanks guys, for discussing this offline and
    > > > getting
    > > >     > some
    > > >     >     > consensus.
    > > >     >     >         >> >
    > > >     >     >         >> > So its clear for myself and others what is
    > > > proposed now
    > > >     > (i
    > > >     >     > think i
    > > >     >     >         >> > understand, but want to make sure)
    > > >     >     >         >> >
    > > >     >     >         >> > Could i ask either directly update the kip to
    > > > detail the
    > > >     >     > migration
    > > >     >     >         >> > strategy, or (re-)state your offline discussed
    > > and
    > > >     > agreed
    > > >     >     > migration
    > > >     >     >         >> > strategy based on a magic byte is in this
    > > thread.
    > > >     >     >         >> >
    > > >     >     >         >> >
    > > >     >     >         >> > The main original driver for the KIP was to
    > > > support
    > > >     >     > compaction where
    > > >     >     >         >> value
    > > >     >     >         >> > isn't null, based off the discussions on
    > KIP-82
    > > > thread.
    > > >     >     >         >> >
    > > >     >     >         >> > We should be able to support non-tombstone +
    > > null
    > > > value
    > > >     > by the
    > > >     >     >         >> completion
    > > >     >     >         >> > of the KIP, as we noted when discussing this
    > > kip,
    > > > having
    > > >     >     > logic based on
    > > >     >     >         >> a
    > > >     >     >         >> > null value isn't very clean and also separates
    > > the
    > > >     > concerns.
    > > >     >     >         >> >
    > > >     >     >         >> > As discussed already though we can split this
    > > into
    > > >     > KIP-87a
    > > >     >     > and KIP-87b
    > > >     >     >         >> >
    > > >     >     >         >> > Where we look to deliver KIP-87a on a
    > compacted
    > > > topic
    > > >     > (to
    > > >     >     > address the
    > > >     >     >         >> > immediate issues)
    > > >     >     >         >> > * tombstone + null value
    > > >     >     >         >> > * tombstone + non-null value
    > > >     >     >         >> > * non-tombstone + non-null value
    > > >     >     >         >> >
    > > >     >     >         >> > Then we can discuss once KIP-87a is completed
    > > > options
    > > >     > later
    > > >     >     > and how we
    > > >     >     >         >> > support the second part KIP-87b to deliver:
    > > >     >     >         >> > * non-tombstone + null value
    > > >     >     >         >> >
    > > >     >     >         >> > Cheers
    > > >     >     >         >> > Mike
    > > >     >     >         >> >
    > > >     >     >         >> >
    > > >     >     >         >> >
    > > >     >     >         >> > ________________________________________
    > > >     >     >         >> > From: Becket Qin <be...@gmail.com>
    > > >     >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
    > > >     >     >         >> > To: dev@kafka.apache.org
    > > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
    > > >     > Tombstone Flag
    > > >     >     >         >> >
    > > >     >     >         >> > Renu, Mayuresh and I had an offline
    > discussion,
    > > > and
    > > >     > following
    > > >     >     > is a brief
    > > >     >     >         >> > summary.
    > > >     >     >         >> >
    > > >     >     >         >> > 1. We agreed that not bumping up magic value
    > may
    > > > result
    > > >     > in
    > > >     >     > losing zero
    > > >     >     >         >> copy
    > > >     >     >         >> > during migration.
    > > >     >     >         >> > 2. Given that bumping up magic value is almost
    > > > free and
    > > >     > has
    > > >     >     > benefit of
    > > >     >     >         >> > avoiding potential performance issue. It is
    > > > probably
    > > >     > worth
    > > >     >     > doing.
    > > >     >     >         >> >
    > > >     >     >         >> > One issue we still need to think about is
    > > whether
    > > > we
    > > >     > want to
    > > >     >     > support a
    > > >     >     >         >> > non-tombstone message with null value.
    > > >     >     >         >> > Currently it is not supported by Kafka. If we
    > > > allow a
    > > >     >     > non-tombstone null
    > > >     >     >         >> > value message to exist after KIP-87. The
    > problem
    > > > is
    > > >     > that such
    > > >     >     > message
    > > >     >     >         >> will
    > > >     >     >         >> > not be supported by the consumers prior to
    > > KIP-87.
    > > >     > Because a
    > > >     >     > null value
    > > >     >     >         >> > will always be interpreted to a tombstone.
    > > >     >     >         >> >
    > > >     >     >         >> > One option is that we keep the current way,
    > i.e.
    > > > do not
    > > >     >     > support such
    > > >     >     >         >> > message. It would be good to know if there is
    > a
    > > >     > concrete use
    > > >     >     > case for
    > > >     >     >         >> such
    > > >     >     >         >> > message. If there is not, we can probably just
    > > not
    > > >     > support it.
    > > >     >     >         >> >
    > > >     >     >         >> > Thanks,
    > > >     >     >         >> >
    > > >     >     >         >> > JIangjie (Becket) Qin
    > > >     >     >         >> >
    > > >     >     >         >> >
    > > >     >     >         >> >
    > > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh
    > > Gharat <
    > > >     >     >         >> > gharatmayuresh15@gmail.com
    > > >     >     >         >> > > wrote:
    > > >     >     >         >> >
    > > >     >     >         >> > > Hi Ismael,
    > > >     >     >         >> > >
    > > >     >     >         >> > > This is something I can think of for
    > migration
    > > > plan:
    > > >     >     >         >> > > So the migration plan can look something
    > like
    > > > this,
    > > >     > with up
    > > >     >     >         >> conversion :
    > > >     >     >         >> > >
    > > >     >     >         >> > > 1) Currently lets say we have Broker at
    > > version
    > > > x.
    > > >     >     >         >> > > 2) Currently we have clients at version x.
    > > >     >     >         >> > > 3) a) We move the version to Broker(x+1) :
    > > > supports
    > > >     > both
    > > >     >     > tombstone and
    > > >     >     >         >> > null
    > > >     >     >         >> > > for log compaction.
    > > >     >     >         >> > >     b) We upgrade the client to version
    > > > client(x+1) :
    > > >     > if in
    > > >     >     > the
    > > >     >     >         >> producer
    > > >     >     >         >> > > client(x+1) the value is set to null, we
    > will
    > > >     > automatically
    > > >     >     > set the
    > > >     >     >         >> > > Tombstone bit internally. If the producer
    > > > client(x+1)
    > > >     > sets
    > > >     >     > the
    > > >     >     >         >> tombstone
    > > >     >     >         >> > > itself, well and good. For producer
    > client(x),
    > > > the
    > > >     > broker
    > > >     >     > will up
    > > >     >     >         >> convert
    > > >     >     >         >> > > to have the tombstone bit. Broker(x+1) is
    > > > supporting
    > > >     > both.
    > > >     >     > Consumer
    > > >     >     >         >> > > client(x+1) will be aware of this and should
    > > be
    > > > able
    > > >     > to
    > > >     >     > handle this.
    > > >     >     >         >> For
    > > >     >     >         >> > > consumer client(x) we will down convert the
    > > > message
    > > >     > on the
    > > >     >     > broker
    > > >     >     >         >> side.
    > > >     >     >         >> > >     c) At this point we will have to
    > specify a
    > > >     > warning or
    > > >     >     > clearly
    > > >     >     >         >> specify
    > > >     >     >         >> > > in docs that this behavior is about to be
    > > > changed for
    > > >     > log
    > > >     >     > compaction.
    > > >     >     >         >> > > 4) a) In next release of the Broker(x+2), we
    > > > say that
    > > >     > only
    > > >     >     > Tombstone
    > > >     >     >         >> is
    > > >     >     >         >> > > used for log compaction on the Broker side.
    > > >     > Clients(x+1)
    > > >     >     > still is
    > > >     >     >         >> > > supported.
    > > >     >     >         >> > >     b) We upgrade the client to version
    > > > client(x+2) :
    > > >     > if
    > > >     >     > value is set
    > > >     >     >         >> to
    > > >     >     >         >> > > null, tombstone will not be set
    > automatically.
    > > > The
    > > >     > client
    > > >     >     > will have to
    > > >     >     >         >> > call
    > > >     >     >         >> > > setTombstone() to actually set the
    > tombstone.
    > > >     >     >         >> > >
    > > >     >     >         >> > > We should compare this migration plan with
    > the
    > > >     > migration
    > > >     >     > plan for
    > > >     >     >         >> magic
    > > >     >     >         >> > > byte bump and do whatever looks good.
    > > >     >     >         >> > > I am just worried that if we go down magic
    > > byte
    > > > route,
    > > >     >     > unless I am
    > > >     >     >         >> > missing
    > > >     >     >         >> > > something, it sounds like kafka will be
    > stuck
    > > > with
    > > >     >     > supporting both
    > > >     >     >         >> null
    > > >     >     >         >> > > value and tombstone bit for log compaction
    > for
    > > > life
    > > >     > long,
    > > >     >     > which does
    > > >     >     >         >> not
    > > >     >     >         >> > > look like a good end state.
    > > >     >     >         >> > >
    > > >     >     >         >> > > Thanks,
    > > >     >     >         >> > >
    > > >     >     >         >> > > Mayuresh
    > > >     >     >         >> > >
    > > >     >     >         >> > >
    > > >     >     >         >> > >
    > > >     >     >         >> > >
    > > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh
    > > > Gharat <
    > > >     >     >         >> > > gharatmayuresh15@gmail.com
    > > >     >     >         >> > > > wrote:
    > > >     >     >         >> > >
    > > >     >     >         >> > > > Hi Ismael,
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > That's a very good point which I might
    > have
    > > > not
    > > >     >     > considered earlier.
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > Here is a plan that I can think of:
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > Stage 1) The broker from now on, up
    > converts
    > > > the
    > > >     > message
    > > >     >     > to have the
    > > >     >     >         >> > > > tombstone marker. The log compaction
    > thread
    > > > does log
    > > >     >     > compaction
    > > >     >     >         >> based
    > > >     >     >         >> > on
    > > >     >     >         >> > > > both null and tombstone marker. This is
    > our
    > > >     > transition
    > > >     >     > period.
    > > >     >     >         >> > > > Stage 2) The next release we only say that
    > > log
    > > >     > compaction
    > > >     >     > is based
    > > >     >     >         >> on
    > > >     >     >         >> > > > tombstone marker. (Open source kafka makes
    > > > this as a
    > > >     >     > policy). By
    > > >     >     >         >> this
    > > >     >     >         >> > > time,
    > > >     >     >         >> > > > the organization which is moving to this
    > > > release
    > > >     > will be
    > > >     >     > sure that
    > > >     >     >         >> they
    > > >     >     >         >> > > > have gone through the entire transition
    > > > period.
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > My only goal of doing this is that Kafka
    > > > clearly
    > > >     >     > specifies the end
    > > >     >     >         >> > state
    > > >     >     >         >> > > > about what log compaction means (is it
    > null
    > > > value
    > > >     > or a
    > > >     >     > tombstone
    > > >     >     >         >> > marker,
    > > >     >     >         >> > > > but not both).
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > What do you think?
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > Thanks,
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > Mayuresh
    > > >     >     >         >> > > > .
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael
    > > Juma <
    > > >     >     > ismael@juma.me.uk>
    > > >     >     >         >> > wrote:
    > > >     >     >         >> > > >
    > > >     >     >         >> > > >> One comment below.
    > > >     >     >         >> > > >>
    > > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
    > > > Gharat <
    > > >     >     >         >> > > >> gharatmayuresh15@gmail.com
    > > >     >     >         >> > > >> > wrote:
    > > >     >     >         >> > > >>
    > > >     >     >         >> > > >> >    - If we don't bump up the magic
    > byte,
    > > > on the
    > > >     > broker
    > > >     >     > side, the
    > > >     >     >         >> > > broker
    > > >     >     >         >> > > >> >    will always have to look at both
    > > > tombstone
    > > >     > bit and
    > > >     >     > the value
    > > >     >     >         >> when
    > > >     >     >         >> > > do
    > > >     >     >         >> > > >> the
    > > >     >     >         >> > > >> >    compaction. Assuming we do not bump
    > up
    > > > the
    > > >     > magic
    > > >     >     > byte,
    > > >     >     >         >> > > >> >    imagine the broker sees a message
    > > which
    > > > does
    > > >     > not
    > > >     >     > have a
    > > >     >     >         >> tombstone
    > > >     >     >         >> > > bit
    > > >     >     >         >> > > >> >    set. The broker does not know when
    > the
    > > >     > message was
    > > >     >     > produced
    > > >     >     >         >> (i.e.
    > > >     >     >         >> > > >> > whether
    > > >     >     >         >> > > >> >    the message has been up converted or
    > > > not), it
    > > >     > has
    > > >     >     > to take a
    > > >     >     >         >> > further
    > > >     >     >         >> > > >> > look at
    > > >     >     >         >> > > >> >    the value to see if it is null or
    > not
    > > in
    > > >     > order to
    > > >     >     > determine
    > > >     >     >         >> if it
    > > >     >     >         >> > > is
    > > >     >     >         >> > > >> a
    > > >     >     >         >> > > >> >    tombstone. The same logic has to be
    > > put
    > > > on the
    > > >     >     > consumer as
    > > >     >     >         >> well
    > > >     >     >         >> > > >> because
    > > >     >     >         >> > > >> > the
    > > >     >     >         >> > > >> >    consumer does not know if the
    > message
    > > > has
    > > >     > been up
    > > >     >     > converted or
    > > >     >     >         >> > not.
    > > >     >     >         >> > > >> >       - If we upconvert while
    > appending,
    > > > this is
    > > >     > not
    > > >     >     > the case,
    > > >     >     >         >> > right?
    > > >     >     >         >> > > >>
    > > >     >     >         >> > > >>
    > > >     >     >         >> > > >> If I understand you correctly, this is
    > not
    > > >     > sufficient
    > > >     >     > because the
    > > >     >     >         >> log
    > > >     >     >         >> > > may
    > > >     >     >         >> > > >> have messages appended before it was
    > > > upgraded to
    > > >     > include
    > > >     >     > KIP-87.
    > > >     >     >         >> > > >>
    > > >     >     >         >> > > >> Ismael
    > > >     >     >         >> > > >>
    > > >     >     >         >> > > >
    > > >     >     >         >> > > >
    > > >     >     >         >> > > >
    > > >     >     >         >> > > > --
    > > >     >     >         >> > > > -Regards,
    > > >     >     >         >> > > > Mayuresh R. Gharat
    > > >     >     >         >> > > > (862) 250-7125
    > > >     >     >         >> > > >
    > > >     >     >         >> > >
    > > >     >     >         >> > >
    > > >     >     >         >> > >
    > > >     >     >         >> > > --
    > > >     >     >         >> > > -Regards,
    > > >     >     >         >> > > Mayuresh R. Gharat
    > > >     >     >         >> > > (862) 250-7125
    > > >     >     >         >> > >
    > > >     >     >         >> > The information contained in this email is
    > > > strictly
    > > >     >     > confidential and for
    > > >     >     >         >> > the use of the addressee only, unless
    > otherwise
    > > >     > indicated. If
    > > >     >     > you are
    > > >     >     >         >> not
    > > >     >     >         >> > the intended recipient, please do not read,
    > > copy,
    > > > use or
    > > >     >     > disclose to
    > > >     >     >         >> others
    > > >     >     >         >> > this message or any attachment. Please also
    > > > notify the
    > > >     > sender
    > > >     >     > by
    > > >     >     >         >> replying
    > > >     >     >         >> > to this email or by telephone (+44(020 7896
    > > 0011)
    > > > and
    > > >     > then
    > > >     >     > delete the
    > > >     >     >         >> email
    > > >     >     >         >> > and any copies of it. Opinions, conclusion
    > (etc)
    > > > that
    > > >     > do not
    > > >     >     > relate to
    > > >     >     >         >> the
    > > >     >     >         >> > official business of this company shall be
    > > > understood as
    > > >     >     > neither given
    > > >     >     >         >> nor
    > > >     >     >         >> > endorsed by it. IG is a trading name of IG
    > > Markets
    > > >     > Limited (a
    > > >     >     > company
    > > >     >     >         >> > registered in England and Wales, company
    > number
    > > >     > 04008957) and
    > > >     >     > IG Index
    > > >     >     >         >> > Limited (a company registered in England and
    > > > Wales,
    > > >     > company
    > > >     >     > number
    > > >     >     >         >> > 01190902). Registered address at Cannon Bridge
    > > > House, 25
    > > >     >     > Dowgate Hill,
    > > >     >     >         >> > London EC4R 2YA. Both IG Markets Limited
    > > (register
    > > >     > number
    > > >     >     > 195355) and IG
    > > >     >     >         >> > Index Limited (register number 114059) are
    > > > authorised
    > > >     > and
    > > >     >     > regulated by
    > > >     >     >         >> the
    > > >     >     >         >> > Financial Conduct Authority.
    > > >     >     >         >> >
    > > >     >     >         >>
    > > >     >     >         >>
    > > >     >     >         >>
    > > >     >     >         >> --
    > > >     >     >         >> -Regards,
    > > >     >     >         >> Mayuresh R. Gharat
    > > >     >     >         >> (862) 250-7125
    > > >     >     >         >> The information contained in this email is
    > > strictly
    > > >     >     > confidential and for
    > > >     >     >         >> the use of the addressee only, unless otherwise
    > > >     > indicated. If
    > > >     >     > you are not
    > > >     >     >         >> the intended recipient, please do not read,
    > copy,
    > > > use or
    > > >     >     > disclose to others
    > > >     >     >         >> this message or any attachment. Please also
    > notify
    > > > the
    > > >     > sender
    > > >     >     > by replying
    > > >     >     >         >> to this email or by telephone (+44(020 7896
    > 0011)
    > > > and then
    > > >     >     > delete the email
    > > >     >     >         >> and any copies of it. Opinions, conclusion (etc)
    > > > that do
    > > >     > not
    > > >     >     > relate to the
    > > >     >     >         >> official business of this company shall be
    > > > understood as
    > > >     >     > neither given nor
    > > >     >     >         >> endorsed by it. IG is a trading name of IG
    > Markets
    > > >     > Limited (a
    > > >     >     > company
    > > >     >     >         >> registered in England and Wales, company number
    > > > 04008957)
    > > >     > and
    > > >     >     > IG Index
    > > >     >     >         >> Limited (a company registered in England and
    > > Wales,
    > > >     > company
    > > >     >     > number
    > > >     >     >         >> 01190902). Registered address at Cannon Bridge
    > > > House, 25
    > > >     >     > Dowgate Hill,
    > > >     >     >         >> London EC4R 2YA. Both IG Markets Limited
    > (register
    > > > number
    > > >     >     > 195355) and IG
    > > >     >     >         >> Index Limited (register number 114059) are
    > > > authorised and
    > > >     >     > regulated by the
    > > >     >     >         >> Financial Conduct Authority.
    > > >     >     >         >>
    > > >     >     >         >
    > > >     >     >         >
    > > >     >     >         >
    > > >     >     >         > --
    > > >     >     >         > -Regards,
    > > >     >     >         > Mayuresh R. Gharat
    > > >     >     >         > (862) 250-7125
    > > >     >     >         >
    > > >     >     >
    > > >     >     >
    > > >     >     >
    > > >     >     >         --
    > > >     >     >         -Regards,
    > > >     >     >         Mayuresh R. Gharat
    > > >     >     >         (862) 250-7125
    > > >     >     >
    > > >     >     >
    > > >     >     >     The information contained in this email is strictly
    > > > confidential
    > > >     > and
    > > >     >     > for the use of the addressee only, unless otherwise
    > > indicated.
    > > > If
    > > >     > you are
    > > >     >     > not the intended recipient, please do not read, copy, use
    > or
    > > >     > disclose to
    > > >     >     > others this message or any attachment. Please also notify
    > the
    > > > sender
    > > >     > by
    > > >     >     > replying to this email or by telephone (+44(020 7896 0011)
    > > and
    > > > then
    > > >     > delete
    > > >     >     > the email and any copies of it. Opinions, conclusion (etc)
    > > > that do
    > > >     > not
    > > >     >     > relate to the official business of this company shall be
    > > > understood
    > > >     > as
    > > >     >     > neither given nor endorsed by it. IG is a trading name of
    > IG
    > > > Markets
    > > >     >     > Limited (a company registered in England and Wales, company
    > > > number
    > > >     >     > 04008957) and IG Index Limited (a company registered in
    > > > England and
    > > >     > Wales,
    > > >     >     > company number 01190902). Registered address at Cannon
    > Bridge
    > > > House,
    > > >     > 25
    > > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
    > > > (register
    > > >     > number
    > > >     >     > 195355) and IG Index Limited (register number 114059) are
    > > > authorised
    > > >     > and
    > > >     >     > regulated by the Financial Conduct Authority.
    > > >     >     >
    > > >     >     >
    > > >     >     >
    > > >     >
    > > >     >
    > > >     > The information contained in this email is strictly confidential
    > > and
    > > > for
    > > >     > the use of the addressee only, unless otherwise indicated. If you
    > > > are not
    > > >     > the intended recipient, please do not read, copy, use or disclose
    > > to
    > > > others
    > > >     > this message or any attachment. Please also notify the sender by
    > > > replying
    > > >     > to this email or by telephone (+44(020 7896 0011) and then delete
    > > > the email
    > > >     > and any copies of it. Opinions, conclusion (etc) that do not
    > relate
    > > > to the
    > > >     > official business of this company shall be understood as neither
    > > > given nor
    > > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
    > > company
    > > >     > registered in England and Wales, company number 04008957) and IG
    > > > Index
    > > >     > Limited (a company registered in England and Wales, company
    > number
    > > >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > > > Hill,
    > > >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
    > > > and IG
    > > >     > Index Limited (register number 114059) are authorised and
    > regulated
    > > > by the
    > > >     > Financial Conduct Authority.
    > > >     >
    > > >
    > > >
    > > > The information contained in this email is strictly confidential and
    > for
    > > > the use of the addressee only, unless otherwise indicated. If you are
    > not
    > > > the intended recipient, please do not read, copy, use or disclose to
    > > others
    > > > this message or any attachment. Please also notify the sender by
    > replying
    > > > to this email or by telephone (+44(020 7896 0011) and then delete the
    > > email
    > > > and any copies of it. Opinions, conclusion (etc) that do not relate to
    > > the
    > > > official business of this company shall be understood as neither given
    > > nor
    > > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > > registered in England and Wales, company number 04008957) and IG Index
    > > > Limited (a company registered in England and Wales, company number
    > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    > IG
    > > > Index Limited (register number 114059) are authorised and regulated by
    > > the
    > > > Financial Conduct Authority.
    > > >
    > > The information contained in this email is strictly confidential and for
    > > the use of the addressee only, unless otherwise indicated. If you are not
    > > the intended recipient, please do not read, copy, use or disclose to
    > others
    > > this message or any attachment. Please also notify the sender by replying
    > > to this email or by telephone (+44(020 7896 0011) and then delete the
    > email
    > > and any copies of it. Opinions, conclusion (etc) that do not relate to
    > the
    > > official business of this company shall be understood as neither given
    > nor
    > > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > registered in England and Wales, company number 04008957) and IG Index
    > > Limited (a company registered in England and Wales, company number
    > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    > > Index Limited (register number 114059) are authorised and regulated by
    > the
    > > Financial Conduct Authority.
    > >
    > >
    > The information contained in this email is strictly confidential and for
    > the use of the addressee only, unless otherwise indicated. If you are not
    > the intended recipient, please do not read, copy, use or disclose to others
    > this message or any attachment. Please also notify the sender by replying
    > to this email or by telephone (+44(020 7896 0011) and then delete the email
    > and any copies of it. Opinions, conclusion (etc) that do not relate to the
    > official business of this company shall be understood as neither given nor
    > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > registered in England and Wales, company number 04008957) and IG Index
    > Limited (a company registered in England and Wales, company number
    > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    > Index Limited (register number 114059) are authorised and regulated by the
    > Financial Conduct Authority.
    >
    >


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jun Rao <ju...@confluent.io>]

Hi, Michael,

4. Then, I think I misunderstood this point. Could you document the
following points in the wiki?
4.1 If producer V3 sets tombstone, but provides a non-null value, does the
send() get an error or does the producer automatically set the value to
null?
4.2 If producer V3 doesn't set tombstone, but provides a null value, does
the send() get an error or does the producer automatically sets the
tombstone?
4.3 Does the broker only expect messages that (a) have no tombstone and
non-null value; (b) have tombstone and null value and reject the messages
with an error code otherwise?
4.4 Do 4.1, 4.2,  4.3 depend on whether the topic is compacted on not?

Thanks,

Jun

On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Not at all.  This only acts on compacted topics just as what occurs today
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jun Rao <ju...@confluent.io>
> Sent: Tuesday, December 6, 2016 6:25:28 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hi, Michael,
>
> 4. Hmm, does that mean the new client library can never send a null message
> even to a regular topic? This seems like a change of the existing behavior.
>
> Thanks,
>
> Jun
>
> On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jun,
> >
> > Re 4) That's because we expect the tombstone value to be set correctly if
> > message bit is 2, as such if an older client sends in on old message the
> > message is upcast and the bit is set correctly. And such no longer need
> to
> > check the value. Mayuresh can you confirm my thinking and understanding
> of
> > what we've discussed?
> >
> > The second point I understand what you're getting at now my apologies.
> Yes
> > this makes sense to save on touching the message, if we're the only kip
> > going in, in this release.
> >
> > Cheers
> > Mike
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jun Rao <ju...@confluent.io>
> > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hi, Michael,
> >
> > 4. Is this updated in the wiki? The text "If the magic byte on message is
> > 2, the broker should use the tombstone bit for log compaction." doesn't
> > seem to have changed.
> >
> > 2. My point is that if we change the message format just for this KIP, we
> > should consider whether it's worth optimizing the down conversion path
> > (i.e., decide whether a conversion is needed by just looking at the
> > tombstone bit in the wrapper message) since tombstone will be used
> rarely.
> > However, if the message format change here is combined with other KIPs,
> > then this optimization likely won't be needed. The latter probably makes
> > the code simpler. Jiangjie, Mayuresh, what do you think?
> >
> > Other than those, +1 from me,
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jun
> > >
> > > do we have your vote on this now?
> > >
> > > Any other concerns?
> > >
> > > Cheers
> > > Mike
> > >
> > > Sent using OWA for iPhone
> > > ________________________________________
> > > From: Michael Pearce <Mi...@ig.com>
> > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > > Hi Jun,
> > >
> > > Have updated. Thanks again for the feedback.
> > >
> > > Agree yes we should align up when it gets to that, I assume you’ve
> > flagged
> > > the same in the other KIP?
> > >
> > > I think for now let’s get this KIP approved, then we can start the work
> > to
> > > get an initial PR, then we can discuss how to align the two if needed.
> > >
> > > Cheers,
> > > Mike
> > >
> > > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> > >
> > >     Hi, Michael,
> > >
> > >     For 2), this is fine. Could you update the KIP wiki to make this
> and
> > > other
> > >     points clearer? Other than that, the KIP looks good to me.
> > >
> > >     An orthogonal thing is that there are other KIPs such as KIP-98
> that
> > > also
> > >     intend to change the message format. If they all get approved, we
> > > should
> > >     think about whether it's better to just bump up the magic byte once
> > to
> > >     incorporate multiple format changes like we did in KIP-31/KIP-32.
> > >
> > >     Thanks,
> > >
> > >     Jun
> > >
> > >
> > >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > >     wrote:
> > >
> > >     > Hi Jao
> > >     >
> > >     > Thanks for the response. Sorry for slow reply, both with personal
> > > sickness
> > >     > and also battling some critical issues encountered since
> upgrading
> > to
> > >     > 0.10.1.0
> > >     >
> > >     > 1) Thans for spotting, Document error where we branched this KIP
> > from
> > >     > KIP-82, will get that removed.
> > >     > 2) Intent is to do this just at the record message level.
> > >     > 3) Thanks for spotting, Will ensure this is corrected.
> > >     > 4) As per discussion thread we will support tombstone + null
> value,
> > >     > tombstone + non null value, no tombstone + null value.
> > >     > 5) I believe this was in the discussion thread, @Mayuresh is this
> > >     > something we’ve overlooked? I thought we would down convert and
> > > remove the
> > >     > value so the old consumer had existing behavior, or is there
> > > something we
> > >     > haven’t thought about?
> > >     >
> > >     > Cheers
> > >     > Mike
> > >     >
> > >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
> > >     >
> > >     >     Hi, Michael,
> > >     >
> > >     >     Thanks for the KIP. A few comments below.
> > >     >
> > >     >     1. The message format change contains "HeadersLength
> Headers".
> > > Is that
> > >     >     intended?
> > >     >
> > >     >     2. For compressed messageset, is the tombstone bit only set
> at
> > > the
> > >     > shallow
> > >     >     level? Do we always leave that bit in the wrapper message
> > unset?
> > > An
> > >     >     alternative is to set the tombstone bit in the wrapper if at
> > > least one
> > >     >     inner message has the tombstone bit set. This makes things a
> > bit
> > > more
> > >     >     complicated, but we could potentially exploit that for
> > > optimizing down
> > >     >     conversion. For example, we only need to convert messages
> with
> > > magic 2
> > >     > to
> > >     >     magic 1 if the wrapper's tombstone bit is set (conversion is
> > > always
> > >     > needed
> > >     >     from magic 2 to magic 0). Not sure if the optimization is
> worth
> > > the
> > >     >     complexity though.
> > >     >
> > >     >     3. The referencing of the new version of
> > > ProducerRequest/FetchRequest
> > >     > is
> > >     >     inconsistent (v4 vs v3). Since our convention starts at
> version
> > > at 0, I
> > >     >     think the new version would be 3.
> > >     >
> > >     >     4. "If the magic byte on message is 2, the broker should use
> > the
> > >     > tombstone
> > >     >     bit for log compaction." What about null value? My
> > understanding
> > > is
> > >     > that
> > >     >     null value will be treated the same as setting the tombstone
> > bit.
> > >     >
> > >     >     5. For the migration path, it would be useful to describe the
> > > down
> > >     >     conversion path to consumers (i.e., brokers on message format
> > > 0.10.2
> > >     > and
> > >     >     consumers on older version).
> > >     >
> > >     >     Thanks,
> > >     >
> > >     >     Jun
> > >     >
> > >     >
> > >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > >     > >
> > >     >     wrote:
> > >     >
> > >     >     > Hi All,
> > >     >     >
> > >     >     > We have been discussing in the below thread and final
> changes
> > > have
> > >     > been
> > >     >     > made to the KIP wiki based on these discussions.
> > >     >     >
> > >     >     > We would now like to put to the vote the following KIP:
> > >     >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 87+-+
> > >     >     > Add+Compaction+Tombstone+Flag
> > >     >     >
> > >     >     > This kip is for having a distinct compaction attribute
> > > “tombstone”
> > >     > flag
> > >     >     > instead of relying on null value, allowing non-null value
> > > delete
> > >     > messages.
> > >     >     >
> > >     >     > Many thanks,
> > >     >     > Michael
> > >     >     >
> > >     >     >
> > >     >     >
> > >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > >     > wrote:
> > >     >     >
> > >     >     >     Hi Mayuresh,
> > >     >     >
> > >     >     >     LGTM. Ive just made one small adjustment updating the
> > wire
> > >     > protocol to
> > >     >     > show the magic byte bump.
> > >     >     >
> > >     >     >     Do we think we’re good to put to a vote? Is there any
> > > other bits
> > >     >     > needing discussion?
> > >     >     >
> > >     >     >     Cheers
> > >     >     >     Mike
> > >     >     >
> > >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> > >     > gharatmayuresh15@gmail.com>
> > >     >     > wrote:
> > >     >     >
> > >     >     >         Hi Michael,
> > >     >     >
> > >     >     >         I have updated the migration section of the KIP.
> Can
> > > you
> > >     > please
> > >     >     > take a look?
> > >     >     >
> > >     >     >         Thanks,
> > >     >     >
> > >     >     >         Mayuresh
> > >     >     >
> > >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> > >     >     > gharatmayuresh15@gmail.com
> > >     >     >         > wrote:
> > >     >     >
> > >     >     >         > Hi Michael,
> > >     >     >         >
> > >     >     >         > That whilst sending tombstone and non null value,
> > the
> > >     > consumer
> > >     >     > can expect
> > >     >     >         > only to receive the non-null message only in step
> > > (3) is
> > >     > this
> > >     >     > correct?
> > >     >     >         > ---> I do agree with you here.
> > >     >     >         >
> > >     >     >         > Becket, Ismael : can you guys review the
> migration
> > > plan
> > >     > listed
> > >     >     > above using
> > >     >     >         > magic byte?
> > >     >     >         >
> > >     >     >         > Thanks,
> > >     >     >         >
> > >     >     >         > Mayuresh
> > >     >     >         >
> > >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
> > >     >     > Michael.Pearce@ig.com>
> > >     >     >         > wrote:
> > >     >     >         >
> > >     >     >         >> Many thanks for this Mayuresh. I don't have any
> > >     > objections.
> > >     >     >         >>
> > >     >     >         >> I assume we should state:
> > >     >     >         >>
> > >     >     >         >> That whilst sending tombstone and non null
> value,
> > > the
> > >     > consumer
> > >     >     > can expect
> > >     >     >         >> only to receive the non-null message only in
> step
> > > (3) is
> > >     > this
> > >     >     > correct?
> > >     >     >         >>
> > >     >     >         >> Cheers
> > >     >     >         >> Mike
> > >     >     >         >>
> > >     >     >         >>
> > >     >     >         >>
> > >     >     >         >> Sent using OWA for iPhone
> > >     >     >         >> ________________________________________
> > >     >     >         >> From: Mayuresh Gharat <
> gharatmayuresh15@gmail.com
> > >
> > >     >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
> > >     >     >         >> To: dev@kafka.apache.org
> > >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> > > Tombstone
> > >     > Flag
> > >     >     >         >>
> > >     >     >         >> Hi Ismael,
> > >     >     >         >>
> > >     >     >         >> Thanks for the explanation.
> > >     >     >         >> Specially I like this part where in you
> mentioned
> > > we can
> > >     > get
> > >     >     > rid of the
> > >     >     >         >> older null value support for log compaction
> later
> > > on,
> > >     > here :
> > >     >     >         >> We can't change semantics of the message format
> > > without
> > >     > having
> > >     >     > a long
> > >     >     >         >> transition period. And we can't rely
> > >     >     >         >> on people reading documentation or acting on a
> > > warning for
> > >     >     > something so
> > >     >     >         >> fundamental. As such, my take is that we need to
> > > bump the
> > >     > magic
> > >     >     > byte. The
> > >     >     >         >> good news is
> > >     >     >         >> that we don't have to support all versions
> > forever.
> > > We
> > >     > have
> > >     >     > said that we
> > >     >     >         >> will support direct upgrades for 2 years. That
> > > means that
> > >     >     > message format
> > >     >     >         >> version n could, in theory, be removed 2 years
> > > after the
> > >     > it's
> > >     >     > introduced.
> > >     >     >         >>
> > >     >     >         >> Just a heads up, I would like to mention that
> even
> > > without
> > >     >     > bumping magic
> > >     >     >         >> byte, we will *NOT* loose zero copy as in the
> > > client(x+1)
> > >     > in my
> > >     >     >         >> explanation
> > >     >     >         >> above will convert internally a null value to
> > have a
> > >     > tombstone
> > >     >     > bit set and
> > >     >     >         >> a tombstone bit set to have a null value
> > > automatically
> > >     >     > internally and by
> > >     >     >         >> the time we move to version (x+2), the clients
> > > would have
> > >     >     > upgraded.
> > >     >     >         >> Obviously if we support a request from
> > consumer(x),
> > > we
> > >     > will
> > >     >     > loose zero
> > >     >     >         >> copy
> > >     >     >         >> but that is the same case with magic byte.
> > >     >     >         >>
> > >     >     >         >> But if magic byte bump makes life easier for
> > > transition
> > >     > for the
> > >     >     > above
> > >     >     >         >> reasons that you explained, I am OK with it
> since
> > > we are
> > >     > going
> > >     >     > to meet the
> > >     >     >         >> end goal down the road :)
> > >     >     >         >>
> > >     >     >         >> On a side note can we update the doc here on
> magic
> > > byte
> > >     > to say
> > >     >     > that "*it
> > >     >     >         >> should be bumped whenever the message format is
> > > changed
> > >     > or the
> > >     >     >         >> interpretation of message format (usage of the
> > > reserved
> > >     > bits as
> > >     >     > well) is
> > >     >     >         >> changed*".
> > >     >     >         >>
> > >     >     >         >>
> > >     >     >         >> Hi Michael,
> > >     >     >         >>
> > >     >     >         >> Here is the update plan that we discussed
> offline
> > >     > yesterday :
> > >     >     >         >>
> > >     >     >         >> Currently the magic-byte which corresponds to
> the
> > >     >     > "message.format.version"
> > >     >     >         >> is set to 1.
> > >     >     >         >>
> > >     >     >         >> 1) On broker it will be set to 1 initially.
> > >     >     >         >>
> > >     >     >         >> 2) When a producer client sends a message with
> > > magic-byte
> > >     > = 2,
> > >     >     > since the
> > >     >     >         >> broker is on magic-byte = 1, we will down
> convert
> > > it,
> > >     > which
> > >     >     > means if the
> > >     >     >         >> tombstone bit is set, the value will be set to
> > > null. A
> > >     > consumer
> > >     >     >         >> understanding magic-byte = 1, will still work
> with
> > > this. A
> > >     >     > consumer
> > >     >     >         >> working
> > >     >     >         >> with magic-byte =2 will also be able to
> understand
> > > this,
> > >     > since
> > >     >     > it
> > >     >     >         >> understands the tombstone.
> > >     >     >         >> Now there is still the question of supporting a
> > >     > non-tombstone
> > >     >     > and null
> > >     >     >         >> value from producer client with magic-byte = 2.*
> > (I
> > > am
> > >     > not sure
> > >     >     > if we
> > >     >     >         >> should support this. Ismael/Becket can comment
> > > here)*
> > >     >     >         >>
> > >     >     >         >> 3) When almost all the clients have upgraded,
> the
> > >     >     > message.format.version
> > >     >     >         >> on
> > >     >     >         >> the broker can be changed to 2, where in the
> down
> > >     > conversion in
> > >     >     > the above
> > >     >     >         >> step will not happen. If at this point we get a
> > > consumer
> > >     >     > request from a
> > >     >     >         >> older consumer, we might have to down convert
> > where
> > > in we
> > >     > loose
> > >     >     > zero copy,
> > >     >     >         >> but these cases should be rare.
> > >     >     >         >>
> > >     >     >         >> Becket can you review this plan and add more
> > > details if I
> > >     > have
> > >     >     >         >> missed/wronged something, before we put it on
> KIP.
> > >     >     >         >>
> > >     >     >         >> Thanks,
> > >     >     >         >>
> > >     >     >         >> Mayuresh
> > >     >     >         >>
> > >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael
> Pearce <
> > >     >     > Michael.Pearce@ig.com>
> > >     >     >         >> wrote:
> > >     >     >         >>
> > >     >     >         >> > Thanks guys, for discussing this offline and
> > > getting
> > >     > some
> > >     >     > consensus.
> > >     >     >         >> >
> > >     >     >         >> > So its clear for myself and others what is
> > > proposed now
> > >     > (i
> > >     >     > think i
> > >     >     >         >> > understand, but want to make sure)
> > >     >     >         >> >
> > >     >     >         >> > Could i ask either directly update the kip to
> > > detail the
> > >     >     > migration
> > >     >     >         >> > strategy, or (re-)state your offline discussed
> > and
> > >     > agreed
> > >     >     > migration
> > >     >     >         >> > strategy based on a magic byte is in this
> > thread.
> > >     >     >         >> >
> > >     >     >         >> >
> > >     >     >         >> > The main original driver for the KIP was to
> > > support
> > >     >     > compaction where
> > >     >     >         >> value
> > >     >     >         >> > isn't null, based off the discussions on
> KIP-82
> > > thread.
> > >     >     >         >> >
> > >     >     >         >> > We should be able to support non-tombstone +
> > null
> > > value
> > >     > by the
> > >     >     >         >> completion
> > >     >     >         >> > of the KIP, as we noted when discussing this
> > kip,
> > > having
> > >     >     > logic based on
> > >     >     >         >> a
> > >     >     >         >> > null value isn't very clean and also separates
> > the
> > >     > concerns.
> > >     >     >         >> >
> > >     >     >         >> > As discussed already though we can split this
> > into
> > >     > KIP-87a
> > >     >     > and KIP-87b
> > >     >     >         >> >
> > >     >     >         >> > Where we look to deliver KIP-87a on a
> compacted
> > > topic
> > >     > (to
> > >     >     > address the
> > >     >     >         >> > immediate issues)
> > >     >     >         >> > * tombstone + null value
> > >     >     >         >> > * tombstone + non-null value
> > >     >     >         >> > * non-tombstone + non-null value
> > >     >     >         >> >
> > >     >     >         >> > Then we can discuss once KIP-87a is completed
> > > options
> > >     > later
> > >     >     > and how we
> > >     >     >         >> > support the second part KIP-87b to deliver:
> > >     >     >         >> > * non-tombstone + null value
> > >     >     >         >> >
> > >     >     >         >> > Cheers
> > >     >     >         >> > Mike
> > >     >     >         >> >
> > >     >     >         >> >
> > >     >     >         >> >
> > >     >     >         >> > ________________________________________
> > >     >     >         >> > From: Becket Qin <be...@gmail.com>
> > >     >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
> > >     >     >         >> > To: dev@kafka.apache.org
> > >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> > >     > Tombstone Flag
> > >     >     >         >> >
> > >     >     >         >> > Renu, Mayuresh and I had an offline
> discussion,
> > > and
> > >     > following
> > >     >     > is a brief
> > >     >     >         >> > summary.
> > >     >     >         >> >
> > >     >     >         >> > 1. We agreed that not bumping up magic value
> may
> > > result
> > >     > in
> > >     >     > losing zero
> > >     >     >         >> copy
> > >     >     >         >> > during migration.
> > >     >     >         >> > 2. Given that bumping up magic value is almost
> > > free and
> > >     > has
> > >     >     > benefit of
> > >     >     >         >> > avoiding potential performance issue. It is
> > > probably
> > >     > worth
> > >     >     > doing.
> > >     >     >         >> >
> > >     >     >         >> > One issue we still need to think about is
> > whether
> > > we
> > >     > want to
> > >     >     > support a
> > >     >     >         >> > non-tombstone message with null value.
> > >     >     >         >> > Currently it is not supported by Kafka. If we
> > > allow a
> > >     >     > non-tombstone null
> > >     >     >         >> > value message to exist after KIP-87. The
> problem
> > > is
> > >     > that such
> > >     >     > message
> > >     >     >         >> will
> > >     >     >         >> > not be supported by the consumers prior to
> > KIP-87.
> > >     > Because a
> > >     >     > null value
> > >     >     >         >> > will always be interpreted to a tombstone.
> > >     >     >         >> >
> > >     >     >         >> > One option is that we keep the current way,
> i.e.
> > > do not
> > >     >     > support such
> > >     >     >         >> > message. It would be good to know if there is
> a
> > >     > concrete use
> > >     >     > case for
> > >     >     >         >> such
> > >     >     >         >> > message. If there is not, we can probably just
> > not
> > >     > support it.
> > >     >     >         >> >
> > >     >     >         >> > Thanks,
> > >     >     >         >> >
> > >     >     >         >> > JIangjie (Becket) Qin
> > >     >     >         >> >
> > >     >     >         >> >
> > >     >     >         >> >
> > >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh
> > Gharat <
> > >     >     >         >> > gharatmayuresh15@gmail.com
> > >     >     >         >> > > wrote:
> > >     >     >         >> >
> > >     >     >         >> > > Hi Ismael,
> > >     >     >         >> > >
> > >     >     >         >> > > This is something I can think of for
> migration
> > > plan:
> > >     >     >         >> > > So the migration plan can look something
> like
> > > this,
> > >     > with up
> > >     >     >         >> conversion :
> > >     >     >         >> > >
> > >     >     >         >> > > 1) Currently lets say we have Broker at
> > version
> > > x.
> > >     >     >         >> > > 2) Currently we have clients at version x.
> > >     >     >         >> > > 3) a) We move the version to Broker(x+1) :
> > > supports
> > >     > both
> > >     >     > tombstone and
> > >     >     >         >> > null
> > >     >     >         >> > > for log compaction.
> > >     >     >         >> > >     b) We upgrade the client to version
> > > client(x+1) :
> > >     > if in
> > >     >     > the
> > >     >     >         >> producer
> > >     >     >         >> > > client(x+1) the value is set to null, we
> will
> > >     > automatically
> > >     >     > set the
> > >     >     >         >> > > Tombstone bit internally. If the producer
> > > client(x+1)
> > >     > sets
> > >     >     > the
> > >     >     >         >> tombstone
> > >     >     >         >> > > itself, well and good. For producer
> client(x),
> > > the
> > >     > broker
> > >     >     > will up
> > >     >     >         >> convert
> > >     >     >         >> > > to have the tombstone bit. Broker(x+1) is
> > > supporting
> > >     > both.
> > >     >     > Consumer
> > >     >     >         >> > > client(x+1) will be aware of this and should
> > be
> > > able
> > >     > to
> > >     >     > handle this.
> > >     >     >         >> For
> > >     >     >         >> > > consumer client(x) we will down convert the
> > > message
> > >     > on the
> > >     >     > broker
> > >     >     >         >> side.
> > >     >     >         >> > >     c) At this point we will have to
> specify a
> > >     > warning or
> > >     >     > clearly
> > >     >     >         >> specify
> > >     >     >         >> > > in docs that this behavior is about to be
> > > changed for
> > >     > log
> > >     >     > compaction.
> > >     >     >         >> > > 4) a) In next release of the Broker(x+2), we
> > > say that
> > >     > only
> > >     >     > Tombstone
> > >     >     >         >> is
> > >     >     >         >> > > used for log compaction on the Broker side.
> > >     > Clients(x+1)
> > >     >     > still is
> > >     >     >         >> > > supported.
> > >     >     >         >> > >     b) We upgrade the client to version
> > > client(x+2) :
> > >     > if
> > >     >     > value is set
> > >     >     >         >> to
> > >     >     >         >> > > null, tombstone will not be set
> automatically.
> > > The
> > >     > client
> > >     >     > will have to
> > >     >     >         >> > call
> > >     >     >         >> > > setTombstone() to actually set the
> tombstone.
> > >     >     >         >> > >
> > >     >     >         >> > > We should compare this migration plan with
> the
> > >     > migration
> > >     >     > plan for
> > >     >     >         >> magic
> > >     >     >         >> > > byte bump and do whatever looks good.
> > >     >     >         >> > > I am just worried that if we go down magic
> > byte
> > > route,
> > >     >     > unless I am
> > >     >     >         >> > missing
> > >     >     >         >> > > something, it sounds like kafka will be
> stuck
> > > with
> > >     >     > supporting both
> > >     >     >         >> null
> > >     >     >         >> > > value and tombstone bit for log compaction
> for
> > > life
> > >     > long,
> > >     >     > which does
> > >     >     >         >> not
> > >     >     >         >> > > look like a good end state.
> > >     >     >         >> > >
> > >     >     >         >> > > Thanks,
> > >     >     >         >> > >
> > >     >     >         >> > > Mayuresh
> > >     >     >         >> > >
> > >     >     >         >> > >
> > >     >     >         >> > >
> > >     >     >         >> > >
> > >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh
> > > Gharat <
> > >     >     >         >> > > gharatmayuresh15@gmail.com
> > >     >     >         >> > > > wrote:
> > >     >     >         >> > >
> > >     >     >         >> > > > Hi Ismael,
> > >     >     >         >> > > >
> > >     >     >         >> > > > That's a very good point which I might
> have
> > > not
> > >     >     > considered earlier.
> > >     >     >         >> > > >
> > >     >     >         >> > > > Here is a plan that I can think of:
> > >     >     >         >> > > >
> > >     >     >         >> > > > Stage 1) The broker from now on, up
> converts
> > > the
> > >     > message
> > >     >     > to have the
> > >     >     >         >> > > > tombstone marker. The log compaction
> thread
> > > does log
> > >     >     > compaction
> > >     >     >         >> based
> > >     >     >         >> > on
> > >     >     >         >> > > > both null and tombstone marker. This is
> our
> > >     > transition
> > >     >     > period.
> > >     >     >         >> > > > Stage 2) The next release we only say that
> > log
> > >     > compaction
> > >     >     > is based
> > >     >     >         >> on
> > >     >     >         >> > > > tombstone marker. (Open source kafka makes
> > > this as a
> > >     >     > policy). By
> > >     >     >         >> this
> > >     >     >         >> > > time,
> > >     >     >         >> > > > the organization which is moving to this
> > > release
> > >     > will be
> > >     >     > sure that
> > >     >     >         >> they
> > >     >     >         >> > > > have gone through the entire transition
> > > period.
> > >     >     >         >> > > >
> > >     >     >         >> > > > My only goal of doing this is that Kafka
> > > clearly
> > >     >     > specifies the end
> > >     >     >         >> > state
> > >     >     >         >> > > > about what log compaction means (is it
> null
> > > value
> > >     > or a
> > >     >     > tombstone
> > >     >     >         >> > marker,
> > >     >     >         >> > > > but not both).
> > >     >     >         >> > > >
> > >     >     >         >> > > > What do you think?
> > >     >     >         >> > > >
> > >     >     >         >> > > > Thanks,
> > >     >     >         >> > > >
> > >     >     >         >> > > > Mayuresh
> > >     >     >         >> > > > .
> > >     >     >         >> > > >
> > >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael
> > Juma <
> > >     >     > ismael@juma.me.uk>
> > >     >     >         >> > wrote:
> > >     >     >         >> > > >
> > >     >     >         >> > > >> One comment below.
> > >     >     >         >> > > >>
> > >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
> > > Gharat <
> > >     >     >         >> > > >> gharatmayuresh15@gmail.com
> > >     >     >         >> > > >> > wrote:
> > >     >     >         >> > > >>
> > >     >     >         >> > > >> >    - If we don't bump up the magic
> byte,
> > > on the
> > >     > broker
> > >     >     > side, the
> > >     >     >         >> > > broker
> > >     >     >         >> > > >> >    will always have to look at both
> > > tombstone
> > >     > bit and
> > >     >     > the value
> > >     >     >         >> when
> > >     >     >         >> > > do
> > >     >     >         >> > > >> the
> > >     >     >         >> > > >> >    compaction. Assuming we do not bump
> up
> > > the
> > >     > magic
> > >     >     > byte,
> > >     >     >         >> > > >> >    imagine the broker sees a message
> > which
> > > does
> > >     > not
> > >     >     > have a
> > >     >     >         >> tombstone
> > >     >     >         >> > > bit
> > >     >     >         >> > > >> >    set. The broker does not know when
> the
> > >     > message was
> > >     >     > produced
> > >     >     >         >> (i.e.
> > >     >     >         >> > > >> > whether
> > >     >     >         >> > > >> >    the message has been up converted or
> > > not), it
> > >     > has
> > >     >     > to take a
> > >     >     >         >> > further
> > >     >     >         >> > > >> > look at
> > >     >     >         >> > > >> >    the value to see if it is null or
> not
> > in
> > >     > order to
> > >     >     > determine
> > >     >     >         >> if it
> > >     >     >         >> > > is
> > >     >     >         >> > > >> a
> > >     >     >         >> > > >> >    tombstone. The same logic has to be
> > put
> > > on the
> > >     >     > consumer as
> > >     >     >         >> well
> > >     >     >         >> > > >> because
> > >     >     >         >> > > >> > the
> > >     >     >         >> > > >> >    consumer does not know if the
> message
> > > has
> > >     > been up
> > >     >     > converted or
> > >     >     >         >> > not.
> > >     >     >         >> > > >> >       - If we upconvert while
> appending,
> > > this is
> > >     > not
> > >     >     > the case,
> > >     >     >         >> > right?
> > >     >     >         >> > > >>
> > >     >     >         >> > > >>
> > >     >     >         >> > > >> If I understand you correctly, this is
> not
> > >     > sufficient
> > >     >     > because the
> > >     >     >         >> log
> > >     >     >         >> > > may
> > >     >     >         >> > > >> have messages appended before it was
> > > upgraded to
> > >     > include
> > >     >     > KIP-87.
> > >     >     >         >> > > >>
> > >     >     >         >> > > >> Ismael
> > >     >     >         >> > > >>
> > >     >     >         >> > > >
> > >     >     >         >> > > >
> > >     >     >         >> > > >
> > >     >     >         >> > > > --
> > >     >     >         >> > > > -Regards,
> > >     >     >         >> > > > Mayuresh R. Gharat
> > >     >     >         >> > > > (862) 250-7125
> > >     >     >         >> > > >
> > >     >     >         >> > >
> > >     >     >         >> > >
> > >     >     >         >> > >
> > >     >     >         >> > > --
> > >     >     >         >> > > -Regards,
> > >     >     >         >> > > Mayuresh R. Gharat
> > >     >     >         >> > > (862) 250-7125
> > >     >     >         >> > >
> > >     >     >         >> > The information contained in this email is
> > > strictly
> > >     >     > confidential and for
> > >     >     >         >> > the use of the addressee only, unless
> otherwise
> > >     > indicated. If
> > >     >     > you are
> > >     >     >         >> not
> > >     >     >         >> > the intended recipient, please do not read,
> > copy,
> > > use or
> > >     >     > disclose to
> > >     >     >         >> others
> > >     >     >         >> > this message or any attachment. Please also
> > > notify the
> > >     > sender
> > >     >     > by
> > >     >     >         >> replying
> > >     >     >         >> > to this email or by telephone (+44(020 7896
> > 0011)
> > > and
> > >     > then
> > >     >     > delete the
> > >     >     >         >> email
> > >     >     >         >> > and any copies of it. Opinions, conclusion
> (etc)
> > > that
> > >     > do not
> > >     >     > relate to
> > >     >     >         >> the
> > >     >     >         >> > official business of this company shall be
> > > understood as
> > >     >     > neither given
> > >     >     >         >> nor
> > >     >     >         >> > endorsed by it. IG is a trading name of IG
> > Markets
> > >     > Limited (a
> > >     >     > company
> > >     >     >         >> > registered in England and Wales, company
> number
> > >     > 04008957) and
> > >     >     > IG Index
> > >     >     >         >> > Limited (a company registered in England and
> > > Wales,
> > >     > company
> > >     >     > number
> > >     >     >         >> > 01190902). Registered address at Cannon Bridge
> > > House, 25
> > >     >     > Dowgate Hill,
> > >     >     >         >> > London EC4R 2YA. Both IG Markets Limited
> > (register
> > >     > number
> > >     >     > 195355) and IG
> > >     >     >         >> > Index Limited (register number 114059) are
> > > authorised
> > >     > and
> > >     >     > regulated by
> > >     >     >         >> the
> > >     >     >         >> > Financial Conduct Authority.
> > >     >     >         >> >
> > >     >     >         >>
> > >     >     >         >>
> > >     >     >         >>
> > >     >     >         >> --
> > >     >     >         >> -Regards,
> > >     >     >         >> Mayuresh R. Gharat
> > >     >     >         >> (862) 250-7125
> > >     >     >         >> The information contained in this email is
> > strictly
> > >     >     > confidential and for
> > >     >     >         >> the use of the addressee only, unless otherwise
> > >     > indicated. If
> > >     >     > you are not
> > >     >     >         >> the intended recipient, please do not read,
> copy,
> > > use or
> > >     >     > disclose to others
> > >     >     >         >> this message or any attachment. Please also
> notify
> > > the
> > >     > sender
> > >     >     > by replying
> > >     >     >         >> to this email or by telephone (+44(020 7896
> 0011)
> > > and then
> > >     >     > delete the email
> > >     >     >         >> and any copies of it. Opinions, conclusion (etc)
> > > that do
> > >     > not
> > >     >     > relate to the
> > >     >     >         >> official business of this company shall be
> > > understood as
> > >     >     > neither given nor
> > >     >     >         >> endorsed by it. IG is a trading name of IG
> Markets
> > >     > Limited (a
> > >     >     > company
> > >     >     >         >> registered in England and Wales, company number
> > > 04008957)
> > >     > and
> > >     >     > IG Index
> > >     >     >         >> Limited (a company registered in England and
> > Wales,
> > >     > company
> > >     >     > number
> > >     >     >         >> 01190902). Registered address at Cannon Bridge
> > > House, 25
> > >     >     > Dowgate Hill,
> > >     >     >         >> London EC4R 2YA. Both IG Markets Limited
> (register
> > > number
> > >     >     > 195355) and IG
> > >     >     >         >> Index Limited (register number 114059) are
> > > authorised and
> > >     >     > regulated by the
> > >     >     >         >> Financial Conduct Authority.
> > >     >     >         >>
> > >     >     >         >
> > >     >     >         >
> > >     >     >         >
> > >     >     >         > --
> > >     >     >         > -Regards,
> > >     >     >         > Mayuresh R. Gharat
> > >     >     >         > (862) 250-7125
> > >     >     >         >
> > >     >     >
> > >     >     >
> > >     >     >
> > >     >     >         --
> > >     >     >         -Regards,
> > >     >     >         Mayuresh R. Gharat
> > >     >     >         (862) 250-7125
> > >     >     >
> > >     >     >
> > >     >     >     The information contained in this email is strictly
> > > confidential
> > >     > and
> > >     >     > for the use of the addressee only, unless otherwise
> > indicated.
> > > If
> > >     > you are
> > >     >     > not the intended recipient, please do not read, copy, use
> or
> > >     > disclose to
> > >     >     > others this message or any attachment. Please also notify
> the
> > > sender
> > >     > by
> > >     >     > replying to this email or by telephone (+44(020 7896 0011)
> > and
> > > then
> > >     > delete
> > >     >     > the email and any copies of it. Opinions, conclusion (etc)
> > > that do
> > >     > not
> > >     >     > relate to the official business of this company shall be
> > > understood
> > >     > as
> > >     >     > neither given nor endorsed by it. IG is a trading name of
> IG
> > > Markets
> > >     >     > Limited (a company registered in England and Wales, company
> > > number
> > >     >     > 04008957) and IG Index Limited (a company registered in
> > > England and
> > >     > Wales,
> > >     >     > company number 01190902). Registered address at Cannon
> Bridge
> > > House,
> > >     > 25
> > >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > > (register
> > >     > number
> > >     >     > 195355) and IG Index Limited (register number 114059) are
> > > authorised
> > >     > and
> > >     >     > regulated by the Financial Conduct Authority.
> > >     >     >
> > >     >     >
> > >     >     >
> > >     >
> > >     >
> > >     > The information contained in this email is strictly confidential
> > and
> > > for
> > >     > the use of the addressee only, unless otherwise indicated. If you
> > > are not
> > >     > the intended recipient, please do not read, copy, use or disclose
> > to
> > > others
> > >     > this message or any attachment. Please also notify the sender by
> > > replying
> > >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > > the email
> > >     > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > > to the
> > >     > official business of this company shall be understood as neither
> > > given nor
> > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > >     > registered in England and Wales, company number 04008957) and IG
> > > Index
> > >     > Limited (a company registered in England and Wales, company
> number
> > >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > > and IG
> > >     > Index Limited (register number 114059) are authorised and
> regulated
> > > by the
> > >     > Financial Conduct Authority.
> > >     >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Not at all.  This only acts on compacted topics just as what occurs today

Sent using OWA for iPhone
________________________________________
From: Jun Rao <ju...@confluent.io>
Sent: Tuesday, December 6, 2016 6:25:28 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi, Michael,

4. Hmm, does that mean the new client library can never send a null message
even to a regular topic? This seems like a change of the existing behavior.

Thanks,

Jun

On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jun,
>
> Re 4) That's because we expect the tombstone value to be set correctly if
> message bit is 2, as such if an older client sends in on old message the
> message is upcast and the bit is set correctly. And such no longer need to
> check the value. Mayuresh can you confirm my thinking and understanding of
> what we've discussed?
>
> The second point I understand what you're getting at now my apologies. Yes
> this makes sense to save on touching the message, if we're the only kip
> going in, in this release.
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jun Rao <ju...@confluent.io>
> Sent: Tuesday, December 6, 2016 5:22:13 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hi, Michael,
>
> 4. Is this updated in the wiki? The text "If the magic byte on message is
> 2, the broker should use the tombstone bit for log compaction." doesn't
> seem to have changed.
>
> 2. My point is that if we change the message format just for this KIP, we
> should consider whether it's worth optimizing the down conversion path
> (i.e., decide whether a conversion is needed by just looking at the
> tombstone bit in the wrapper message) since tombstone will be used rarely.
> However, if the message format change here is combined with other KIPs,
> then this optimization likely won't be needed. The latter probably makes
> the code simpler. Jiangjie, Mayuresh, what do you think?
>
> Other than those, +1 from me,
>
> Thanks,
>
> Jun
>
> On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jun
> >
> > do we have your vote on this now?
> >
> > Any other concerns?
> >
> > Cheers
> > Mike
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Michael Pearce <Mi...@ig.com>
> > Sent: Saturday, December 3, 2016 1:37:45 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hi Jun,
> >
> > Have updated. Thanks again for the feedback.
> >
> > Agree yes we should align up when it gets to that, I assume you’ve
> flagged
> > the same in the other KIP?
> >
> > I think for now let’s get this KIP approved, then we can start the work
> to
> > get an initial PR, then we can discuss how to align the two if needed.
> >
> > Cheers,
> > Mike
> >
> > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> >
> >     Hi, Michael,
> >
> >     For 2), this is fine. Could you update the KIP wiki to make this and
> > other
> >     points clearer? Other than that, the KIP looks good to me.
> >
> >     An orthogonal thing is that there are other KIPs such as KIP-98 that
> > also
> >     intend to change the message format. If they all get approved, we
> > should
> >     think about whether it's better to just bump up the magic byte once
> to
> >     incorporate multiple format changes like we did in KIP-31/KIP-32.
> >
> >     Thanks,
> >
> >     Jun
> >
> >
> >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> >     wrote:
> >
> >     > Hi Jao
> >     >
> >     > Thanks for the response. Sorry for slow reply, both with personal
> > sickness
> >     > and also battling some critical issues encountered since upgrading
> to
> >     > 0.10.1.0
> >     >
> >     > 1) Thans for spotting, Document error where we branched this KIP
> from
> >     > KIP-82, will get that removed.
> >     > 2) Intent is to do this just at the record message level.
> >     > 3) Thanks for spotting, Will ensure this is corrected.
> >     > 4) As per discussion thread we will support tombstone + null value,
> >     > tombstone + non null value, no tombstone + null value.
> >     > 5) I believe this was in the discussion thread, @Mayuresh is this
> >     > something we’ve overlooked? I thought we would down convert and
> > remove the
> >     > value so the old consumer had existing behavior, or is there
> > something we
> >     > haven’t thought about?
> >     >
> >     > Cheers
> >     > Mike
> >     >
> >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
> >     >
> >     >     Hi, Michael,
> >     >
> >     >     Thanks for the KIP. A few comments below.
> >     >
> >     >     1. The message format change contains "HeadersLength Headers".
> > Is that
> >     >     intended?
> >     >
> >     >     2. For compressed messageset, is the tombstone bit only set at
> > the
> >     > shallow
> >     >     level? Do we always leave that bit in the wrapper message
> unset?
> > An
> >     >     alternative is to set the tombstone bit in the wrapper if at
> > least one
> >     >     inner message has the tombstone bit set. This makes things a
> bit
> > more
> >     >     complicated, but we could potentially exploit that for
> > optimizing down
> >     >     conversion. For example, we only need to convert messages with
> > magic 2
> >     > to
> >     >     magic 1 if the wrapper's tombstone bit is set (conversion is
> > always
> >     > needed
> >     >     from magic 2 to magic 0). Not sure if the optimization is worth
> > the
> >     >     complexity though.
> >     >
> >     >     3. The referencing of the new version of
> > ProducerRequest/FetchRequest
> >     > is
> >     >     inconsistent (v4 vs v3). Since our convention starts at version
> > at 0, I
> >     >     think the new version would be 3.
> >     >
> >     >     4. "If the magic byte on message is 2, the broker should use
> the
> >     > tombstone
> >     >     bit for log compaction." What about null value? My
> understanding
> > is
> >     > that
> >     >     null value will be treated the same as setting the tombstone
> bit.
> >     >
> >     >     5. For the migration path, it would be useful to describe the
> > down
> >     >     conversion path to consumers (i.e., brokers on message format
> > 0.10.2
> >     > and
> >     >     consumers on older version).
> >     >
> >     >     Thanks,
> >     >
> >     >     Jun
> >     >
> >     >
> >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> >     > >
> >     >     wrote:
> >     >
> >     >     > Hi All,
> >     >     >
> >     >     > We have been discussing in the below thread and final changes
> > have
> >     > been
> >     >     > made to the KIP wiki based on these discussions.
> >     >     >
> >     >     > We would now like to put to the vote the following KIP:
> >     >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
> >     >     > Add+Compaction+Tombstone+Flag
> >     >     >
> >     >     > This kip is for having a distinct compaction attribute
> > “tombstone”
> >     > flag
> >     >     > instead of relying on null value, allowing non-null value
> > delete
> >     > messages.
> >     >     >
> >     >     > Many thanks,
> >     >     > Michael
> >     >     >
> >     >     >
> >     >     >
> >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> Michael.Pearce@ig.com>
> >     > wrote:
> >     >     >
> >     >     >     Hi Mayuresh,
> >     >     >
> >     >     >     LGTM. Ive just made one small adjustment updating the
> wire
> >     > protocol to
> >     >     > show the magic byte bump.
> >     >     >
> >     >     >     Do we think we’re good to put to a vote? Is there any
> > other bits
> >     >     > needing discussion?
> >     >     >
> >     >     >     Cheers
> >     >     >     Mike
> >     >     >
> >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> >     > gharatmayuresh15@gmail.com>
> >     >     > wrote:
> >     >     >
> >     >     >         Hi Michael,
> >     >     >
> >     >     >         I have updated the migration section of the KIP. Can
> > you
> >     > please
> >     >     > take a look?
> >     >     >
> >     >     >         Thanks,
> >     >     >
> >     >     >         Mayuresh
> >     >     >
> >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> >     >     > gharatmayuresh15@gmail.com
> >     >     >         > wrote:
> >     >     >
> >     >     >         > Hi Michael,
> >     >     >         >
> >     >     >         > That whilst sending tombstone and non null value,
> the
> >     > consumer
> >     >     > can expect
> >     >     >         > only to receive the non-null message only in step
> > (3) is
> >     > this
> >     >     > correct?
> >     >     >         > ---> I do agree with you here.
> >     >     >         >
> >     >     >         > Becket, Ismael : can you guys review the migration
> > plan
> >     > listed
> >     >     > above using
> >     >     >         > magic byte?
> >     >     >         >
> >     >     >         > Thanks,
> >     >     >         >
> >     >     >         > Mayuresh
> >     >     >         >
> >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
> >     >     > Michael.Pearce@ig.com>
> >     >     >         > wrote:
> >     >     >         >
> >     >     >         >> Many thanks for this Mayuresh. I don't have any
> >     > objections.
> >     >     >         >>
> >     >     >         >> I assume we should state:
> >     >     >         >>
> >     >     >         >> That whilst sending tombstone and non null value,
> > the
> >     > consumer
> >     >     > can expect
> >     >     >         >> only to receive the non-null message only in step
> > (3) is
> >     > this
> >     >     > correct?
> >     >     >         >>
> >     >     >         >> Cheers
> >     >     >         >> Mike
> >     >     >         >>
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> Sent using OWA for iPhone
> >     >     >         >> ________________________________________
> >     >     >         >> From: Mayuresh Gharat <gharatmayuresh15@gmail.com
> >
> >     >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
> >     >     >         >> To: dev@kafka.apache.org
> >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> > Tombstone
> >     > Flag
> >     >     >         >>
> >     >     >         >> Hi Ismael,
> >     >     >         >>
> >     >     >         >> Thanks for the explanation.
> >     >     >         >> Specially I like this part where in you mentioned
> > we can
> >     > get
> >     >     > rid of the
> >     >     >         >> older null value support for log compaction later
> > on,
> >     > here :
> >     >     >         >> We can't change semantics of the message format
> > without
> >     > having
> >     >     > a long
> >     >     >         >> transition period. And we can't rely
> >     >     >         >> on people reading documentation or acting on a
> > warning for
> >     >     > something so
> >     >     >         >> fundamental. As such, my take is that we need to
> > bump the
> >     > magic
> >     >     > byte. The
> >     >     >         >> good news is
> >     >     >         >> that we don't have to support all versions
> forever.
> > We
> >     > have
> >     >     > said that we
> >     >     >         >> will support direct upgrades for 2 years. That
> > means that
> >     >     > message format
> >     >     >         >> version n could, in theory, be removed 2 years
> > after the
> >     > it's
> >     >     > introduced.
> >     >     >         >>
> >     >     >         >> Just a heads up, I would like to mention that even
> > without
> >     >     > bumping magic
> >     >     >         >> byte, we will *NOT* loose zero copy as in the
> > client(x+1)
> >     > in my
> >     >     >         >> explanation
> >     >     >         >> above will convert internally a null value to
> have a
> >     > tombstone
> >     >     > bit set and
> >     >     >         >> a tombstone bit set to have a null value
> > automatically
> >     >     > internally and by
> >     >     >         >> the time we move to version (x+2), the clients
> > would have
> >     >     > upgraded.
> >     >     >         >> Obviously if we support a request from
> consumer(x),
> > we
> >     > will
> >     >     > loose zero
> >     >     >         >> copy
> >     >     >         >> but that is the same case with magic byte.
> >     >     >         >>
> >     >     >         >> But if magic byte bump makes life easier for
> > transition
> >     > for the
> >     >     > above
> >     >     >         >> reasons that you explained, I am OK with it since
> > we are
> >     > going
> >     >     > to meet the
> >     >     >         >> end goal down the road :)
> >     >     >         >>
> >     >     >         >> On a side note can we update the doc here on magic
> > byte
> >     > to say
> >     >     > that "*it
> >     >     >         >> should be bumped whenever the message format is
> > changed
> >     > or the
> >     >     >         >> interpretation of message format (usage of the
> > reserved
> >     > bits as
> >     >     > well) is
> >     >     >         >> changed*".
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> Hi Michael,
> >     >     >         >>
> >     >     >         >> Here is the update plan that we discussed offline
> >     > yesterday :
> >     >     >         >>
> >     >     >         >> Currently the magic-byte which corresponds to the
> >     >     > "message.format.version"
> >     >     >         >> is set to 1.
> >     >     >         >>
> >     >     >         >> 1) On broker it will be set to 1 initially.
> >     >     >         >>
> >     >     >         >> 2) When a producer client sends a message with
> > magic-byte
> >     > = 2,
> >     >     > since the
> >     >     >         >> broker is on magic-byte = 1, we will down convert
> > it,
> >     > which
> >     >     > means if the
> >     >     >         >> tombstone bit is set, the value will be set to
> > null. A
> >     > consumer
> >     >     >         >> understanding magic-byte = 1, will still work with
> > this. A
> >     >     > consumer
> >     >     >         >> working
> >     >     >         >> with magic-byte =2 will also be able to understand
> > this,
> >     > since
> >     >     > it
> >     >     >         >> understands the tombstone.
> >     >     >         >> Now there is still the question of supporting a
> >     > non-tombstone
> >     >     > and null
> >     >     >         >> value from producer client with magic-byte = 2.*
> (I
> > am
> >     > not sure
> >     >     > if we
> >     >     >         >> should support this. Ismael/Becket can comment
> > here)*
> >     >     >         >>
> >     >     >         >> 3) When almost all the clients have upgraded, the
> >     >     > message.format.version
> >     >     >         >> on
> >     >     >         >> the broker can be changed to 2, where in the down
> >     > conversion in
> >     >     > the above
> >     >     >         >> step will not happen. If at this point we get a
> > consumer
> >     >     > request from a
> >     >     >         >> older consumer, we might have to down convert
> where
> > in we
> >     > loose
> >     >     > zero copy,
> >     >     >         >> but these cases should be rare.
> >     >     >         >>
> >     >     >         >> Becket can you review this plan and add more
> > details if I
> >     > have
> >     >     >         >> missed/wronged something, before we put it on KIP.
> >     >     >         >>
> >     >     >         >> Thanks,
> >     >     >         >>
> >     >     >         >> Mayuresh
> >     >     >         >>
> >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
> >     >     > Michael.Pearce@ig.com>
> >     >     >         >> wrote:
> >     >     >         >>
> >     >     >         >> > Thanks guys, for discussing this offline and
> > getting
> >     > some
> >     >     > consensus.
> >     >     >         >> >
> >     >     >         >> > So its clear for myself and others what is
> > proposed now
> >     > (i
> >     >     > think i
> >     >     >         >> > understand, but want to make sure)
> >     >     >         >> >
> >     >     >         >> > Could i ask either directly update the kip to
> > detail the
> >     >     > migration
> >     >     >         >> > strategy, or (re-)state your offline discussed
> and
> >     > agreed
> >     >     > migration
> >     >     >         >> > strategy based on a magic byte is in this
> thread.
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > The main original driver for the KIP was to
> > support
> >     >     > compaction where
> >     >     >         >> value
> >     >     >         >> > isn't null, based off the discussions on KIP-82
> > thread.
> >     >     >         >> >
> >     >     >         >> > We should be able to support non-tombstone +
> null
> > value
> >     > by the
> >     >     >         >> completion
> >     >     >         >> > of the KIP, as we noted when discussing this
> kip,
> > having
> >     >     > logic based on
> >     >     >         >> a
> >     >     >         >> > null value isn't very clean and also separates
> the
> >     > concerns.
> >     >     >         >> >
> >     >     >         >> > As discussed already though we can split this
> into
> >     > KIP-87a
> >     >     > and KIP-87b
> >     >     >         >> >
> >     >     >         >> > Where we look to deliver KIP-87a on a compacted
> > topic
> >     > (to
> >     >     > address the
> >     >     >         >> > immediate issues)
> >     >     >         >> > * tombstone + null value
> >     >     >         >> > * tombstone + non-null value
> >     >     >         >> > * non-tombstone + non-null value
> >     >     >         >> >
> >     >     >         >> > Then we can discuss once KIP-87a is completed
> > options
> >     > later
> >     >     > and how we
> >     >     >         >> > support the second part KIP-87b to deliver:
> >     >     >         >> > * non-tombstone + null value
> >     >     >         >> >
> >     >     >         >> > Cheers
> >     >     >         >> > Mike
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > ________________________________________
> >     >     >         >> > From: Becket Qin <be...@gmail.com>
> >     >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
> >     >     >         >> > To: dev@kafka.apache.org
> >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> >     > Tombstone Flag
> >     >     >         >> >
> >     >     >         >> > Renu, Mayuresh and I had an offline discussion,
> > and
> >     > following
> >     >     > is a brief
> >     >     >         >> > summary.
> >     >     >         >> >
> >     >     >         >> > 1. We agreed that not bumping up magic value may
> > result
> >     > in
> >     >     > losing zero
> >     >     >         >> copy
> >     >     >         >> > during migration.
> >     >     >         >> > 2. Given that bumping up magic value is almost
> > free and
> >     > has
> >     >     > benefit of
> >     >     >         >> > avoiding potential performance issue. It is
> > probably
> >     > worth
> >     >     > doing.
> >     >     >         >> >
> >     >     >         >> > One issue we still need to think about is
> whether
> > we
> >     > want to
> >     >     > support a
> >     >     >         >> > non-tombstone message with null value.
> >     >     >         >> > Currently it is not supported by Kafka. If we
> > allow a
> >     >     > non-tombstone null
> >     >     >         >> > value message to exist after KIP-87. The problem
> > is
> >     > that such
> >     >     > message
> >     >     >         >> will
> >     >     >         >> > not be supported by the consumers prior to
> KIP-87.
> >     > Because a
> >     >     > null value
> >     >     >         >> > will always be interpreted to a tombstone.
> >     >     >         >> >
> >     >     >         >> > One option is that we keep the current way, i.e.
> > do not
> >     >     > support such
> >     >     >         >> > message. It would be good to know if there is a
> >     > concrete use
> >     >     > case for
> >     >     >         >> such
> >     >     >         >> > message. If there is not, we can probably just
> not
> >     > support it.
> >     >     >         >> >
> >     >     >         >> > Thanks,
> >     >     >         >> >
> >     >     >         >> > JIangjie (Becket) Qin
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh
> Gharat <
> >     >     >         >> > gharatmayuresh15@gmail.com
> >     >     >         >> > > wrote:
> >     >     >         >> >
> >     >     >         >> > > Hi Ismael,
> >     >     >         >> > >
> >     >     >         >> > > This is something I can think of for migration
> > plan:
> >     >     >         >> > > So the migration plan can look something like
> > this,
> >     > with up
> >     >     >         >> conversion :
> >     >     >         >> > >
> >     >     >         >> > > 1) Currently lets say we have Broker at
> version
> > x.
> >     >     >         >> > > 2) Currently we have clients at version x.
> >     >     >         >> > > 3) a) We move the version to Broker(x+1) :
> > supports
> >     > both
> >     >     > tombstone and
> >     >     >         >> > null
> >     >     >         >> > > for log compaction.
> >     >     >         >> > >     b) We upgrade the client to version
> > client(x+1) :
> >     > if in
> >     >     > the
> >     >     >         >> producer
> >     >     >         >> > > client(x+1) the value is set to null, we will
> >     > automatically
> >     >     > set the
> >     >     >         >> > > Tombstone bit internally. If the producer
> > client(x+1)
> >     > sets
> >     >     > the
> >     >     >         >> tombstone
> >     >     >         >> > > itself, well and good. For producer client(x),
> > the
> >     > broker
> >     >     > will up
> >     >     >         >> convert
> >     >     >         >> > > to have the tombstone bit. Broker(x+1) is
> > supporting
> >     > both.
> >     >     > Consumer
> >     >     >         >> > > client(x+1) will be aware of this and should
> be
> > able
> >     > to
> >     >     > handle this.
> >     >     >         >> For
> >     >     >         >> > > consumer client(x) we will down convert the
> > message
> >     > on the
> >     >     > broker
> >     >     >         >> side.
> >     >     >         >> > >     c) At this point we will have to specify a
> >     > warning or
> >     >     > clearly
> >     >     >         >> specify
> >     >     >         >> > > in docs that this behavior is about to be
> > changed for
> >     > log
> >     >     > compaction.
> >     >     >         >> > > 4) a) In next release of the Broker(x+2), we
> > say that
> >     > only
> >     >     > Tombstone
> >     >     >         >> is
> >     >     >         >> > > used for log compaction on the Broker side.
> >     > Clients(x+1)
> >     >     > still is
> >     >     >         >> > > supported.
> >     >     >         >> > >     b) We upgrade the client to version
> > client(x+2) :
> >     > if
> >     >     > value is set
> >     >     >         >> to
> >     >     >         >> > > null, tombstone will not be set automatically.
> > The
> >     > client
> >     >     > will have to
> >     >     >         >> > call
> >     >     >         >> > > setTombstone() to actually set the tombstone.
> >     >     >         >> > >
> >     >     >         >> > > We should compare this migration plan with the
> >     > migration
> >     >     > plan for
> >     >     >         >> magic
> >     >     >         >> > > byte bump and do whatever looks good.
> >     >     >         >> > > I am just worried that if we go down magic
> byte
> > route,
> >     >     > unless I am
> >     >     >         >> > missing
> >     >     >         >> > > something, it sounds like kafka will be stuck
> > with
> >     >     > supporting both
> >     >     >         >> null
> >     >     >         >> > > value and tombstone bit for log compaction for
> > life
> >     > long,
> >     >     > which does
> >     >     >         >> not
> >     >     >         >> > > look like a good end state.
> >     >     >         >> > >
> >     >     >         >> > > Thanks,
> >     >     >         >> > >
> >     >     >         >> > > Mayuresh
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh
> > Gharat <
> >     >     >         >> > > gharatmayuresh15@gmail.com
> >     >     >         >> > > > wrote:
> >     >     >         >> > >
> >     >     >         >> > > > Hi Ismael,
> >     >     >         >> > > >
> >     >     >         >> > > > That's a very good point which I might have
> > not
> >     >     > considered earlier.
> >     >     >         >> > > >
> >     >     >         >> > > > Here is a plan that I can think of:
> >     >     >         >> > > >
> >     >     >         >> > > > Stage 1) The broker from now on, up converts
> > the
> >     > message
> >     >     > to have the
> >     >     >         >> > > > tombstone marker. The log compaction thread
> > does log
> >     >     > compaction
> >     >     >         >> based
> >     >     >         >> > on
> >     >     >         >> > > > both null and tombstone marker. This is our
> >     > transition
> >     >     > period.
> >     >     >         >> > > > Stage 2) The next release we only say that
> log
> >     > compaction
> >     >     > is based
> >     >     >         >> on
> >     >     >         >> > > > tombstone marker. (Open source kafka makes
> > this as a
> >     >     > policy). By
> >     >     >         >> this
> >     >     >         >> > > time,
> >     >     >         >> > > > the organization which is moving to this
> > release
> >     > will be
> >     >     > sure that
> >     >     >         >> they
> >     >     >         >> > > > have gone through the entire transition
> > period.
> >     >     >         >> > > >
> >     >     >         >> > > > My only goal of doing this is that Kafka
> > clearly
> >     >     > specifies the end
> >     >     >         >> > state
> >     >     >         >> > > > about what log compaction means (is it null
> > value
> >     > or a
> >     >     > tombstone
> >     >     >         >> > marker,
> >     >     >         >> > > > but not both).
> >     >     >         >> > > >
> >     >     >         >> > > > What do you think?
> >     >     >         >> > > >
> >     >     >         >> > > > Thanks,
> >     >     >         >> > > >
> >     >     >         >> > > > Mayuresh
> >     >     >         >> > > > .
> >     >     >         >> > > >
> >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael
> Juma <
> >     >     > ismael@juma.me.uk>
> >     >     >         >> > wrote:
> >     >     >         >> > > >
> >     >     >         >> > > >> One comment below.
> >     >     >         >> > > >>
> >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
> > Gharat <
> >     >     >         >> > > >> gharatmayuresh15@gmail.com
> >     >     >         >> > > >> > wrote:
> >     >     >         >> > > >>
> >     >     >         >> > > >> >    - If we don't bump up the magic byte,
> > on the
> >     > broker
> >     >     > side, the
> >     >     >         >> > > broker
> >     >     >         >> > > >> >    will always have to look at both
> > tombstone
> >     > bit and
> >     >     > the value
> >     >     >         >> when
> >     >     >         >> > > do
> >     >     >         >> > > >> the
> >     >     >         >> > > >> >    compaction. Assuming we do not bump up
> > the
> >     > magic
> >     >     > byte,
> >     >     >         >> > > >> >    imagine the broker sees a message
> which
> > does
> >     > not
> >     >     > have a
> >     >     >         >> tombstone
> >     >     >         >> > > bit
> >     >     >         >> > > >> >    set. The broker does not know when the
> >     > message was
> >     >     > produced
> >     >     >         >> (i.e.
> >     >     >         >> > > >> > whether
> >     >     >         >> > > >> >    the message has been up converted or
> > not), it
> >     > has
> >     >     > to take a
> >     >     >         >> > further
> >     >     >         >> > > >> > look at
> >     >     >         >> > > >> >    the value to see if it is null or not
> in
> >     > order to
> >     >     > determine
> >     >     >         >> if it
> >     >     >         >> > > is
> >     >     >         >> > > >> a
> >     >     >         >> > > >> >    tombstone. The same logic has to be
> put
> > on the
> >     >     > consumer as
> >     >     >         >> well
> >     >     >         >> > > >> because
> >     >     >         >> > > >> > the
> >     >     >         >> > > >> >    consumer does not know if the message
> > has
> >     > been up
> >     >     > converted or
> >     >     >         >> > not.
> >     >     >         >> > > >> >       - If we upconvert while appending,
> > this is
> >     > not
> >     >     > the case,
> >     >     >         >> > right?
> >     >     >         >> > > >>
> >     >     >         >> > > >>
> >     >     >         >> > > >> If I understand you correctly, this is not
> >     > sufficient
> >     >     > because the
> >     >     >         >> log
> >     >     >         >> > > may
> >     >     >         >> > > >> have messages appended before it was
> > upgraded to
> >     > include
> >     >     > KIP-87.
> >     >     >         >> > > >>
> >     >     >         >> > > >> Ismael
> >     >     >         >> > > >>
> >     >     >         >> > > >
> >     >     >         >> > > >
> >     >     >         >> > > >
> >     >     >         >> > > > --
> >     >     >         >> > > > -Regards,
> >     >     >         >> > > > Mayuresh R. Gharat
> >     >     >         >> > > > (862) 250-7125
> >     >     >         >> > > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > > --
> >     >     >         >> > > -Regards,
> >     >     >         >> > > Mayuresh R. Gharat
> >     >     >         >> > > (862) 250-7125
> >     >     >         >> > >
> >     >     >         >> > The information contained in this email is
> > strictly
> >     >     > confidential and for
> >     >     >         >> > the use of the addressee only, unless otherwise
> >     > indicated. If
> >     >     > you are
> >     >     >         >> not
> >     >     >         >> > the intended recipient, please do not read,
> copy,
> > use or
> >     >     > disclose to
> >     >     >         >> others
> >     >     >         >> > this message or any attachment. Please also
> > notify the
> >     > sender
> >     >     > by
> >     >     >         >> replying
> >     >     >         >> > to this email or by telephone (+44(020 7896
> 0011)
> > and
> >     > then
> >     >     > delete the
> >     >     >         >> email
> >     >     >         >> > and any copies of it. Opinions, conclusion (etc)
> > that
> >     > do not
> >     >     > relate to
> >     >     >         >> the
> >     >     >         >> > official business of this company shall be
> > understood as
> >     >     > neither given
> >     >     >         >> nor
> >     >     >         >> > endorsed by it. IG is a trading name of IG
> Markets
> >     > Limited (a
> >     >     > company
> >     >     >         >> > registered in England and Wales, company number
> >     > 04008957) and
> >     >     > IG Index
> >     >     >         >> > Limited (a company registered in England and
> > Wales,
> >     > company
> >     >     > number
> >     >     >         >> > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     >     > Dowgate Hill,
> >     >     >         >> > London EC4R 2YA. Both IG Markets Limited
> (register
> >     > number
> >     >     > 195355) and IG
> >     >     >         >> > Index Limited (register number 114059) are
> > authorised
> >     > and
> >     >     > regulated by
> >     >     >         >> the
> >     >     >         >> > Financial Conduct Authority.
> >     >     >         >> >
> >     >     >         >>
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> --
> >     >     >         >> -Regards,
> >     >     >         >> Mayuresh R. Gharat
> >     >     >         >> (862) 250-7125
> >     >     >         >> The information contained in this email is
> strictly
> >     >     > confidential and for
> >     >     >         >> the use of the addressee only, unless otherwise
> >     > indicated. If
> >     >     > you are not
> >     >     >         >> the intended recipient, please do not read, copy,
> > use or
> >     >     > disclose to others
> >     >     >         >> this message or any attachment. Please also notify
> > the
> >     > sender
> >     >     > by replying
> >     >     >         >> to this email or by telephone (+44(020 7896 0011)
> > and then
> >     >     > delete the email
> >     >     >         >> and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     >     > relate to the
> >     >     >         >> official business of this company shall be
> > understood as
> >     >     > neither given nor
> >     >     >         >> endorsed by it. IG is a trading name of IG Markets
> >     > Limited (a
> >     >     > company
> >     >     >         >> registered in England and Wales, company number
> > 04008957)
> >     > and
> >     >     > IG Index
> >     >     >         >> Limited (a company registered in England and
> Wales,
> >     > company
> >     >     > number
> >     >     >         >> 01190902). Registered address at Cannon Bridge
> > House, 25
> >     >     > Dowgate Hill,
> >     >     >         >> London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     >     > 195355) and IG
> >     >     >         >> Index Limited (register number 114059) are
> > authorised and
> >     >     > regulated by the
> >     >     >         >> Financial Conduct Authority.
> >     >     >         >>
> >     >     >         >
> >     >     >         >
> >     >     >         >
> >     >     >         > --
> >     >     >         > -Regards,
> >     >     >         > Mayuresh R. Gharat
> >     >     >         > (862) 250-7125
> >     >     >         >
> >     >     >
> >     >     >
> >     >     >
> >     >     >         --
> >     >     >         -Regards,
> >     >     >         Mayuresh R. Gharat
> >     >     >         (862) 250-7125
> >     >     >
> >     >     >
> >     >     >     The information contained in this email is strictly
> > confidential
> >     > and
> >     >     > for the use of the addressee only, unless otherwise
> indicated.
> > If
> >     > you are
> >     >     > not the intended recipient, please do not read, copy, use or
> >     > disclose to
> >     >     > others this message or any attachment. Please also notify the
> > sender
> >     > by
> >     >     > replying to this email or by telephone (+44(020 7896 0011)
> and
> > then
> >     > delete
> >     >     > the email and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     >     > relate to the official business of this company shall be
> > understood
> >     > as
> >     >     > neither given nor endorsed by it. IG is a trading name of IG
> > Markets
> >     >     > Limited (a company registered in England and Wales, company
> > number
> >     >     > 04008957) and IG Index Limited (a company registered in
> > England and
> >     > Wales,
> >     >     > company number 01190902). Registered address at Cannon Bridge
> > House,
> >     > 25
> >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > number
> >     >     > 195355) and IG Index Limited (register number 114059) are
> > authorised
> >     > and
> >     >     > regulated by the Financial Conduct Authority.
> >     >     >
> >     >     >
> >     >     >
> >     >
> >     >
> >     > The information contained in this email is strictly confidential
> and
> > for
> >     > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> >     > this message or any attachment. Please also notify the sender by
> > replying
> >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > the email
> >     > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to the
> >     > official business of this company shall be understood as neither
> > given nor
> >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >     > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > Limited (a company registered in England and Wales, company number
> >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > Index Limited (register number 114059) are authorised and regulated
> > by the
> >     > Financial Conduct Authority.
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jun Rao <ju...@confluent.io>]

Hi, Michael,

4. Hmm, does that mean the new client library can never send a null message
even to a regular topic? This seems like a change of the existing behavior.

Thanks,

Jun

On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jun,
>
> Re 4) That's because we expect the tombstone value to be set correctly if
> message bit is 2, as such if an older client sends in on old message the
> message is upcast and the bit is set correctly. And such no longer need to
> check the value. Mayuresh can you confirm my thinking and understanding of
> what we've discussed?
>
> The second point I understand what you're getting at now my apologies. Yes
> this makes sense to save on touching the message, if we're the only kip
> going in, in this release.
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jun Rao <ju...@confluent.io>
> Sent: Tuesday, December 6, 2016 5:22:13 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hi, Michael,
>
> 4. Is this updated in the wiki? The text "If the magic byte on message is
> 2, the broker should use the tombstone bit for log compaction." doesn't
> seem to have changed.
>
> 2. My point is that if we change the message format just for this KIP, we
> should consider whether it's worth optimizing the down conversion path
> (i.e., decide whether a conversion is needed by just looking at the
> tombstone bit in the wrapper message) since tombstone will be used rarely.
> However, if the message format change here is combined with other KIPs,
> then this optimization likely won't be needed. The latter probably makes
> the code simpler. Jiangjie, Mayuresh, what do you think?
>
> Other than those, +1 from me,
>
> Thanks,
>
> Jun
>
> On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jun
> >
> > do we have your vote on this now?
> >
> > Any other concerns?
> >
> > Cheers
> > Mike
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Michael Pearce <Mi...@ig.com>
> > Sent: Saturday, December 3, 2016 1:37:45 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hi Jun,
> >
> > Have updated. Thanks again for the feedback.
> >
> > Agree yes we should align up when it gets to that, I assume you’ve
> flagged
> > the same in the other KIP?
> >
> > I think for now let’s get this KIP approved, then we can start the work
> to
> > get an initial PR, then we can discuss how to align the two if needed.
> >
> > Cheers,
> > Mike
> >
> > On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
> >
> >     Hi, Michael,
> >
> >     For 2), this is fine. Could you update the KIP wiki to make this and
> > other
> >     points clearer? Other than that, the KIP looks good to me.
> >
> >     An orthogonal thing is that there are other KIPs such as KIP-98 that
> > also
> >     intend to change the message format. If they all get approved, we
> > should
> >     think about whether it's better to just bump up the magic byte once
> to
> >     incorporate multiple format changes like we did in KIP-31/KIP-32.
> >
> >     Thanks,
> >
> >     Jun
> >
> >
> >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> >     wrote:
> >
> >     > Hi Jao
> >     >
> >     > Thanks for the response. Sorry for slow reply, both with personal
> > sickness
> >     > and also battling some critical issues encountered since upgrading
> to
> >     > 0.10.1.0
> >     >
> >     > 1) Thans for spotting, Document error where we branched this KIP
> from
> >     > KIP-82, will get that removed.
> >     > 2) Intent is to do this just at the record message level.
> >     > 3) Thanks for spotting, Will ensure this is corrected.
> >     > 4) As per discussion thread we will support tombstone + null value,
> >     > tombstone + non null value, no tombstone + null value.
> >     > 5) I believe this was in the discussion thread, @Mayuresh is this
> >     > something we’ve overlooked? I thought we would down convert and
> > remove the
> >     > value so the old consumer had existing behavior, or is there
> > something we
> >     > haven’t thought about?
> >     >
> >     > Cheers
> >     > Mike
> >     >
> >     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
> >     >
> >     >     Hi, Michael,
> >     >
> >     >     Thanks for the KIP. A few comments below.
> >     >
> >     >     1. The message format change contains "HeadersLength Headers".
> > Is that
> >     >     intended?
> >     >
> >     >     2. For compressed messageset, is the tombstone bit only set at
> > the
> >     > shallow
> >     >     level? Do we always leave that bit in the wrapper message
> unset?
> > An
> >     >     alternative is to set the tombstone bit in the wrapper if at
> > least one
> >     >     inner message has the tombstone bit set. This makes things a
> bit
> > more
> >     >     complicated, but we could potentially exploit that for
> > optimizing down
> >     >     conversion. For example, we only need to convert messages with
> > magic 2
> >     > to
> >     >     magic 1 if the wrapper's tombstone bit is set (conversion is
> > always
> >     > needed
> >     >     from magic 2 to magic 0). Not sure if the optimization is worth
> > the
> >     >     complexity though.
> >     >
> >     >     3. The referencing of the new version of
> > ProducerRequest/FetchRequest
> >     > is
> >     >     inconsistent (v4 vs v3). Since our convention starts at version
> > at 0, I
> >     >     think the new version would be 3.
> >     >
> >     >     4. "If the magic byte on message is 2, the broker should use
> the
> >     > tombstone
> >     >     bit for log compaction." What about null value? My
> understanding
> > is
> >     > that
> >     >     null value will be treated the same as setting the tombstone
> bit.
> >     >
> >     >     5. For the migration path, it would be useful to describe the
> > down
> >     >     conversion path to consumers (i.e., brokers on message format
> > 0.10.2
> >     > and
> >     >     consumers on older version).
> >     >
> >     >     Thanks,
> >     >
> >     >     Jun
> >     >
> >     >
> >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> >     > >
> >     >     wrote:
> >     >
> >     >     > Hi All,
> >     >     >
> >     >     > We have been discussing in the below thread and final changes
> > have
> >     > been
> >     >     > made to the KIP wiki based on these discussions.
> >     >     >
> >     >     > We would now like to put to the vote the following KIP:
> >     >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
> >     >     > Add+Compaction+Tombstone+Flag
> >     >     >
> >     >     > This kip is for having a distinct compaction attribute
> > “tombstone”
> >     > flag
> >     >     > instead of relying on null value, allowing non-null value
> > delete
> >     > messages.
> >     >     >
> >     >     > Many thanks,
> >     >     > Michael
> >     >     >
> >     >     >
> >     >     >
> >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> Michael.Pearce@ig.com>
> >     > wrote:
> >     >     >
> >     >     >     Hi Mayuresh,
> >     >     >
> >     >     >     LGTM. Ive just made one small adjustment updating the
> wire
> >     > protocol to
> >     >     > show the magic byte bump.
> >     >     >
> >     >     >     Do we think we’re good to put to a vote? Is there any
> > other bits
> >     >     > needing discussion?
> >     >     >
> >     >     >     Cheers
> >     >     >     Mike
> >     >     >
> >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> >     > gharatmayuresh15@gmail.com>
> >     >     > wrote:
> >     >     >
> >     >     >         Hi Michael,
> >     >     >
> >     >     >         I have updated the migration section of the KIP. Can
> > you
> >     > please
> >     >     > take a look?
> >     >     >
> >     >     >         Thanks,
> >     >     >
> >     >     >         Mayuresh
> >     >     >
> >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> >     >     > gharatmayuresh15@gmail.com
> >     >     >         > wrote:
> >     >     >
> >     >     >         > Hi Michael,
> >     >     >         >
> >     >     >         > That whilst sending tombstone and non null value,
> the
> >     > consumer
> >     >     > can expect
> >     >     >         > only to receive the non-null message only in step
> > (3) is
> >     > this
> >     >     > correct?
> >     >     >         > ---> I do agree with you here.
> >     >     >         >
> >     >     >         > Becket, Ismael : can you guys review the migration
> > plan
> >     > listed
> >     >     > above using
> >     >     >         > magic byte?
> >     >     >         >
> >     >     >         > Thanks,
> >     >     >         >
> >     >     >         > Mayuresh
> >     >     >         >
> >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
> >     >     > Michael.Pearce@ig.com>
> >     >     >         > wrote:
> >     >     >         >
> >     >     >         >> Many thanks for this Mayuresh. I don't have any
> >     > objections.
> >     >     >         >>
> >     >     >         >> I assume we should state:
> >     >     >         >>
> >     >     >         >> That whilst sending tombstone and non null value,
> > the
> >     > consumer
> >     >     > can expect
> >     >     >         >> only to receive the non-null message only in step
> > (3) is
> >     > this
> >     >     > correct?
> >     >     >         >>
> >     >     >         >> Cheers
> >     >     >         >> Mike
> >     >     >         >>
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> Sent using OWA for iPhone
> >     >     >         >> ________________________________________
> >     >     >         >> From: Mayuresh Gharat <gharatmayuresh15@gmail.com
> >
> >     >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
> >     >     >         >> To: dev@kafka.apache.org
> >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> > Tombstone
> >     > Flag
> >     >     >         >>
> >     >     >         >> Hi Ismael,
> >     >     >         >>
> >     >     >         >> Thanks for the explanation.
> >     >     >         >> Specially I like this part where in you mentioned
> > we can
> >     > get
> >     >     > rid of the
> >     >     >         >> older null value support for log compaction later
> > on,
> >     > here :
> >     >     >         >> We can't change semantics of the message format
> > without
> >     > having
> >     >     > a long
> >     >     >         >> transition period. And we can't rely
> >     >     >         >> on people reading documentation or acting on a
> > warning for
> >     >     > something so
> >     >     >         >> fundamental. As such, my take is that we need to
> > bump the
> >     > magic
> >     >     > byte. The
> >     >     >         >> good news is
> >     >     >         >> that we don't have to support all versions
> forever.
> > We
> >     > have
> >     >     > said that we
> >     >     >         >> will support direct upgrades for 2 years. That
> > means that
> >     >     > message format
> >     >     >         >> version n could, in theory, be removed 2 years
> > after the
> >     > it's
> >     >     > introduced.
> >     >     >         >>
> >     >     >         >> Just a heads up, I would like to mention that even
> > without
> >     >     > bumping magic
> >     >     >         >> byte, we will *NOT* loose zero copy as in the
> > client(x+1)
> >     > in my
> >     >     >         >> explanation
> >     >     >         >> above will convert internally a null value to
> have a
> >     > tombstone
> >     >     > bit set and
> >     >     >         >> a tombstone bit set to have a null value
> > automatically
> >     >     > internally and by
> >     >     >         >> the time we move to version (x+2), the clients
> > would have
> >     >     > upgraded.
> >     >     >         >> Obviously if we support a request from
> consumer(x),
> > we
> >     > will
> >     >     > loose zero
> >     >     >         >> copy
> >     >     >         >> but that is the same case with magic byte.
> >     >     >         >>
> >     >     >         >> But if magic byte bump makes life easier for
> > transition
> >     > for the
> >     >     > above
> >     >     >         >> reasons that you explained, I am OK with it since
> > we are
> >     > going
> >     >     > to meet the
> >     >     >         >> end goal down the road :)
> >     >     >         >>
> >     >     >         >> On a side note can we update the doc here on magic
> > byte
> >     > to say
> >     >     > that "*it
> >     >     >         >> should be bumped whenever the message format is
> > changed
> >     > or the
> >     >     >         >> interpretation of message format (usage of the
> > reserved
> >     > bits as
> >     >     > well) is
> >     >     >         >> changed*".
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> Hi Michael,
> >     >     >         >>
> >     >     >         >> Here is the update plan that we discussed offline
> >     > yesterday :
> >     >     >         >>
> >     >     >         >> Currently the magic-byte which corresponds to the
> >     >     > "message.format.version"
> >     >     >         >> is set to 1.
> >     >     >         >>
> >     >     >         >> 1) On broker it will be set to 1 initially.
> >     >     >         >>
> >     >     >         >> 2) When a producer client sends a message with
> > magic-byte
> >     > = 2,
> >     >     > since the
> >     >     >         >> broker is on magic-byte = 1, we will down convert
> > it,
> >     > which
> >     >     > means if the
> >     >     >         >> tombstone bit is set, the value will be set to
> > null. A
> >     > consumer
> >     >     >         >> understanding magic-byte = 1, will still work with
> > this. A
> >     >     > consumer
> >     >     >         >> working
> >     >     >         >> with magic-byte =2 will also be able to understand
> > this,
> >     > since
> >     >     > it
> >     >     >         >> understands the tombstone.
> >     >     >         >> Now there is still the question of supporting a
> >     > non-tombstone
> >     >     > and null
> >     >     >         >> value from producer client with magic-byte = 2.*
> (I
> > am
> >     > not sure
> >     >     > if we
> >     >     >         >> should support this. Ismael/Becket can comment
> > here)*
> >     >     >         >>
> >     >     >         >> 3) When almost all the clients have upgraded, the
> >     >     > message.format.version
> >     >     >         >> on
> >     >     >         >> the broker can be changed to 2, where in the down
> >     > conversion in
> >     >     > the above
> >     >     >         >> step will not happen. If at this point we get a
> > consumer
> >     >     > request from a
> >     >     >         >> older consumer, we might have to down convert
> where
> > in we
> >     > loose
> >     >     > zero copy,
> >     >     >         >> but these cases should be rare.
> >     >     >         >>
> >     >     >         >> Becket can you review this plan and add more
> > details if I
> >     > have
> >     >     >         >> missed/wronged something, before we put it on KIP.
> >     >     >         >>
> >     >     >         >> Thanks,
> >     >     >         >>
> >     >     >         >> Mayuresh
> >     >     >         >>
> >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
> >     >     > Michael.Pearce@ig.com>
> >     >     >         >> wrote:
> >     >     >         >>
> >     >     >         >> > Thanks guys, for discussing this offline and
> > getting
> >     > some
> >     >     > consensus.
> >     >     >         >> >
> >     >     >         >> > So its clear for myself and others what is
> > proposed now
> >     > (i
> >     >     > think i
> >     >     >         >> > understand, but want to make sure)
> >     >     >         >> >
> >     >     >         >> > Could i ask either directly update the kip to
> > detail the
> >     >     > migration
> >     >     >         >> > strategy, or (re-)state your offline discussed
> and
> >     > agreed
> >     >     > migration
> >     >     >         >> > strategy based on a magic byte is in this
> thread.
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > The main original driver for the KIP was to
> > support
> >     >     > compaction where
> >     >     >         >> value
> >     >     >         >> > isn't null, based off the discussions on KIP-82
> > thread.
> >     >     >         >> >
> >     >     >         >> > We should be able to support non-tombstone +
> null
> > value
> >     > by the
> >     >     >         >> completion
> >     >     >         >> > of the KIP, as we noted when discussing this
> kip,
> > having
> >     >     > logic based on
> >     >     >         >> a
> >     >     >         >> > null value isn't very clean and also separates
> the
> >     > concerns.
> >     >     >         >> >
> >     >     >         >> > As discussed already though we can split this
> into
> >     > KIP-87a
> >     >     > and KIP-87b
> >     >     >         >> >
> >     >     >         >> > Where we look to deliver KIP-87a on a compacted
> > topic
> >     > (to
> >     >     > address the
> >     >     >         >> > immediate issues)
> >     >     >         >> > * tombstone + null value
> >     >     >         >> > * tombstone + non-null value
> >     >     >         >> > * non-tombstone + non-null value
> >     >     >         >> >
> >     >     >         >> > Then we can discuss once KIP-87a is completed
> > options
> >     > later
> >     >     > and how we
> >     >     >         >> > support the second part KIP-87b to deliver:
> >     >     >         >> > * non-tombstone + null value
> >     >     >         >> >
> >     >     >         >> > Cheers
> >     >     >         >> > Mike
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > ________________________________________
> >     >     >         >> > From: Becket Qin <be...@gmail.com>
> >     >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
> >     >     >         >> > To: dev@kafka.apache.org
> >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> >     > Tombstone Flag
> >     >     >         >> >
> >     >     >         >> > Renu, Mayuresh and I had an offline discussion,
> > and
> >     > following
> >     >     > is a brief
> >     >     >         >> > summary.
> >     >     >         >> >
> >     >     >         >> > 1. We agreed that not bumping up magic value may
> > result
> >     > in
> >     >     > losing zero
> >     >     >         >> copy
> >     >     >         >> > during migration.
> >     >     >         >> > 2. Given that bumping up magic value is almost
> > free and
> >     > has
> >     >     > benefit of
> >     >     >         >> > avoiding potential performance issue. It is
> > probably
> >     > worth
> >     >     > doing.
> >     >     >         >> >
> >     >     >         >> > One issue we still need to think about is
> whether
> > we
> >     > want to
> >     >     > support a
> >     >     >         >> > non-tombstone message with null value.
> >     >     >         >> > Currently it is not supported by Kafka. If we
> > allow a
> >     >     > non-tombstone null
> >     >     >         >> > value message to exist after KIP-87. The problem
> > is
> >     > that such
> >     >     > message
> >     >     >         >> will
> >     >     >         >> > not be supported by the consumers prior to
> KIP-87.
> >     > Because a
> >     >     > null value
> >     >     >         >> > will always be interpreted to a tombstone.
> >     >     >         >> >
> >     >     >         >> > One option is that we keep the current way, i.e.
> > do not
> >     >     > support such
> >     >     >         >> > message. It would be good to know if there is a
> >     > concrete use
> >     >     > case for
> >     >     >         >> such
> >     >     >         >> > message. If there is not, we can probably just
> not
> >     > support it.
> >     >     >         >> >
> >     >     >         >> > Thanks,
> >     >     >         >> >
> >     >     >         >> > JIangjie (Becket) Qin
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh
> Gharat <
> >     >     >         >> > gharatmayuresh15@gmail.com
> >     >     >         >> > > wrote:
> >     >     >         >> >
> >     >     >         >> > > Hi Ismael,
> >     >     >         >> > >
> >     >     >         >> > > This is something I can think of for migration
> > plan:
> >     >     >         >> > > So the migration plan can look something like
> > this,
> >     > with up
> >     >     >         >> conversion :
> >     >     >         >> > >
> >     >     >         >> > > 1) Currently lets say we have Broker at
> version
> > x.
> >     >     >         >> > > 2) Currently we have clients at version x.
> >     >     >         >> > > 3) a) We move the version to Broker(x+1) :
> > supports
> >     > both
> >     >     > tombstone and
> >     >     >         >> > null
> >     >     >         >> > > for log compaction.
> >     >     >         >> > >     b) We upgrade the client to version
> > client(x+1) :
> >     > if in
> >     >     > the
> >     >     >         >> producer
> >     >     >         >> > > client(x+1) the value is set to null, we will
> >     > automatically
> >     >     > set the
> >     >     >         >> > > Tombstone bit internally. If the producer
> > client(x+1)
> >     > sets
> >     >     > the
> >     >     >         >> tombstone
> >     >     >         >> > > itself, well and good. For producer client(x),
> > the
> >     > broker
> >     >     > will up
> >     >     >         >> convert
> >     >     >         >> > > to have the tombstone bit. Broker(x+1) is
> > supporting
> >     > both.
> >     >     > Consumer
> >     >     >         >> > > client(x+1) will be aware of this and should
> be
> > able
> >     > to
> >     >     > handle this.
> >     >     >         >> For
> >     >     >         >> > > consumer client(x) we will down convert the
> > message
> >     > on the
> >     >     > broker
> >     >     >         >> side.
> >     >     >         >> > >     c) At this point we will have to specify a
> >     > warning or
> >     >     > clearly
> >     >     >         >> specify
> >     >     >         >> > > in docs that this behavior is about to be
> > changed for
> >     > log
> >     >     > compaction.
> >     >     >         >> > > 4) a) In next release of the Broker(x+2), we
> > say that
> >     > only
> >     >     > Tombstone
> >     >     >         >> is
> >     >     >         >> > > used for log compaction on the Broker side.
> >     > Clients(x+1)
> >     >     > still is
> >     >     >         >> > > supported.
> >     >     >         >> > >     b) We upgrade the client to version
> > client(x+2) :
> >     > if
> >     >     > value is set
> >     >     >         >> to
> >     >     >         >> > > null, tombstone will not be set automatically.
> > The
> >     > client
> >     >     > will have to
> >     >     >         >> > call
> >     >     >         >> > > setTombstone() to actually set the tombstone.
> >     >     >         >> > >
> >     >     >         >> > > We should compare this migration plan with the
> >     > migration
> >     >     > plan for
> >     >     >         >> magic
> >     >     >         >> > > byte bump and do whatever looks good.
> >     >     >         >> > > I am just worried that if we go down magic
> byte
> > route,
> >     >     > unless I am
> >     >     >         >> > missing
> >     >     >         >> > > something, it sounds like kafka will be stuck
> > with
> >     >     > supporting both
> >     >     >         >> null
> >     >     >         >> > > value and tombstone bit for log compaction for
> > life
> >     > long,
> >     >     > which does
> >     >     >         >> not
> >     >     >         >> > > look like a good end state.
> >     >     >         >> > >
> >     >     >         >> > > Thanks,
> >     >     >         >> > >
> >     >     >         >> > > Mayuresh
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh
> > Gharat <
> >     >     >         >> > > gharatmayuresh15@gmail.com
> >     >     >         >> > > > wrote:
> >     >     >         >> > >
> >     >     >         >> > > > Hi Ismael,
> >     >     >         >> > > >
> >     >     >         >> > > > That's a very good point which I might have
> > not
> >     >     > considered earlier.
> >     >     >         >> > > >
> >     >     >         >> > > > Here is a plan that I can think of:
> >     >     >         >> > > >
> >     >     >         >> > > > Stage 1) The broker from now on, up converts
> > the
> >     > message
> >     >     > to have the
> >     >     >         >> > > > tombstone marker. The log compaction thread
> > does log
> >     >     > compaction
> >     >     >         >> based
> >     >     >         >> > on
> >     >     >         >> > > > both null and tombstone marker. This is our
> >     > transition
> >     >     > period.
> >     >     >         >> > > > Stage 2) The next release we only say that
> log
> >     > compaction
> >     >     > is based
> >     >     >         >> on
> >     >     >         >> > > > tombstone marker. (Open source kafka makes
> > this as a
> >     >     > policy). By
> >     >     >         >> this
> >     >     >         >> > > time,
> >     >     >         >> > > > the organization which is moving to this
> > release
> >     > will be
> >     >     > sure that
> >     >     >         >> they
> >     >     >         >> > > > have gone through the entire transition
> > period.
> >     >     >         >> > > >
> >     >     >         >> > > > My only goal of doing this is that Kafka
> > clearly
> >     >     > specifies the end
> >     >     >         >> > state
> >     >     >         >> > > > about what log compaction means (is it null
> > value
> >     > or a
> >     >     > tombstone
> >     >     >         >> > marker,
> >     >     >         >> > > > but not both).
> >     >     >         >> > > >
> >     >     >         >> > > > What do you think?
> >     >     >         >> > > >
> >     >     >         >> > > > Thanks,
> >     >     >         >> > > >
> >     >     >         >> > > > Mayuresh
> >     >     >         >> > > > .
> >     >     >         >> > > >
> >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael
> Juma <
> >     >     > ismael@juma.me.uk>
> >     >     >         >> > wrote:
> >     >     >         >> > > >
> >     >     >         >> > > >> One comment below.
> >     >     >         >> > > >>
> >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
> > Gharat <
> >     >     >         >> > > >> gharatmayuresh15@gmail.com
> >     >     >         >> > > >> > wrote:
> >     >     >         >> > > >>
> >     >     >         >> > > >> >    - If we don't bump up the magic byte,
> > on the
> >     > broker
> >     >     > side, the
> >     >     >         >> > > broker
> >     >     >         >> > > >> >    will always have to look at both
> > tombstone
> >     > bit and
> >     >     > the value
> >     >     >         >> when
> >     >     >         >> > > do
> >     >     >         >> > > >> the
> >     >     >         >> > > >> >    compaction. Assuming we do not bump up
> > the
> >     > magic
> >     >     > byte,
> >     >     >         >> > > >> >    imagine the broker sees a message
> which
> > does
> >     > not
> >     >     > have a
> >     >     >         >> tombstone
> >     >     >         >> > > bit
> >     >     >         >> > > >> >    set. The broker does not know when the
> >     > message was
> >     >     > produced
> >     >     >         >> (i.e.
> >     >     >         >> > > >> > whether
> >     >     >         >> > > >> >    the message has been up converted or
> > not), it
> >     > has
> >     >     > to take a
> >     >     >         >> > further
> >     >     >         >> > > >> > look at
> >     >     >         >> > > >> >    the value to see if it is null or not
> in
> >     > order to
> >     >     > determine
> >     >     >         >> if it
> >     >     >         >> > > is
> >     >     >         >> > > >> a
> >     >     >         >> > > >> >    tombstone. The same logic has to be
> put
> > on the
> >     >     > consumer as
> >     >     >         >> well
> >     >     >         >> > > >> because
> >     >     >         >> > > >> > the
> >     >     >         >> > > >> >    consumer does not know if the message
> > has
> >     > been up
> >     >     > converted or
> >     >     >         >> > not.
> >     >     >         >> > > >> >       - If we upconvert while appending,
> > this is
> >     > not
> >     >     > the case,
> >     >     >         >> > right?
> >     >     >         >> > > >>
> >     >     >         >> > > >>
> >     >     >         >> > > >> If I understand you correctly, this is not
> >     > sufficient
> >     >     > because the
> >     >     >         >> log
> >     >     >         >> > > may
> >     >     >         >> > > >> have messages appended before it was
> > upgraded to
> >     > include
> >     >     > KIP-87.
> >     >     >         >> > > >>
> >     >     >         >> > > >> Ismael
> >     >     >         >> > > >>
> >     >     >         >> > > >
> >     >     >         >> > > >
> >     >     >         >> > > >
> >     >     >         >> > > > --
> >     >     >         >> > > > -Regards,
> >     >     >         >> > > > Mayuresh R. Gharat
> >     >     >         >> > > > (862) 250-7125
> >     >     >         >> > > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > > --
> >     >     >         >> > > -Regards,
> >     >     >         >> > > Mayuresh R. Gharat
> >     >     >         >> > > (862) 250-7125
> >     >     >         >> > >
> >     >     >         >> > The information contained in this email is
> > strictly
> >     >     > confidential and for
> >     >     >         >> > the use of the addressee only, unless otherwise
> >     > indicated. If
> >     >     > you are
> >     >     >         >> not
> >     >     >         >> > the intended recipient, please do not read,
> copy,
> > use or
> >     >     > disclose to
> >     >     >         >> others
> >     >     >         >> > this message or any attachment. Please also
> > notify the
> >     > sender
> >     >     > by
> >     >     >         >> replying
> >     >     >         >> > to this email or by telephone (+44(020 7896
> 0011)
> > and
> >     > then
> >     >     > delete the
> >     >     >         >> email
> >     >     >         >> > and any copies of it. Opinions, conclusion (etc)
> > that
> >     > do not
> >     >     > relate to
> >     >     >         >> the
> >     >     >         >> > official business of this company shall be
> > understood as
> >     >     > neither given
> >     >     >         >> nor
> >     >     >         >> > endorsed by it. IG is a trading name of IG
> Markets
> >     > Limited (a
> >     >     > company
> >     >     >         >> > registered in England and Wales, company number
> >     > 04008957) and
> >     >     > IG Index
> >     >     >         >> > Limited (a company registered in England and
> > Wales,
> >     > company
> >     >     > number
> >     >     >         >> > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     >     > Dowgate Hill,
> >     >     >         >> > London EC4R 2YA. Both IG Markets Limited
> (register
> >     > number
> >     >     > 195355) and IG
> >     >     >         >> > Index Limited (register number 114059) are
> > authorised
> >     > and
> >     >     > regulated by
> >     >     >         >> the
> >     >     >         >> > Financial Conduct Authority.
> >     >     >         >> >
> >     >     >         >>
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> --
> >     >     >         >> -Regards,
> >     >     >         >> Mayuresh R. Gharat
> >     >     >         >> (862) 250-7125
> >     >     >         >> The information contained in this email is
> strictly
> >     >     > confidential and for
> >     >     >         >> the use of the addressee only, unless otherwise
> >     > indicated. If
> >     >     > you are not
> >     >     >         >> the intended recipient, please do not read, copy,
> > use or
> >     >     > disclose to others
> >     >     >         >> this message or any attachment. Please also notify
> > the
> >     > sender
> >     >     > by replying
> >     >     >         >> to this email or by telephone (+44(020 7896 0011)
> > and then
> >     >     > delete the email
> >     >     >         >> and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     >     > relate to the
> >     >     >         >> official business of this company shall be
> > understood as
> >     >     > neither given nor
> >     >     >         >> endorsed by it. IG is a trading name of IG Markets
> >     > Limited (a
> >     >     > company
> >     >     >         >> registered in England and Wales, company number
> > 04008957)
> >     > and
> >     >     > IG Index
> >     >     >         >> Limited (a company registered in England and
> Wales,
> >     > company
> >     >     > number
> >     >     >         >> 01190902). Registered address at Cannon Bridge
> > House, 25
> >     >     > Dowgate Hill,
> >     >     >         >> London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     >     > 195355) and IG
> >     >     >         >> Index Limited (register number 114059) are
> > authorised and
> >     >     > regulated by the
> >     >     >         >> Financial Conduct Authority.
> >     >     >         >>
> >     >     >         >
> >     >     >         >
> >     >     >         >
> >     >     >         > --
> >     >     >         > -Regards,
> >     >     >         > Mayuresh R. Gharat
> >     >     >         > (862) 250-7125
> >     >     >         >
> >     >     >
> >     >     >
> >     >     >
> >     >     >         --
> >     >     >         -Regards,
> >     >     >         Mayuresh R. Gharat
> >     >     >         (862) 250-7125
> >     >     >
> >     >     >
> >     >     >     The information contained in this email is strictly
> > confidential
> >     > and
> >     >     > for the use of the addressee only, unless otherwise
> indicated.
> > If
> >     > you are
> >     >     > not the intended recipient, please do not read, copy, use or
> >     > disclose to
> >     >     > others this message or any attachment. Please also notify the
> > sender
> >     > by
> >     >     > replying to this email or by telephone (+44(020 7896 0011)
> and
> > then
> >     > delete
> >     >     > the email and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     >     > relate to the official business of this company shall be
> > understood
> >     > as
> >     >     > neither given nor endorsed by it. IG is a trading name of IG
> > Markets
> >     >     > Limited (a company registered in England and Wales, company
> > number
> >     >     > 04008957) and IG Index Limited (a company registered in
> > England and
> >     > Wales,
> >     >     > company number 01190902). Registered address at Cannon Bridge
> > House,
> >     > 25
> >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > number
> >     >     > 195355) and IG Index Limited (register number 114059) are
> > authorised
> >     > and
> >     >     > regulated by the Financial Conduct Authority.
> >     >     >
> >     >     >
> >     >     >
> >     >
> >     >
> >     > The information contained in this email is strictly confidential
> and
> > for
> >     > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> >     > this message or any attachment. Please also notify the sender by
> > replying
> >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > the email
> >     > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to the
> >     > official business of this company shall be understood as neither
> > given nor
> >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >     > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > Limited (a company registered in England and Wales, company number
> >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > Index Limited (register number 114059) are authorised and regulated
> > by the
> >     > Financial Conduct Authority.
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jun,

Re 4) That's because we expect the tombstone value to be set correctly if message bit is 2, as such if an older client sends in on old message the message is upcast and the bit is set correctly. And such no longer need to check the value. Mayuresh can you confirm my thinking and understanding of what we've discussed?

The second point I understand what you're getting at now my apologies. Yes this makes sense to save on touching the message, if we're the only kip going in, in this release.

Cheers
Mike

Sent using OWA for iPhone
________________________________________
From: Jun Rao <ju...@confluent.io>
Sent: Tuesday, December 6, 2016 5:22:13 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi, Michael,

4. Is this updated in the wiki? The text "If the magic byte on message is
2, the broker should use the tombstone bit for log compaction." doesn't
seem to have changed.

2. My point is that if we change the message format just for this KIP, we
should consider whether it's worth optimizing the down conversion path
(i.e., decide whether a conversion is needed by just looking at the
tombstone bit in the wrapper message) since tombstone will be used rarely.
However, if the message format change here is combined with other KIPs,
then this optimization likely won't be needed. The latter probably makes
the code simpler. Jiangjie, Mayuresh, what do you think?

Other than those, +1 from me,

Thanks,

Jun

On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jun
>
> do we have your vote on this now?
>
> Any other concerns?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Michael Pearce <Mi...@ig.com>
> Sent: Saturday, December 3, 2016 1:37:45 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hi Jun,
>
> Have updated. Thanks again for the feedback.
>
> Agree yes we should align up when it gets to that, I assume you’ve flagged
> the same in the other KIP?
>
> I think for now let’s get this KIP approved, then we can start the work to
> get an initial PR, then we can discuss how to align the two if needed.
>
> Cheers,
> Mike
>
> On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
>
>     Hi, Michael,
>
>     For 2), this is fine. Could you update the KIP wiki to make this and
> other
>     points clearer? Other than that, the KIP looks good to me.
>
>     An orthogonal thing is that there are other KIPs such as KIP-98 that
> also
>     intend to change the message format. If they all get approved, we
> should
>     think about whether it's better to just bump up the magic byte once to
>     incorporate multiple format changes like we did in KIP-31/KIP-32.
>
>     Thanks,
>
>     Jun
>
>
>     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <Mi...@ig.com>
>     wrote:
>
>     > Hi Jao
>     >
>     > Thanks for the response. Sorry for slow reply, both with personal
> sickness
>     > and also battling some critical issues encountered since upgrading to
>     > 0.10.1.0
>     >
>     > 1) Thans for spotting, Document error where we branched this KIP from
>     > KIP-82, will get that removed.
>     > 2) Intent is to do this just at the record message level.
>     > 3) Thanks for spotting, Will ensure this is corrected.
>     > 4) As per discussion thread we will support tombstone + null value,
>     > tombstone + non null value, no tombstone + null value.
>     > 5) I believe this was in the discussion thread, @Mayuresh is this
>     > something we’ve overlooked? I thought we would down convert and
> remove the
>     > value so the old consumer had existing behavior, or is there
> something we
>     > haven’t thought about?
>     >
>     > Cheers
>     > Mike
>     >
>     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
>     >
>     >     Hi, Michael,
>     >
>     >     Thanks for the KIP. A few comments below.
>     >
>     >     1. The message format change contains "HeadersLength Headers".
> Is that
>     >     intended?
>     >
>     >     2. For compressed messageset, is the tombstone bit only set at
> the
>     > shallow
>     >     level? Do we always leave that bit in the wrapper message unset?
> An
>     >     alternative is to set the tombstone bit in the wrapper if at
> least one
>     >     inner message has the tombstone bit set. This makes things a bit
> more
>     >     complicated, but we could potentially exploit that for
> optimizing down
>     >     conversion. For example, we only need to convert messages with
> magic 2
>     > to
>     >     magic 1 if the wrapper's tombstone bit is set (conversion is
> always
>     > needed
>     >     from magic 2 to magic 0). Not sure if the optimization is worth
> the
>     >     complexity though.
>     >
>     >     3. The referencing of the new version of
> ProducerRequest/FetchRequest
>     > is
>     >     inconsistent (v4 vs v3). Since our convention starts at version
> at 0, I
>     >     think the new version would be 3.
>     >
>     >     4. "If the magic byte on message is 2, the broker should use the
>     > tombstone
>     >     bit for log compaction." What about null value? My understanding
> is
>     > that
>     >     null value will be treated the same as setting the tombstone bit.
>     >
>     >     5. For the migration path, it would be useful to describe the
> down
>     >     conversion path to consumers (i.e., brokers on message format
> 0.10.2
>     > and
>     >     consumers on older version).
>     >
>     >     Thanks,
>     >
>     >     Jun
>     >
>     >
>     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> Michael.Pearce@ig.com
>     > >
>     >     wrote:
>     >
>     >     > Hi All,
>     >     >
>     >     > We have been discussing in the below thread and final changes
> have
>     > been
>     >     > made to the KIP wiki based on these discussions.
>     >     >
>     >     > We would now like to put to the vote the following KIP:
>     >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
>     >     > Add+Compaction+Tombstone+Flag
>     >     >
>     >     > This kip is for having a distinct compaction attribute
> “tombstone”
>     > flag
>     >     > instead of relying on null value, allowing non-null value
> delete
>     > messages.
>     >     >
>     >     > Many thanks,
>     >     > Michael
>     >     >
>     >     >
>     >     >
>     >     > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com>
>     > wrote:
>     >     >
>     >     >     Hi Mayuresh,
>     >     >
>     >     >     LGTM. Ive just made one small adjustment updating the wire
>     > protocol to
>     >     > show the magic byte bump.
>     >     >
>     >     >     Do we think we’re good to put to a vote? Is there any
> other bits
>     >     > needing discussion?
>     >     >
>     >     >     Cheers
>     >     >     Mike
>     >     >
>     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
>     > gharatmayuresh15@gmail.com>
>     >     > wrote:
>     >     >
>     >     >         Hi Michael,
>     >     >
>     >     >         I have updated the migration section of the KIP. Can
> you
>     > please
>     >     > take a look?
>     >     >
>     >     >         Thanks,
>     >     >
>     >     >         Mayuresh
>     >     >
>     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
>     >     > gharatmayuresh15@gmail.com
>     >     >         > wrote:
>     >     >
>     >     >         > Hi Michael,
>     >     >         >
>     >     >         > That whilst sending tombstone and non null value, the
>     > consumer
>     >     > can expect
>     >     >         > only to receive the non-null message only in step
> (3) is
>     > this
>     >     > correct?
>     >     >         > ---> I do agree with you here.
>     >     >         >
>     >     >         > Becket, Ismael : can you guys review the migration
> plan
>     > listed
>     >     > above using
>     >     >         > magic byte?
>     >     >         >
>     >     >         > Thanks,
>     >     >         >
>     >     >         > Mayuresh
>     >     >         >
>     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
>     >     > Michael.Pearce@ig.com>
>     >     >         > wrote:
>     >     >         >
>     >     >         >> Many thanks for this Mayuresh. I don't have any
>     > objections.
>     >     >         >>
>     >     >         >> I assume we should state:
>     >     >         >>
>     >     >         >> That whilst sending tombstone and non null value,
> the
>     > consumer
>     >     > can expect
>     >     >         >> only to receive the non-null message only in step
> (3) is
>     > this
>     >     > correct?
>     >     >         >>
>     >     >         >> Cheers
>     >     >         >> Mike
>     >     >         >>
>     >     >         >>
>     >     >         >>
>     >     >         >> Sent using OWA for iPhone
>     >     >         >> ________________________________________
>     >     >         >> From: Mayuresh Gharat <gh...@gmail.com>
>     >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
>     >     >         >> To: dev@kafka.apache.org
>     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> Tombstone
>     > Flag
>     >     >         >>
>     >     >         >> Hi Ismael,
>     >     >         >>
>     >     >         >> Thanks for the explanation.
>     >     >         >> Specially I like this part where in you mentioned
> we can
>     > get
>     >     > rid of the
>     >     >         >> older null value support for log compaction later
> on,
>     > here :
>     >     >         >> We can't change semantics of the message format
> without
>     > having
>     >     > a long
>     >     >         >> transition period. And we can't rely
>     >     >         >> on people reading documentation or acting on a
> warning for
>     >     > something so
>     >     >         >> fundamental. As such, my take is that we need to
> bump the
>     > magic
>     >     > byte. The
>     >     >         >> good news is
>     >     >         >> that we don't have to support all versions forever.
> We
>     > have
>     >     > said that we
>     >     >         >> will support direct upgrades for 2 years. That
> means that
>     >     > message format
>     >     >         >> version n could, in theory, be removed 2 years
> after the
>     > it's
>     >     > introduced.
>     >     >         >>
>     >     >         >> Just a heads up, I would like to mention that even
> without
>     >     > bumping magic
>     >     >         >> byte, we will *NOT* loose zero copy as in the
> client(x+1)
>     > in my
>     >     >         >> explanation
>     >     >         >> above will convert internally a null value to have a
>     > tombstone
>     >     > bit set and
>     >     >         >> a tombstone bit set to have a null value
> automatically
>     >     > internally and by
>     >     >         >> the time we move to version (x+2), the clients
> would have
>     >     > upgraded.
>     >     >         >> Obviously if we support a request from consumer(x),
> we
>     > will
>     >     > loose zero
>     >     >         >> copy
>     >     >         >> but that is the same case with magic byte.
>     >     >         >>
>     >     >         >> But if magic byte bump makes life easier for
> transition
>     > for the
>     >     > above
>     >     >         >> reasons that you explained, I am OK with it since
> we are
>     > going
>     >     > to meet the
>     >     >         >> end goal down the road :)
>     >     >         >>
>     >     >         >> On a side note can we update the doc here on magic
> byte
>     > to say
>     >     > that "*it
>     >     >         >> should be bumped whenever the message format is
> changed
>     > or the
>     >     >         >> interpretation of message format (usage of the
> reserved
>     > bits as
>     >     > well) is
>     >     >         >> changed*".
>     >     >         >>
>     >     >         >>
>     >     >         >> Hi Michael,
>     >     >         >>
>     >     >         >> Here is the update plan that we discussed offline
>     > yesterday :
>     >     >         >>
>     >     >         >> Currently the magic-byte which corresponds to the
>     >     > "message.format.version"
>     >     >         >> is set to 1.
>     >     >         >>
>     >     >         >> 1) On broker it will be set to 1 initially.
>     >     >         >>
>     >     >         >> 2) When a producer client sends a message with
> magic-byte
>     > = 2,
>     >     > since the
>     >     >         >> broker is on magic-byte = 1, we will down convert
> it,
>     > which
>     >     > means if the
>     >     >         >> tombstone bit is set, the value will be set to
> null. A
>     > consumer
>     >     >         >> understanding magic-byte = 1, will still work with
> this. A
>     >     > consumer
>     >     >         >> working
>     >     >         >> with magic-byte =2 will also be able to understand
> this,
>     > since
>     >     > it
>     >     >         >> understands the tombstone.
>     >     >         >> Now there is still the question of supporting a
>     > non-tombstone
>     >     > and null
>     >     >         >> value from producer client with magic-byte = 2.* (I
> am
>     > not sure
>     >     > if we
>     >     >         >> should support this. Ismael/Becket can comment
> here)*
>     >     >         >>
>     >     >         >> 3) When almost all the clients have upgraded, the
>     >     > message.format.version
>     >     >         >> on
>     >     >         >> the broker can be changed to 2, where in the down
>     > conversion in
>     >     > the above
>     >     >         >> step will not happen. If at this point we get a
> consumer
>     >     > request from a
>     >     >         >> older consumer, we might have to down convert where
> in we
>     > loose
>     >     > zero copy,
>     >     >         >> but these cases should be rare.
>     >     >         >>
>     >     >         >> Becket can you review this plan and add more
> details if I
>     > have
>     >     >         >> missed/wronged something, before we put it on KIP.
>     >     >         >>
>     >     >         >> Thanks,
>     >     >         >>
>     >     >         >> Mayuresh
>     >     >         >>
>     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
>     >     > Michael.Pearce@ig.com>
>     >     >         >> wrote:
>     >     >         >>
>     >     >         >> > Thanks guys, for discussing this offline and
> getting
>     > some
>     >     > consensus.
>     >     >         >> >
>     >     >         >> > So its clear for myself and others what is
> proposed now
>     > (i
>     >     > think i
>     >     >         >> > understand, but want to make sure)
>     >     >         >> >
>     >     >         >> > Could i ask either directly update the kip to
> detail the
>     >     > migration
>     >     >         >> > strategy, or (re-)state your offline discussed and
>     > agreed
>     >     > migration
>     >     >         >> > strategy based on a magic byte is in this thread.
>     >     >         >> >
>     >     >         >> >
>     >     >         >> > The main original driver for the KIP was to
> support
>     >     > compaction where
>     >     >         >> value
>     >     >         >> > isn't null, based off the discussions on KIP-82
> thread.
>     >     >         >> >
>     >     >         >> > We should be able to support non-tombstone + null
> value
>     > by the
>     >     >         >> completion
>     >     >         >> > of the KIP, as we noted when discussing this kip,
> having
>     >     > logic based on
>     >     >         >> a
>     >     >         >> > null value isn't very clean and also separates the
>     > concerns.
>     >     >         >> >
>     >     >         >> > As discussed already though we can split this into
>     > KIP-87a
>     >     > and KIP-87b
>     >     >         >> >
>     >     >         >> > Where we look to deliver KIP-87a on a compacted
> topic
>     > (to
>     >     > address the
>     >     >         >> > immediate issues)
>     >     >         >> > * tombstone + null value
>     >     >         >> > * tombstone + non-null value
>     >     >         >> > * non-tombstone + non-null value
>     >     >         >> >
>     >     >         >> > Then we can discuss once KIP-87a is completed
> options
>     > later
>     >     > and how we
>     >     >         >> > support the second part KIP-87b to deliver:
>     >     >         >> > * non-tombstone + null value
>     >     >         >> >
>     >     >         >> > Cheers
>     >     >         >> > Mike
>     >     >         >> >
>     >     >         >> >
>     >     >         >> >
>     >     >         >> > ________________________________________
>     >     >         >> > From: Becket Qin <be...@gmail.com>
>     >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
>     >     >         >> > To: dev@kafka.apache.org
>     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
>     > Tombstone Flag
>     >     >         >> >
>     >     >         >> > Renu, Mayuresh and I had an offline discussion,
> and
>     > following
>     >     > is a brief
>     >     >         >> > summary.
>     >     >         >> >
>     >     >         >> > 1. We agreed that not bumping up magic value may
> result
>     > in
>     >     > losing zero
>     >     >         >> copy
>     >     >         >> > during migration.
>     >     >         >> > 2. Given that bumping up magic value is almost
> free and
>     > has
>     >     > benefit of
>     >     >         >> > avoiding potential performance issue. It is
> probably
>     > worth
>     >     > doing.
>     >     >         >> >
>     >     >         >> > One issue we still need to think about is whether
> we
>     > want to
>     >     > support a
>     >     >         >> > non-tombstone message with null value.
>     >     >         >> > Currently it is not supported by Kafka. If we
> allow a
>     >     > non-tombstone null
>     >     >         >> > value message to exist after KIP-87. The problem
> is
>     > that such
>     >     > message
>     >     >         >> will
>     >     >         >> > not be supported by the consumers prior to KIP-87.
>     > Because a
>     >     > null value
>     >     >         >> > will always be interpreted to a tombstone.
>     >     >         >> >
>     >     >         >> > One option is that we keep the current way, i.e.
> do not
>     >     > support such
>     >     >         >> > message. It would be good to know if there is a
>     > concrete use
>     >     > case for
>     >     >         >> such
>     >     >         >> > message. If there is not, we can probably just not
>     > support it.
>     >     >         >> >
>     >     >         >> > Thanks,
>     >     >         >> >
>     >     >         >> > JIangjie (Becket) Qin
>     >     >         >> >
>     >     >         >> >
>     >     >         >> >
>     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
>     >     >         >> > gharatmayuresh15@gmail.com
>     >     >         >> > > wrote:
>     >     >         >> >
>     >     >         >> > > Hi Ismael,
>     >     >         >> > >
>     >     >         >> > > This is something I can think of for migration
> plan:
>     >     >         >> > > So the migration plan can look something like
> this,
>     > with up
>     >     >         >> conversion :
>     >     >         >> > >
>     >     >         >> > > 1) Currently lets say we have Broker at version
> x.
>     >     >         >> > > 2) Currently we have clients at version x.
>     >     >         >> > > 3) a) We move the version to Broker(x+1) :
> supports
>     > both
>     >     > tombstone and
>     >     >         >> > null
>     >     >         >> > > for log compaction.
>     >     >         >> > >     b) We upgrade the client to version
> client(x+1) :
>     > if in
>     >     > the
>     >     >         >> producer
>     >     >         >> > > client(x+1) the value is set to null, we will
>     > automatically
>     >     > set the
>     >     >         >> > > Tombstone bit internally. If the producer
> client(x+1)
>     > sets
>     >     > the
>     >     >         >> tombstone
>     >     >         >> > > itself, well and good. For producer client(x),
> the
>     > broker
>     >     > will up
>     >     >         >> convert
>     >     >         >> > > to have the tombstone bit. Broker(x+1) is
> supporting
>     > both.
>     >     > Consumer
>     >     >         >> > > client(x+1) will be aware of this and should be
> able
>     > to
>     >     > handle this.
>     >     >         >> For
>     >     >         >> > > consumer client(x) we will down convert the
> message
>     > on the
>     >     > broker
>     >     >         >> side.
>     >     >         >> > >     c) At this point we will have to specify a
>     > warning or
>     >     > clearly
>     >     >         >> specify
>     >     >         >> > > in docs that this behavior is about to be
> changed for
>     > log
>     >     > compaction.
>     >     >         >> > > 4) a) In next release of the Broker(x+2), we
> say that
>     > only
>     >     > Tombstone
>     >     >         >> is
>     >     >         >> > > used for log compaction on the Broker side.
>     > Clients(x+1)
>     >     > still is
>     >     >         >> > > supported.
>     >     >         >> > >     b) We upgrade the client to version
> client(x+2) :
>     > if
>     >     > value is set
>     >     >         >> to
>     >     >         >> > > null, tombstone will not be set automatically.
> The
>     > client
>     >     > will have to
>     >     >         >> > call
>     >     >         >> > > setTombstone() to actually set the tombstone.
>     >     >         >> > >
>     >     >         >> > > We should compare this migration plan with the
>     > migration
>     >     > plan for
>     >     >         >> magic
>     >     >         >> > > byte bump and do whatever looks good.
>     >     >         >> > > I am just worried that if we go down magic byte
> route,
>     >     > unless I am
>     >     >         >> > missing
>     >     >         >> > > something, it sounds like kafka will be stuck
> with
>     >     > supporting both
>     >     >         >> null
>     >     >         >> > > value and tombstone bit for log compaction for
> life
>     > long,
>     >     > which does
>     >     >         >> not
>     >     >         >> > > look like a good end state.
>     >     >         >> > >
>     >     >         >> > > Thanks,
>     >     >         >> > >
>     >     >         >> > > Mayuresh
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh
> Gharat <
>     >     >         >> > > gharatmayuresh15@gmail.com
>     >     >         >> > > > wrote:
>     >     >         >> > >
>     >     >         >> > > > Hi Ismael,
>     >     >         >> > > >
>     >     >         >> > > > That's a very good point which I might have
> not
>     >     > considered earlier.
>     >     >         >> > > >
>     >     >         >> > > > Here is a plan that I can think of:
>     >     >         >> > > >
>     >     >         >> > > > Stage 1) The broker from now on, up converts
> the
>     > message
>     >     > to have the
>     >     >         >> > > > tombstone marker. The log compaction thread
> does log
>     >     > compaction
>     >     >         >> based
>     >     >         >> > on
>     >     >         >> > > > both null and tombstone marker. This is our
>     > transition
>     >     > period.
>     >     >         >> > > > Stage 2) The next release we only say that log
>     > compaction
>     >     > is based
>     >     >         >> on
>     >     >         >> > > > tombstone marker. (Open source kafka makes
> this as a
>     >     > policy). By
>     >     >         >> this
>     >     >         >> > > time,
>     >     >         >> > > > the organization which is moving to this
> release
>     > will be
>     >     > sure that
>     >     >         >> they
>     >     >         >> > > > have gone through the entire transition
> period.
>     >     >         >> > > >
>     >     >         >> > > > My only goal of doing this is that Kafka
> clearly
>     >     > specifies the end
>     >     >         >> > state
>     >     >         >> > > > about what log compaction means (is it null
> value
>     > or a
>     >     > tombstone
>     >     >         >> > marker,
>     >     >         >> > > > but not both).
>     >     >         >> > > >
>     >     >         >> > > > What do you think?
>     >     >         >> > > >
>     >     >         >> > > > Thanks,
>     >     >         >> > > >
>     >     >         >> > > > Mayuresh
>     >     >         >> > > > .
>     >     >         >> > > >
>     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
>     >     > ismael@juma.me.uk>
>     >     >         >> > wrote:
>     >     >         >> > > >
>     >     >         >> > > >> One comment below.
>     >     >         >> > > >>
>     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
> Gharat <
>     >     >         >> > > >> gharatmayuresh15@gmail.com
>     >     >         >> > > >> > wrote:
>     >     >         >> > > >>
>     >     >         >> > > >> >    - If we don't bump up the magic byte,
> on the
>     > broker
>     >     > side, the
>     >     >         >> > > broker
>     >     >         >> > > >> >    will always have to look at both
> tombstone
>     > bit and
>     >     > the value
>     >     >         >> when
>     >     >         >> > > do
>     >     >         >> > > >> the
>     >     >         >> > > >> >    compaction. Assuming we do not bump up
> the
>     > magic
>     >     > byte,
>     >     >         >> > > >> >    imagine the broker sees a message which
> does
>     > not
>     >     > have a
>     >     >         >> tombstone
>     >     >         >> > > bit
>     >     >         >> > > >> >    set. The broker does not know when the
>     > message was
>     >     > produced
>     >     >         >> (i.e.
>     >     >         >> > > >> > whether
>     >     >         >> > > >> >    the message has been up converted or
> not), it
>     > has
>     >     > to take a
>     >     >         >> > further
>     >     >         >> > > >> > look at
>     >     >         >> > > >> >    the value to see if it is null or not in
>     > order to
>     >     > determine
>     >     >         >> if it
>     >     >         >> > > is
>     >     >         >> > > >> a
>     >     >         >> > > >> >    tombstone. The same logic has to be put
> on the
>     >     > consumer as
>     >     >         >> well
>     >     >         >> > > >> because
>     >     >         >> > > >> > the
>     >     >         >> > > >> >    consumer does not know if the message
> has
>     > been up
>     >     > converted or
>     >     >         >> > not.
>     >     >         >> > > >> >       - If we upconvert while appending,
> this is
>     > not
>     >     > the case,
>     >     >         >> > right?
>     >     >         >> > > >>
>     >     >         >> > > >>
>     >     >         >> > > >> If I understand you correctly, this is not
>     > sufficient
>     >     > because the
>     >     >         >> log
>     >     >         >> > > may
>     >     >         >> > > >> have messages appended before it was
> upgraded to
>     > include
>     >     > KIP-87.
>     >     >         >> > > >>
>     >     >         >> > > >> Ismael
>     >     >         >> > > >>
>     >     >         >> > > >
>     >     >         >> > > >
>     >     >         >> > > >
>     >     >         >> > > > --
>     >     >         >> > > > -Regards,
>     >     >         >> > > > Mayuresh R. Gharat
>     >     >         >> > > > (862) 250-7125
>     >     >         >> > > >
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > > --
>     >     >         >> > > -Regards,
>     >     >         >> > > Mayuresh R. Gharat
>     >     >         >> > > (862) 250-7125
>     >     >         >> > >
>     >     >         >> > The information contained in this email is
> strictly
>     >     > confidential and for
>     >     >         >> > the use of the addressee only, unless otherwise
>     > indicated. If
>     >     > you are
>     >     >         >> not
>     >     >         >> > the intended recipient, please do not read, copy,
> use or
>     >     > disclose to
>     >     >         >> others
>     >     >         >> > this message or any attachment. Please also
> notify the
>     > sender
>     >     > by
>     >     >         >> replying
>     >     >         >> > to this email or by telephone (+44(020 7896 0011)
> and
>     > then
>     >     > delete the
>     >     >         >> email
>     >     >         >> > and any copies of it. Opinions, conclusion (etc)
> that
>     > do not
>     >     > relate to
>     >     >         >> the
>     >     >         >> > official business of this company shall be
> understood as
>     >     > neither given
>     >     >         >> nor
>     >     >         >> > endorsed by it. IG is a trading name of IG Markets
>     > Limited (a
>     >     > company
>     >     >         >> > registered in England and Wales, company number
>     > 04008957) and
>     >     > IG Index
>     >     >         >> > Limited (a company registered in England and
> Wales,
>     > company
>     >     > number
>     >     >         >> > 01190902). Registered address at Cannon Bridge
> House, 25
>     >     > Dowgate Hill,
>     >     >         >> > London EC4R 2YA. Both IG Markets Limited (register
>     > number
>     >     > 195355) and IG
>     >     >         >> > Index Limited (register number 114059) are
> authorised
>     > and
>     >     > regulated by
>     >     >         >> the
>     >     >         >> > Financial Conduct Authority.
>     >     >         >> >
>     >     >         >>
>     >     >         >>
>     >     >         >>
>     >     >         >> --
>     >     >         >> -Regards,
>     >     >         >> Mayuresh R. Gharat
>     >     >         >> (862) 250-7125
>     >     >         >> The information contained in this email is strictly
>     >     > confidential and for
>     >     >         >> the use of the addressee only, unless otherwise
>     > indicated. If
>     >     > you are not
>     >     >         >> the intended recipient, please do not read, copy,
> use or
>     >     > disclose to others
>     >     >         >> this message or any attachment. Please also notify
> the
>     > sender
>     >     > by replying
>     >     >         >> to this email or by telephone (+44(020 7896 0011)
> and then
>     >     > delete the email
>     >     >         >> and any copies of it. Opinions, conclusion (etc)
> that do
>     > not
>     >     > relate to the
>     >     >         >> official business of this company shall be
> understood as
>     >     > neither given nor
>     >     >         >> endorsed by it. IG is a trading name of IG Markets
>     > Limited (a
>     >     > company
>     >     >         >> registered in England and Wales, company number
> 04008957)
>     > and
>     >     > IG Index
>     >     >         >> Limited (a company registered in England and Wales,
>     > company
>     >     > number
>     >     >         >> 01190902). Registered address at Cannon Bridge
> House, 25
>     >     > Dowgate Hill,
>     >     >         >> London EC4R 2YA. Both IG Markets Limited (register
> number
>     >     > 195355) and IG
>     >     >         >> Index Limited (register number 114059) are
> authorised and
>     >     > regulated by the
>     >     >         >> Financial Conduct Authority.
>     >     >         >>
>     >     >         >
>     >     >         >
>     >     >         >
>     >     >         > --
>     >     >         > -Regards,
>     >     >         > Mayuresh R. Gharat
>     >     >         > (862) 250-7125
>     >     >         >
>     >     >
>     >     >
>     >     >
>     >     >         --
>     >     >         -Regards,
>     >     >         Mayuresh R. Gharat
>     >     >         (862) 250-7125
>     >     >
>     >     >
>     >     >     The information contained in this email is strictly
> confidential
>     > and
>     >     > for the use of the addressee only, unless otherwise indicated.
> If
>     > you are
>     >     > not the intended recipient, please do not read, copy, use or
>     > disclose to
>     >     > others this message or any attachment. Please also notify the
> sender
>     > by
>     >     > replying to this email or by telephone (+44(020 7896 0011) and
> then
>     > delete
>     >     > the email and any copies of it. Opinions, conclusion (etc)
> that do
>     > not
>     >     > relate to the official business of this company shall be
> understood
>     > as
>     >     > neither given nor endorsed by it. IG is a trading name of IG
> Markets
>     >     > Limited (a company registered in England and Wales, company
> number
>     >     > 04008957) and IG Index Limited (a company registered in
> England and
>     > Wales,
>     >     > company number 01190902). Registered address at Cannon Bridge
> House,
>     > 25
>     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> (register
>     > number
>     >     > 195355) and IG Index Limited (register number 114059) are
> authorised
>     > and
>     >     > regulated by the Financial Conduct Authority.
>     >     >
>     >     >
>     >     >
>     >
>     >
>     > The information contained in this email is strictly confidential and
> for
>     > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > the intended recipient, please do not read, copy, use or disclose to
> others
>     > this message or any attachment. Please also notify the sender by
> replying
>     > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
>     > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
>     > official business of this company shall be understood as neither
> given nor
>     > endorsed by it. IG is a trading name of IG Markets Limited (a company
>     > registered in England and Wales, company number 04008957) and IG
> Index
>     > Limited (a company registered in England and Wales, company number
>     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > Index Limited (register number 114059) are authorised and regulated
> by the
>     > Financial Conduct Authority.
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jun Rao <ju...@confluent.io>]

Hi, Michael,

4. Is this updated in the wiki? The text "If the magic byte on message is
2, the broker should use the tombstone bit for log compaction." doesn't
seem to have changed.

2. My point is that if we change the message format just for this KIP, we
should consider whether it's worth optimizing the down conversion path
(i.e., decide whether a conversion is needed by just looking at the
tombstone bit in the wrapper message) since tombstone will be used rarely.
However, if the message format change here is combined with other KIPs,
then this optimization likely won't be needed. The latter probably makes
the code simpler. Jiangjie, Mayuresh, what do you think?

Other than those, +1 from me,

Thanks,

Jun

On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jun
>
> do we have your vote on this now?
>
> Any other concerns?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Michael Pearce <Mi...@ig.com>
> Sent: Saturday, December 3, 2016 1:37:45 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hi Jun,
>
> Have updated. Thanks again for the feedback.
>
> Agree yes we should align up when it gets to that, I assume you’ve flagged
> the same in the other KIP?
>
> I think for now let’s get this KIP approved, then we can start the work to
> get an initial PR, then we can discuss how to align the two if needed.
>
> Cheers,
> Mike
>
> On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:
>
>     Hi, Michael,
>
>     For 2), this is fine. Could you update the KIP wiki to make this and
> other
>     points clearer? Other than that, the KIP looks good to me.
>
>     An orthogonal thing is that there are other KIPs such as KIP-98 that
> also
>     intend to change the message format. If they all get approved, we
> should
>     think about whether it's better to just bump up the magic byte once to
>     incorporate multiple format changes like we did in KIP-31/KIP-32.
>
>     Thanks,
>
>     Jun
>
>
>     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <Mi...@ig.com>
>     wrote:
>
>     > Hi Jao
>     >
>     > Thanks for the response. Sorry for slow reply, both with personal
> sickness
>     > and also battling some critical issues encountered since upgrading to
>     > 0.10.1.0
>     >
>     > 1) Thans for spotting, Document error where we branched this KIP from
>     > KIP-82, will get that removed.
>     > 2) Intent is to do this just at the record message level.
>     > 3) Thanks for spotting, Will ensure this is corrected.
>     > 4) As per discussion thread we will support tombstone + null value,
>     > tombstone + non null value, no tombstone + null value.
>     > 5) I believe this was in the discussion thread, @Mayuresh is this
>     > something we’ve overlooked? I thought we would down convert and
> remove the
>     > value so the old consumer had existing behavior, or is there
> something we
>     > haven’t thought about?
>     >
>     > Cheers
>     > Mike
>     >
>     > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
>     >
>     >     Hi, Michael,
>     >
>     >     Thanks for the KIP. A few comments below.
>     >
>     >     1. The message format change contains "HeadersLength Headers".
> Is that
>     >     intended?
>     >
>     >     2. For compressed messageset, is the tombstone bit only set at
> the
>     > shallow
>     >     level? Do we always leave that bit in the wrapper message unset?
> An
>     >     alternative is to set the tombstone bit in the wrapper if at
> least one
>     >     inner message has the tombstone bit set. This makes things a bit
> more
>     >     complicated, but we could potentially exploit that for
> optimizing down
>     >     conversion. For example, we only need to convert messages with
> magic 2
>     > to
>     >     magic 1 if the wrapper's tombstone bit is set (conversion is
> always
>     > needed
>     >     from magic 2 to magic 0). Not sure if the optimization is worth
> the
>     >     complexity though.
>     >
>     >     3. The referencing of the new version of
> ProducerRequest/FetchRequest
>     > is
>     >     inconsistent (v4 vs v3). Since our convention starts at version
> at 0, I
>     >     think the new version would be 3.
>     >
>     >     4. "If the magic byte on message is 2, the broker should use the
>     > tombstone
>     >     bit for log compaction." What about null value? My understanding
> is
>     > that
>     >     null value will be treated the same as setting the tombstone bit.
>     >
>     >     5. For the migration path, it would be useful to describe the
> down
>     >     conversion path to consumers (i.e., brokers on message format
> 0.10.2
>     > and
>     >     consumers on older version).
>     >
>     >     Thanks,
>     >
>     >     Jun
>     >
>     >
>     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> Michael.Pearce@ig.com
>     > >
>     >     wrote:
>     >
>     >     > Hi All,
>     >     >
>     >     > We have been discussing in the below thread and final changes
> have
>     > been
>     >     > made to the KIP wiki based on these discussions.
>     >     >
>     >     > We would now like to put to the vote the following KIP:
>     >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
>     >     > Add+Compaction+Tombstone+Flag
>     >     >
>     >     > This kip is for having a distinct compaction attribute
> “tombstone”
>     > flag
>     >     > instead of relying on null value, allowing non-null value
> delete
>     > messages.
>     >     >
>     >     > Many thanks,
>     >     > Michael
>     >     >
>     >     >
>     >     >
>     >     > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com>
>     > wrote:
>     >     >
>     >     >     Hi Mayuresh,
>     >     >
>     >     >     LGTM. Ive just made one small adjustment updating the wire
>     > protocol to
>     >     > show the magic byte bump.
>     >     >
>     >     >     Do we think we’re good to put to a vote? Is there any
> other bits
>     >     > needing discussion?
>     >     >
>     >     >     Cheers
>     >     >     Mike
>     >     >
>     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
>     > gharatmayuresh15@gmail.com>
>     >     > wrote:
>     >     >
>     >     >         Hi Michael,
>     >     >
>     >     >         I have updated the migration section of the KIP. Can
> you
>     > please
>     >     > take a look?
>     >     >
>     >     >         Thanks,
>     >     >
>     >     >         Mayuresh
>     >     >
>     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
>     >     > gharatmayuresh15@gmail.com
>     >     >         > wrote:
>     >     >
>     >     >         > Hi Michael,
>     >     >         >
>     >     >         > That whilst sending tombstone and non null value, the
>     > consumer
>     >     > can expect
>     >     >         > only to receive the non-null message only in step
> (3) is
>     > this
>     >     > correct?
>     >     >         > ---> I do agree with you here.
>     >     >         >
>     >     >         > Becket, Ismael : can you guys review the migration
> plan
>     > listed
>     >     > above using
>     >     >         > magic byte?
>     >     >         >
>     >     >         > Thanks,
>     >     >         >
>     >     >         > Mayuresh
>     >     >         >
>     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
>     >     > Michael.Pearce@ig.com>
>     >     >         > wrote:
>     >     >         >
>     >     >         >> Many thanks for this Mayuresh. I don't have any
>     > objections.
>     >     >         >>
>     >     >         >> I assume we should state:
>     >     >         >>
>     >     >         >> That whilst sending tombstone and non null value,
> the
>     > consumer
>     >     > can expect
>     >     >         >> only to receive the non-null message only in step
> (3) is
>     > this
>     >     > correct?
>     >     >         >>
>     >     >         >> Cheers
>     >     >         >> Mike
>     >     >         >>
>     >     >         >>
>     >     >         >>
>     >     >         >> Sent using OWA for iPhone
>     >     >         >> ________________________________________
>     >     >         >> From: Mayuresh Gharat <gh...@gmail.com>
>     >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
>     >     >         >> To: dev@kafka.apache.org
>     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> Tombstone
>     > Flag
>     >     >         >>
>     >     >         >> Hi Ismael,
>     >     >         >>
>     >     >         >> Thanks for the explanation.
>     >     >         >> Specially I like this part where in you mentioned
> we can
>     > get
>     >     > rid of the
>     >     >         >> older null value support for log compaction later
> on,
>     > here :
>     >     >         >> We can't change semantics of the message format
> without
>     > having
>     >     > a long
>     >     >         >> transition period. And we can't rely
>     >     >         >> on people reading documentation or acting on a
> warning for
>     >     > something so
>     >     >         >> fundamental. As such, my take is that we need to
> bump the
>     > magic
>     >     > byte. The
>     >     >         >> good news is
>     >     >         >> that we don't have to support all versions forever.
> We
>     > have
>     >     > said that we
>     >     >         >> will support direct upgrades for 2 years. That
> means that
>     >     > message format
>     >     >         >> version n could, in theory, be removed 2 years
> after the
>     > it's
>     >     > introduced.
>     >     >         >>
>     >     >         >> Just a heads up, I would like to mention that even
> without
>     >     > bumping magic
>     >     >         >> byte, we will *NOT* loose zero copy as in the
> client(x+1)
>     > in my
>     >     >         >> explanation
>     >     >         >> above will convert internally a null value to have a
>     > tombstone
>     >     > bit set and
>     >     >         >> a tombstone bit set to have a null value
> automatically
>     >     > internally and by
>     >     >         >> the time we move to version (x+2), the clients
> would have
>     >     > upgraded.
>     >     >         >> Obviously if we support a request from consumer(x),
> we
>     > will
>     >     > loose zero
>     >     >         >> copy
>     >     >         >> but that is the same case with magic byte.
>     >     >         >>
>     >     >         >> But if magic byte bump makes life easier for
> transition
>     > for the
>     >     > above
>     >     >         >> reasons that you explained, I am OK with it since
> we are
>     > going
>     >     > to meet the
>     >     >         >> end goal down the road :)
>     >     >         >>
>     >     >         >> On a side note can we update the doc here on magic
> byte
>     > to say
>     >     > that "*it
>     >     >         >> should be bumped whenever the message format is
> changed
>     > or the
>     >     >         >> interpretation of message format (usage of the
> reserved
>     > bits as
>     >     > well) is
>     >     >         >> changed*".
>     >     >         >>
>     >     >         >>
>     >     >         >> Hi Michael,
>     >     >         >>
>     >     >         >> Here is the update plan that we discussed offline
>     > yesterday :
>     >     >         >>
>     >     >         >> Currently the magic-byte which corresponds to the
>     >     > "message.format.version"
>     >     >         >> is set to 1.
>     >     >         >>
>     >     >         >> 1) On broker it will be set to 1 initially.
>     >     >         >>
>     >     >         >> 2) When a producer client sends a message with
> magic-byte
>     > = 2,
>     >     > since the
>     >     >         >> broker is on magic-byte = 1, we will down convert
> it,
>     > which
>     >     > means if the
>     >     >         >> tombstone bit is set, the value will be set to
> null. A
>     > consumer
>     >     >         >> understanding magic-byte = 1, will still work with
> this. A
>     >     > consumer
>     >     >         >> working
>     >     >         >> with magic-byte =2 will also be able to understand
> this,
>     > since
>     >     > it
>     >     >         >> understands the tombstone.
>     >     >         >> Now there is still the question of supporting a
>     > non-tombstone
>     >     > and null
>     >     >         >> value from producer client with magic-byte = 2.* (I
> am
>     > not sure
>     >     > if we
>     >     >         >> should support this. Ismael/Becket can comment
> here)*
>     >     >         >>
>     >     >         >> 3) When almost all the clients have upgraded, the
>     >     > message.format.version
>     >     >         >> on
>     >     >         >> the broker can be changed to 2, where in the down
>     > conversion in
>     >     > the above
>     >     >         >> step will not happen. If at this point we get a
> consumer
>     >     > request from a
>     >     >         >> older consumer, we might have to down convert where
> in we
>     > loose
>     >     > zero copy,
>     >     >         >> but these cases should be rare.
>     >     >         >>
>     >     >         >> Becket can you review this plan and add more
> details if I
>     > have
>     >     >         >> missed/wronged something, before we put it on KIP.
>     >     >         >>
>     >     >         >> Thanks,
>     >     >         >>
>     >     >         >> Mayuresh
>     >     >         >>
>     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
>     >     > Michael.Pearce@ig.com>
>     >     >         >> wrote:
>     >     >         >>
>     >     >         >> > Thanks guys, for discussing this offline and
> getting
>     > some
>     >     > consensus.
>     >     >         >> >
>     >     >         >> > So its clear for myself and others what is
> proposed now
>     > (i
>     >     > think i
>     >     >         >> > understand, but want to make sure)
>     >     >         >> >
>     >     >         >> > Could i ask either directly update the kip to
> detail the
>     >     > migration
>     >     >         >> > strategy, or (re-)state your offline discussed and
>     > agreed
>     >     > migration
>     >     >         >> > strategy based on a magic byte is in this thread.
>     >     >         >> >
>     >     >         >> >
>     >     >         >> > The main original driver for the KIP was to
> support
>     >     > compaction where
>     >     >         >> value
>     >     >         >> > isn't null, based off the discussions on KIP-82
> thread.
>     >     >         >> >
>     >     >         >> > We should be able to support non-tombstone + null
> value
>     > by the
>     >     >         >> completion
>     >     >         >> > of the KIP, as we noted when discussing this kip,
> having
>     >     > logic based on
>     >     >         >> a
>     >     >         >> > null value isn't very clean and also separates the
>     > concerns.
>     >     >         >> >
>     >     >         >> > As discussed already though we can split this into
>     > KIP-87a
>     >     > and KIP-87b
>     >     >         >> >
>     >     >         >> > Where we look to deliver KIP-87a on a compacted
> topic
>     > (to
>     >     > address the
>     >     >         >> > immediate issues)
>     >     >         >> > * tombstone + null value
>     >     >         >> > * tombstone + non-null value
>     >     >         >> > * non-tombstone + non-null value
>     >     >         >> >
>     >     >         >> > Then we can discuss once KIP-87a is completed
> options
>     > later
>     >     > and how we
>     >     >         >> > support the second part KIP-87b to deliver:
>     >     >         >> > * non-tombstone + null value
>     >     >         >> >
>     >     >         >> > Cheers
>     >     >         >> > Mike
>     >     >         >> >
>     >     >         >> >
>     >     >         >> >
>     >     >         >> > ________________________________________
>     >     >         >> > From: Becket Qin <be...@gmail.com>
>     >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
>     >     >         >> > To: dev@kafka.apache.org
>     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
>     > Tombstone Flag
>     >     >         >> >
>     >     >         >> > Renu, Mayuresh and I had an offline discussion,
> and
>     > following
>     >     > is a brief
>     >     >         >> > summary.
>     >     >         >> >
>     >     >         >> > 1. We agreed that not bumping up magic value may
> result
>     > in
>     >     > losing zero
>     >     >         >> copy
>     >     >         >> > during migration.
>     >     >         >> > 2. Given that bumping up magic value is almost
> free and
>     > has
>     >     > benefit of
>     >     >         >> > avoiding potential performance issue. It is
> probably
>     > worth
>     >     > doing.
>     >     >         >> >
>     >     >         >> > One issue we still need to think about is whether
> we
>     > want to
>     >     > support a
>     >     >         >> > non-tombstone message with null value.
>     >     >         >> > Currently it is not supported by Kafka. If we
> allow a
>     >     > non-tombstone null
>     >     >         >> > value message to exist after KIP-87. The problem
> is
>     > that such
>     >     > message
>     >     >         >> will
>     >     >         >> > not be supported by the consumers prior to KIP-87.
>     > Because a
>     >     > null value
>     >     >         >> > will always be interpreted to a tombstone.
>     >     >         >> >
>     >     >         >> > One option is that we keep the current way, i.e.
> do not
>     >     > support such
>     >     >         >> > message. It would be good to know if there is a
>     > concrete use
>     >     > case for
>     >     >         >> such
>     >     >         >> > message. If there is not, we can probably just not
>     > support it.
>     >     >         >> >
>     >     >         >> > Thanks,
>     >     >         >> >
>     >     >         >> > JIangjie (Becket) Qin
>     >     >         >> >
>     >     >         >> >
>     >     >         >> >
>     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
>     >     >         >> > gharatmayuresh15@gmail.com
>     >     >         >> > > wrote:
>     >     >         >> >
>     >     >         >> > > Hi Ismael,
>     >     >         >> > >
>     >     >         >> > > This is something I can think of for migration
> plan:
>     >     >         >> > > So the migration plan can look something like
> this,
>     > with up
>     >     >         >> conversion :
>     >     >         >> > >
>     >     >         >> > > 1) Currently lets say we have Broker at version
> x.
>     >     >         >> > > 2) Currently we have clients at version x.
>     >     >         >> > > 3) a) We move the version to Broker(x+1) :
> supports
>     > both
>     >     > tombstone and
>     >     >         >> > null
>     >     >         >> > > for log compaction.
>     >     >         >> > >     b) We upgrade the client to version
> client(x+1) :
>     > if in
>     >     > the
>     >     >         >> producer
>     >     >         >> > > client(x+1) the value is set to null, we will
>     > automatically
>     >     > set the
>     >     >         >> > > Tombstone bit internally. If the producer
> client(x+1)
>     > sets
>     >     > the
>     >     >         >> tombstone
>     >     >         >> > > itself, well and good. For producer client(x),
> the
>     > broker
>     >     > will up
>     >     >         >> convert
>     >     >         >> > > to have the tombstone bit. Broker(x+1) is
> supporting
>     > both.
>     >     > Consumer
>     >     >         >> > > client(x+1) will be aware of this and should be
> able
>     > to
>     >     > handle this.
>     >     >         >> For
>     >     >         >> > > consumer client(x) we will down convert the
> message
>     > on the
>     >     > broker
>     >     >         >> side.
>     >     >         >> > >     c) At this point we will have to specify a
>     > warning or
>     >     > clearly
>     >     >         >> specify
>     >     >         >> > > in docs that this behavior is about to be
> changed for
>     > log
>     >     > compaction.
>     >     >         >> > > 4) a) In next release of the Broker(x+2), we
> say that
>     > only
>     >     > Tombstone
>     >     >         >> is
>     >     >         >> > > used for log compaction on the Broker side.
>     > Clients(x+1)
>     >     > still is
>     >     >         >> > > supported.
>     >     >         >> > >     b) We upgrade the client to version
> client(x+2) :
>     > if
>     >     > value is set
>     >     >         >> to
>     >     >         >> > > null, tombstone will not be set automatically.
> The
>     > client
>     >     > will have to
>     >     >         >> > call
>     >     >         >> > > setTombstone() to actually set the tombstone.
>     >     >         >> > >
>     >     >         >> > > We should compare this migration plan with the
>     > migration
>     >     > plan for
>     >     >         >> magic
>     >     >         >> > > byte bump and do whatever looks good.
>     >     >         >> > > I am just worried that if we go down magic byte
> route,
>     >     > unless I am
>     >     >         >> > missing
>     >     >         >> > > something, it sounds like kafka will be stuck
> with
>     >     > supporting both
>     >     >         >> null
>     >     >         >> > > value and tombstone bit for log compaction for
> life
>     > long,
>     >     > which does
>     >     >         >> not
>     >     >         >> > > look like a good end state.
>     >     >         >> > >
>     >     >         >> > > Thanks,
>     >     >         >> > >
>     >     >         >> > > Mayuresh
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh
> Gharat <
>     >     >         >> > > gharatmayuresh15@gmail.com
>     >     >         >> > > > wrote:
>     >     >         >> > >
>     >     >         >> > > > Hi Ismael,
>     >     >         >> > > >
>     >     >         >> > > > That's a very good point which I might have
> not
>     >     > considered earlier.
>     >     >         >> > > >
>     >     >         >> > > > Here is a plan that I can think of:
>     >     >         >> > > >
>     >     >         >> > > > Stage 1) The broker from now on, up converts
> the
>     > message
>     >     > to have the
>     >     >         >> > > > tombstone marker. The log compaction thread
> does log
>     >     > compaction
>     >     >         >> based
>     >     >         >> > on
>     >     >         >> > > > both null and tombstone marker. This is our
>     > transition
>     >     > period.
>     >     >         >> > > > Stage 2) The next release we only say that log
>     > compaction
>     >     > is based
>     >     >         >> on
>     >     >         >> > > > tombstone marker. (Open source kafka makes
> this as a
>     >     > policy). By
>     >     >         >> this
>     >     >         >> > > time,
>     >     >         >> > > > the organization which is moving to this
> release
>     > will be
>     >     > sure that
>     >     >         >> they
>     >     >         >> > > > have gone through the entire transition
> period.
>     >     >         >> > > >
>     >     >         >> > > > My only goal of doing this is that Kafka
> clearly
>     >     > specifies the end
>     >     >         >> > state
>     >     >         >> > > > about what log compaction means (is it null
> value
>     > or a
>     >     > tombstone
>     >     >         >> > marker,
>     >     >         >> > > > but not both).
>     >     >         >> > > >
>     >     >         >> > > > What do you think?
>     >     >         >> > > >
>     >     >         >> > > > Thanks,
>     >     >         >> > > >
>     >     >         >> > > > Mayuresh
>     >     >         >> > > > .
>     >     >         >> > > >
>     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
>     >     > ismael@juma.me.uk>
>     >     >         >> > wrote:
>     >     >         >> > > >
>     >     >         >> > > >> One comment below.
>     >     >         >> > > >>
>     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
> Gharat <
>     >     >         >> > > >> gharatmayuresh15@gmail.com
>     >     >         >> > > >> > wrote:
>     >     >         >> > > >>
>     >     >         >> > > >> >    - If we don't bump up the magic byte,
> on the
>     > broker
>     >     > side, the
>     >     >         >> > > broker
>     >     >         >> > > >> >    will always have to look at both
> tombstone
>     > bit and
>     >     > the value
>     >     >         >> when
>     >     >         >> > > do
>     >     >         >> > > >> the
>     >     >         >> > > >> >    compaction. Assuming we do not bump up
> the
>     > magic
>     >     > byte,
>     >     >         >> > > >> >    imagine the broker sees a message which
> does
>     > not
>     >     > have a
>     >     >         >> tombstone
>     >     >         >> > > bit
>     >     >         >> > > >> >    set. The broker does not know when the
>     > message was
>     >     > produced
>     >     >         >> (i.e.
>     >     >         >> > > >> > whether
>     >     >         >> > > >> >    the message has been up converted or
> not), it
>     > has
>     >     > to take a
>     >     >         >> > further
>     >     >         >> > > >> > look at
>     >     >         >> > > >> >    the value to see if it is null or not in
>     > order to
>     >     > determine
>     >     >         >> if it
>     >     >         >> > > is
>     >     >         >> > > >> a
>     >     >         >> > > >> >    tombstone. The same logic has to be put
> on the
>     >     > consumer as
>     >     >         >> well
>     >     >         >> > > >> because
>     >     >         >> > > >> > the
>     >     >         >> > > >> >    consumer does not know if the message
> has
>     > been up
>     >     > converted or
>     >     >         >> > not.
>     >     >         >> > > >> >       - If we upconvert while appending,
> this is
>     > not
>     >     > the case,
>     >     >         >> > right?
>     >     >         >> > > >>
>     >     >         >> > > >>
>     >     >         >> > > >> If I understand you correctly, this is not
>     > sufficient
>     >     > because the
>     >     >         >> log
>     >     >         >> > > may
>     >     >         >> > > >> have messages appended before it was
> upgraded to
>     > include
>     >     > KIP-87.
>     >     >         >> > > >>
>     >     >         >> > > >> Ismael
>     >     >         >> > > >>
>     >     >         >> > > >
>     >     >         >> > > >
>     >     >         >> > > >
>     >     >         >> > > > --
>     >     >         >> > > > -Regards,
>     >     >         >> > > > Mayuresh R. Gharat
>     >     >         >> > > > (862) 250-7125
>     >     >         >> > > >
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > >
>     >     >         >> > > --
>     >     >         >> > > -Regards,
>     >     >         >> > > Mayuresh R. Gharat
>     >     >         >> > > (862) 250-7125
>     >     >         >> > >
>     >     >         >> > The information contained in this email is
> strictly
>     >     > confidential and for
>     >     >         >> > the use of the addressee only, unless otherwise
>     > indicated. If
>     >     > you are
>     >     >         >> not
>     >     >         >> > the intended recipient, please do not read, copy,
> use or
>     >     > disclose to
>     >     >         >> others
>     >     >         >> > this message or any attachment. Please also
> notify the
>     > sender
>     >     > by
>     >     >         >> replying
>     >     >         >> > to this email or by telephone (+44(020 7896 0011)
> and
>     > then
>     >     > delete the
>     >     >         >> email
>     >     >         >> > and any copies of it. Opinions, conclusion (etc)
> that
>     > do not
>     >     > relate to
>     >     >         >> the
>     >     >         >> > official business of this company shall be
> understood as
>     >     > neither given
>     >     >         >> nor
>     >     >         >> > endorsed by it. IG is a trading name of IG Markets
>     > Limited (a
>     >     > company
>     >     >         >> > registered in England and Wales, company number
>     > 04008957) and
>     >     > IG Index
>     >     >         >> > Limited (a company registered in England and
> Wales,
>     > company
>     >     > number
>     >     >         >> > 01190902). Registered address at Cannon Bridge
> House, 25
>     >     > Dowgate Hill,
>     >     >         >> > London EC4R 2YA. Both IG Markets Limited (register
>     > number
>     >     > 195355) and IG
>     >     >         >> > Index Limited (register number 114059) are
> authorised
>     > and
>     >     > regulated by
>     >     >         >> the
>     >     >         >> > Financial Conduct Authority.
>     >     >         >> >
>     >     >         >>
>     >     >         >>
>     >     >         >>
>     >     >         >> --
>     >     >         >> -Regards,
>     >     >         >> Mayuresh R. Gharat
>     >     >         >> (862) 250-7125
>     >     >         >> The information contained in this email is strictly
>     >     > confidential and for
>     >     >         >> the use of the addressee only, unless otherwise
>     > indicated. If
>     >     > you are not
>     >     >         >> the intended recipient, please do not read, copy,
> use or
>     >     > disclose to others
>     >     >         >> this message or any attachment. Please also notify
> the
>     > sender
>     >     > by replying
>     >     >         >> to this email or by telephone (+44(020 7896 0011)
> and then
>     >     > delete the email
>     >     >         >> and any copies of it. Opinions, conclusion (etc)
> that do
>     > not
>     >     > relate to the
>     >     >         >> official business of this company shall be
> understood as
>     >     > neither given nor
>     >     >         >> endorsed by it. IG is a trading name of IG Markets
>     > Limited (a
>     >     > company
>     >     >         >> registered in England and Wales, company number
> 04008957)
>     > and
>     >     > IG Index
>     >     >         >> Limited (a company registered in England and Wales,
>     > company
>     >     > number
>     >     >         >> 01190902). Registered address at Cannon Bridge
> House, 25
>     >     > Dowgate Hill,
>     >     >         >> London EC4R 2YA. Both IG Markets Limited (register
> number
>     >     > 195355) and IG
>     >     >         >> Index Limited (register number 114059) are
> authorised and
>     >     > regulated by the
>     >     >         >> Financial Conduct Authority.
>     >     >         >>
>     >     >         >
>     >     >         >
>     >     >         >
>     >     >         > --
>     >     >         > -Regards,
>     >     >         > Mayuresh R. Gharat
>     >     >         > (862) 250-7125
>     >     >         >
>     >     >
>     >     >
>     >     >
>     >     >         --
>     >     >         -Regards,
>     >     >         Mayuresh R. Gharat
>     >     >         (862) 250-7125
>     >     >
>     >     >
>     >     >     The information contained in this email is strictly
> confidential
>     > and
>     >     > for the use of the addressee only, unless otherwise indicated.
> If
>     > you are
>     >     > not the intended recipient, please do not read, copy, use or
>     > disclose to
>     >     > others this message or any attachment. Please also notify the
> sender
>     > by
>     >     > replying to this email or by telephone (+44(020 7896 0011) and
> then
>     > delete
>     >     > the email and any copies of it. Opinions, conclusion (etc)
> that do
>     > not
>     >     > relate to the official business of this company shall be
> understood
>     > as
>     >     > neither given nor endorsed by it. IG is a trading name of IG
> Markets
>     >     > Limited (a company registered in England and Wales, company
> number
>     >     > 04008957) and IG Index Limited (a company registered in
> England and
>     > Wales,
>     >     > company number 01190902). Registered address at Cannon Bridge
> House,
>     > 25
>     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> (register
>     > number
>     >     > 195355) and IG Index Limited (register number 114059) are
> authorised
>     > and
>     >     > regulated by the Financial Conduct Authority.
>     >     >
>     >     >
>     >     >
>     >
>     >
>     > The information contained in this email is strictly confidential and
> for
>     > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > the intended recipient, please do not read, copy, use or disclose to
> others
>     > this message or any attachment. Please also notify the sender by
> replying
>     > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
>     > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
>     > official business of this company shall be understood as neither
> given nor
>     > endorsed by it. IG is a trading name of IG Markets Limited (a company
>     > registered in England and Wales, company number 04008957) and IG
> Index
>     > Limited (a company registered in England and Wales, company number
>     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > Index Limited (register number 114059) are authorised and regulated
> by the
>     > Financial Conduct Authority.
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jun

do we have your vote on this now?

Any other concerns?

Cheers
Mike

Sent using OWA for iPhone
________________________________________
From: Michael Pearce <Mi...@ig.com>
Sent: Saturday, December 3, 2016 1:37:45 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Jun,

Have updated. Thanks again for the feedback.

Agree yes we should align up when it gets to that, I assume you’ve flagged the same in the other KIP?

I think for now let’s get this KIP approved, then we can start the work to get an initial PR, then we can discuss how to align the two if needed.

Cheers,
Mike

On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:

    Hi, Michael,

    For 2), this is fine. Could you update the KIP wiki to make this and other
    points clearer? Other than that, the KIP looks good to me.

    An orthogonal thing is that there are other KIPs such as KIP-98 that also
    intend to change the message format. If they all get approved, we should
    think about whether it's better to just bump up the magic byte once to
    incorporate multiple format changes like we did in KIP-31/KIP-32.

    Thanks,

    Jun


    On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <Mi...@ig.com>
    wrote:

    > Hi Jao
    >
    > Thanks for the response. Sorry for slow reply, both with personal sickness
    > and also battling some critical issues encountered since upgrading to
    > 0.10.1.0
    >
    > 1) Thans for spotting, Document error where we branched this KIP from
    > KIP-82, will get that removed.
    > 2) Intent is to do this just at the record message level.
    > 3) Thanks for spotting, Will ensure this is corrected.
    > 4) As per discussion thread we will support tombstone + null value,
    > tombstone + non null value, no tombstone + null value.
    > 5) I believe this was in the discussion thread, @Mayuresh is this
    > something we’ve overlooked? I thought we would down convert and remove the
    > value so the old consumer had existing behavior, or is there something we
    > haven’t thought about?
    >
    > Cheers
    > Mike
    >
    > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
    >
    >     Hi, Michael,
    >
    >     Thanks for the KIP. A few comments below.
    >
    >     1. The message format change contains "HeadersLength Headers". Is that
    >     intended?
    >
    >     2. For compressed messageset, is the tombstone bit only set at the
    > shallow
    >     level? Do we always leave that bit in the wrapper message unset? An
    >     alternative is to set the tombstone bit in the wrapper if at least one
    >     inner message has the tombstone bit set. This makes things a bit more
    >     complicated, but we could potentially exploit that for optimizing down
    >     conversion. For example, we only need to convert messages with magic 2
    > to
    >     magic 1 if the wrapper's tombstone bit is set (conversion is always
    > needed
    >     from magic 2 to magic 0). Not sure if the optimization is worth the
    >     complexity though.
    >
    >     3. The referencing of the new version of ProducerRequest/FetchRequest
    > is
    >     inconsistent (v4 vs v3). Since our convention starts at version at 0, I
    >     think the new version would be 3.
    >
    >     4. "If the magic byte on message is 2, the broker should use the
    > tombstone
    >     bit for log compaction." What about null value? My understanding is
    > that
    >     null value will be treated the same as setting the tombstone bit.
    >
    >     5. For the migration path, it would be useful to describe the down
    >     conversion path to consumers (i.e., brokers on message format 0.10.2
    > and
    >     consumers on older version).
    >
    >     Thanks,
    >
    >     Jun
    >
    >
    >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <Michael.Pearce@ig.com
    > >
    >     wrote:
    >
    >     > Hi All,
    >     >
    >     > We have been discussing in the below thread and final changes have
    > been
    >     > made to the KIP wiki based on these discussions.
    >     >
    >     > We would now like to put to the vote the following KIP:
    >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
    >     > Add+Compaction+Tombstone+Flag
    >     >
    >     > This kip is for having a distinct compaction attribute “tombstone”
    > flag
    >     > instead of relying on null value, allowing non-null value delete
    > messages.
    >     >
    >     > Many thanks,
    >     > Michael
    >     >
    >     >
    >     >
    >     > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com>
    > wrote:
    >     >
    >     >     Hi Mayuresh,
    >     >
    >     >     LGTM. Ive just made one small adjustment updating the wire
    > protocol to
    >     > show the magic byte bump.
    >     >
    >     >     Do we think we’re good to put to a vote? Is there any other bits
    >     > needing discussion?
    >     >
    >     >     Cheers
    >     >     Mike
    >     >
    >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
    > gharatmayuresh15@gmail.com>
    >     > wrote:
    >     >
    >     >         Hi Michael,
    >     >
    >     >         I have updated the migration section of the KIP. Can you
    > please
    >     > take a look?
    >     >
    >     >         Thanks,
    >     >
    >     >         Mayuresh
    >     >
    >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
    >     > gharatmayuresh15@gmail.com
    >     >         > wrote:
    >     >
    >     >         > Hi Michael,
    >     >         >
    >     >         > That whilst sending tombstone and non null value, the
    > consumer
    >     > can expect
    >     >         > only to receive the non-null message only in step (3) is
    > this
    >     > correct?
    >     >         > ---> I do agree with you here.
    >     >         >
    >     >         > Becket, Ismael : can you guys review the migration plan
    > listed
    >     > above using
    >     >         > magic byte?
    >     >         >
    >     >         > Thanks,
    >     >         >
    >     >         > Mayuresh
    >     >         >
    >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
    >     > Michael.Pearce@ig.com>
    >     >         > wrote:
    >     >         >
    >     >         >> Many thanks for this Mayuresh. I don't have any
    > objections.
    >     >         >>
    >     >         >> I assume we should state:
    >     >         >>
    >     >         >> That whilst sending tombstone and non null value, the
    > consumer
    >     > can expect
    >     >         >> only to receive the non-null message only in step (3) is
    > this
    >     > correct?
    >     >         >>
    >     >         >> Cheers
    >     >         >> Mike
    >     >         >>
    >     >         >>
    >     >         >>
    >     >         >> Sent using OWA for iPhone
    >     >         >> ________________________________________
    >     >         >> From: Mayuresh Gharat <gh...@gmail.com>
    >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
    >     >         >> To: dev@kafka.apache.org
    >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone
    > Flag
    >     >         >>
    >     >         >> Hi Ismael,
    >     >         >>
    >     >         >> Thanks for the explanation.
    >     >         >> Specially I like this part where in you mentioned we can
    > get
    >     > rid of the
    >     >         >> older null value support for log compaction later on,
    > here :
    >     >         >> We can't change semantics of the message format without
    > having
    >     > a long
    >     >         >> transition period. And we can't rely
    >     >         >> on people reading documentation or acting on a warning for
    >     > something so
    >     >         >> fundamental. As such, my take is that we need to bump the
    > magic
    >     > byte. The
    >     >         >> good news is
    >     >         >> that we don't have to support all versions forever. We
    > have
    >     > said that we
    >     >         >> will support direct upgrades for 2 years. That means that
    >     > message format
    >     >         >> version n could, in theory, be removed 2 years after the
    > it's
    >     > introduced.
    >     >         >>
    >     >         >> Just a heads up, I would like to mention that even without
    >     > bumping magic
    >     >         >> byte, we will *NOT* loose zero copy as in the client(x+1)
    > in my
    >     >         >> explanation
    >     >         >> above will convert internally a null value to have a
    > tombstone
    >     > bit set and
    >     >         >> a tombstone bit set to have a null value automatically
    >     > internally and by
    >     >         >> the time we move to version (x+2), the clients would have
    >     > upgraded.
    >     >         >> Obviously if we support a request from consumer(x), we
    > will
    >     > loose zero
    >     >         >> copy
    >     >         >> but that is the same case with magic byte.
    >     >         >>
    >     >         >> But if magic byte bump makes life easier for transition
    > for the
    >     > above
    >     >         >> reasons that you explained, I am OK with it since we are
    > going
    >     > to meet the
    >     >         >> end goal down the road :)
    >     >         >>
    >     >         >> On a side note can we update the doc here on magic byte
    > to say
    >     > that "*it
    >     >         >> should be bumped whenever the message format is changed
    > or the
    >     >         >> interpretation of message format (usage of the reserved
    > bits as
    >     > well) is
    >     >         >> changed*".
    >     >         >>
    >     >         >>
    >     >         >> Hi Michael,
    >     >         >>
    >     >         >> Here is the update plan that we discussed offline
    > yesterday :
    >     >         >>
    >     >         >> Currently the magic-byte which corresponds to the
    >     > "message.format.version"
    >     >         >> is set to 1.
    >     >         >>
    >     >         >> 1) On broker it will be set to 1 initially.
    >     >         >>
    >     >         >> 2) When a producer client sends a message with magic-byte
    > = 2,
    >     > since the
    >     >         >> broker is on magic-byte = 1, we will down convert it,
    > which
    >     > means if the
    >     >         >> tombstone bit is set, the value will be set to null. A
    > consumer
    >     >         >> understanding magic-byte = 1, will still work with this. A
    >     > consumer
    >     >         >> working
    >     >         >> with magic-byte =2 will also be able to understand this,
    > since
    >     > it
    >     >         >> understands the tombstone.
    >     >         >> Now there is still the question of supporting a
    > non-tombstone
    >     > and null
    >     >         >> value from producer client with magic-byte = 2.* (I am
    > not sure
    >     > if we
    >     >         >> should support this. Ismael/Becket can comment here)*
    >     >         >>
    >     >         >> 3) When almost all the clients have upgraded, the
    >     > message.format.version
    >     >         >> on
    >     >         >> the broker can be changed to 2, where in the down
    > conversion in
    >     > the above
    >     >         >> step will not happen. If at this point we get a consumer
    >     > request from a
    >     >         >> older consumer, we might have to down convert where in we
    > loose
    >     > zero copy,
    >     >         >> but these cases should be rare.
    >     >         >>
    >     >         >> Becket can you review this plan and add more details if I
    > have
    >     >         >> missed/wronged something, before we put it on KIP.
    >     >         >>
    >     >         >> Thanks,
    >     >         >>
    >     >         >> Mayuresh
    >     >         >>
    >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
    >     > Michael.Pearce@ig.com>
    >     >         >> wrote:
    >     >         >>
    >     >         >> > Thanks guys, for discussing this offline and getting
    > some
    >     > consensus.
    >     >         >> >
    >     >         >> > So its clear for myself and others what is proposed now
    > (i
    >     > think i
    >     >         >> > understand, but want to make sure)
    >     >         >> >
    >     >         >> > Could i ask either directly update the kip to detail the
    >     > migration
    >     >         >> > strategy, or (re-)state your offline discussed and
    > agreed
    >     > migration
    >     >         >> > strategy based on a magic byte is in this thread.
    >     >         >> >
    >     >         >> >
    >     >         >> > The main original driver for the KIP was to support
    >     > compaction where
    >     >         >> value
    >     >         >> > isn't null, based off the discussions on KIP-82 thread.
    >     >         >> >
    >     >         >> > We should be able to support non-tombstone + null value
    > by the
    >     >         >> completion
    >     >         >> > of the KIP, as we noted when discussing this kip, having
    >     > logic based on
    >     >         >> a
    >     >         >> > null value isn't very clean and also separates the
    > concerns.
    >     >         >> >
    >     >         >> > As discussed already though we can split this into
    > KIP-87a
    >     > and KIP-87b
    >     >         >> >
    >     >         >> > Where we look to deliver KIP-87a on a compacted topic
    > (to
    >     > address the
    >     >         >> > immediate issues)
    >     >         >> > * tombstone + null value
    >     >         >> > * tombstone + non-null value
    >     >         >> > * non-tombstone + non-null value
    >     >         >> >
    >     >         >> > Then we can discuss once KIP-87a is completed options
    > later
    >     > and how we
    >     >         >> > support the second part KIP-87b to deliver:
    >     >         >> > * non-tombstone + null value
    >     >         >> >
    >     >         >> > Cheers
    >     >         >> > Mike
    >     >         >> >
    >     >         >> >
    >     >         >> >
    >     >         >> > ________________________________________
    >     >         >> > From: Becket Qin <be...@gmail.com>
    >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
    >     >         >> > To: dev@kafka.apache.org
    >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
    > Tombstone Flag
    >     >         >> >
    >     >         >> > Renu, Mayuresh and I had an offline discussion, and
    > following
    >     > is a brief
    >     >         >> > summary.
    >     >         >> >
    >     >         >> > 1. We agreed that not bumping up magic value may result
    > in
    >     > losing zero
    >     >         >> copy
    >     >         >> > during migration.
    >     >         >> > 2. Given that bumping up magic value is almost free and
    > has
    >     > benefit of
    >     >         >> > avoiding potential performance issue. It is probably
    > worth
    >     > doing.
    >     >         >> >
    >     >         >> > One issue we still need to think about is whether we
    > want to
    >     > support a
    >     >         >> > non-tombstone message with null value.
    >     >         >> > Currently it is not supported by Kafka. If we allow a
    >     > non-tombstone null
    >     >         >> > value message to exist after KIP-87. The problem is
    > that such
    >     > message
    >     >         >> will
    >     >         >> > not be supported by the consumers prior to KIP-87.
    > Because a
    >     > null value
    >     >         >> > will always be interpreted to a tombstone.
    >     >         >> >
    >     >         >> > One option is that we keep the current way, i.e. do not
    >     > support such
    >     >         >> > message. It would be good to know if there is a
    > concrete use
    >     > case for
    >     >         >> such
    >     >         >> > message. If there is not, we can probably just not
    > support it.
    >     >         >> >
    >     >         >> > Thanks,
    >     >         >> >
    >     >         >> > JIangjie (Becket) Qin
    >     >         >> >
    >     >         >> >
    >     >         >> >
    >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
    >     >         >> > gharatmayuresh15@gmail.com
    >     >         >> > > wrote:
    >     >         >> >
    >     >         >> > > Hi Ismael,
    >     >         >> > >
    >     >         >> > > This is something I can think of for migration plan:
    >     >         >> > > So the migration plan can look something like this,
    > with up
    >     >         >> conversion :
    >     >         >> > >
    >     >         >> > > 1) Currently lets say we have Broker at version x.
    >     >         >> > > 2) Currently we have clients at version x.
    >     >         >> > > 3) a) We move the version to Broker(x+1) : supports
    > both
    >     > tombstone and
    >     >         >> > null
    >     >         >> > > for log compaction.
    >     >         >> > >     b) We upgrade the client to version client(x+1) :
    > if in
    >     > the
    >     >         >> producer
    >     >         >> > > client(x+1) the value is set to null, we will
    > automatically
    >     > set the
    >     >         >> > > Tombstone bit internally. If the producer client(x+1)
    > sets
    >     > the
    >     >         >> tombstone
    >     >         >> > > itself, well and good. For producer client(x), the
    > broker
    >     > will up
    >     >         >> convert
    >     >         >> > > to have the tombstone bit. Broker(x+1) is supporting
    > both.
    >     > Consumer
    >     >         >> > > client(x+1) will be aware of this and should be able
    > to
    >     > handle this.
    >     >         >> For
    >     >         >> > > consumer client(x) we will down convert the message
    > on the
    >     > broker
    >     >         >> side.
    >     >         >> > >     c) At this point we will have to specify a
    > warning or
    >     > clearly
    >     >         >> specify
    >     >         >> > > in docs that this behavior is about to be changed for
    > log
    >     > compaction.
    >     >         >> > > 4) a) In next release of the Broker(x+2), we say that
    > only
    >     > Tombstone
    >     >         >> is
    >     >         >> > > used for log compaction on the Broker side.
    > Clients(x+1)
    >     > still is
    >     >         >> > > supported.
    >     >         >> > >     b) We upgrade the client to version client(x+2) :
    > if
    >     > value is set
    >     >         >> to
    >     >         >> > > null, tombstone will not be set automatically. The
    > client
    >     > will have to
    >     >         >> > call
    >     >         >> > > setTombstone() to actually set the tombstone.
    >     >         >> > >
    >     >         >> > > We should compare this migration plan with the
    > migration
    >     > plan for
    >     >         >> magic
    >     >         >> > > byte bump and do whatever looks good.
    >     >         >> > > I am just worried that if we go down magic byte route,
    >     > unless I am
    >     >         >> > missing
    >     >         >> > > something, it sounds like kafka will be stuck with
    >     > supporting both
    >     >         >> null
    >     >         >> > > value and tombstone bit for log compaction for life
    > long,
    >     > which does
    >     >         >> not
    >     >         >> > > look like a good end state.
    >     >         >> > >
    >     >         >> > > Thanks,
    >     >         >> > >
    >     >         >> > > Mayuresh
    >     >         >> > >
    >     >         >> > >
    >     >         >> > >
    >     >         >> > >
    >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
    >     >         >> > > gharatmayuresh15@gmail.com
    >     >         >> > > > wrote:
    >     >         >> > >
    >     >         >> > > > Hi Ismael,
    >     >         >> > > >
    >     >         >> > > > That's a very good point which I might have not
    >     > considered earlier.
    >     >         >> > > >
    >     >         >> > > > Here is a plan that I can think of:
    >     >         >> > > >
    >     >         >> > > > Stage 1) The broker from now on, up converts the
    > message
    >     > to have the
    >     >         >> > > > tombstone marker. The log compaction thread does log
    >     > compaction
    >     >         >> based
    >     >         >> > on
    >     >         >> > > > both null and tombstone marker. This is our
    > transition
    >     > period.
    >     >         >> > > > Stage 2) The next release we only say that log
    > compaction
    >     > is based
    >     >         >> on
    >     >         >> > > > tombstone marker. (Open source kafka makes this as a
    >     > policy). By
    >     >         >> this
    >     >         >> > > time,
    >     >         >> > > > the organization which is moving to this release
    > will be
    >     > sure that
    >     >         >> they
    >     >         >> > > > have gone through the entire transition period.
    >     >         >> > > >
    >     >         >> > > > My only goal of doing this is that Kafka clearly
    >     > specifies the end
    >     >         >> > state
    >     >         >> > > > about what log compaction means (is it null value
    > or a
    >     > tombstone
    >     >         >> > marker,
    >     >         >> > > > but not both).
    >     >         >> > > >
    >     >         >> > > > What do you think?
    >     >         >> > > >
    >     >         >> > > > Thanks,
    >     >         >> > > >
    >     >         >> > > > Mayuresh
    >     >         >> > > > .
    >     >         >> > > >
    >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
    >     > ismael@juma.me.uk>
    >     >         >> > wrote:
    >     >         >> > > >
    >     >         >> > > >> One comment below.
    >     >         >> > > >>
    >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
    >     >         >> > > >> gharatmayuresh15@gmail.com
    >     >         >> > > >> > wrote:
    >     >         >> > > >>
    >     >         >> > > >> >    - If we don't bump up the magic byte, on the
    > broker
    >     > side, the
    >     >         >> > > broker
    >     >         >> > > >> >    will always have to look at both tombstone
    > bit and
    >     > the value
    >     >         >> when
    >     >         >> > > do
    >     >         >> > > >> the
    >     >         >> > > >> >    compaction. Assuming we do not bump up the
    > magic
    >     > byte,
    >     >         >> > > >> >    imagine the broker sees a message which does
    > not
    >     > have a
    >     >         >> tombstone
    >     >         >> > > bit
    >     >         >> > > >> >    set. The broker does not know when the
    > message was
    >     > produced
    >     >         >> (i.e.
    >     >         >> > > >> > whether
    >     >         >> > > >> >    the message has been up converted or not), it
    > has
    >     > to take a
    >     >         >> > further
    >     >         >> > > >> > look at
    >     >         >> > > >> >    the value to see if it is null or not in
    > order to
    >     > determine
    >     >         >> if it
    >     >         >> > > is
    >     >         >> > > >> a
    >     >         >> > > >> >    tombstone. The same logic has to be put on the
    >     > consumer as
    >     >         >> well
    >     >         >> > > >> because
    >     >         >> > > >> > the
    >     >         >> > > >> >    consumer does not know if the message has
    > been up
    >     > converted or
    >     >         >> > not.
    >     >         >> > > >> >       - If we upconvert while appending, this is
    > not
    >     > the case,
    >     >         >> > right?
    >     >         >> > > >>
    >     >         >> > > >>
    >     >         >> > > >> If I understand you correctly, this is not
    > sufficient
    >     > because the
    >     >         >> log
    >     >         >> > > may
    >     >         >> > > >> have messages appended before it was upgraded to
    > include
    >     > KIP-87.
    >     >         >> > > >>
    >     >         >> > > >> Ismael
    >     >         >> > > >>
    >     >         >> > > >
    >     >         >> > > >
    >     >         >> > > >
    >     >         >> > > > --
    >     >         >> > > > -Regards,
    >     >         >> > > > Mayuresh R. Gharat
    >     >         >> > > > (862) 250-7125
    >     >         >> > > >
    >     >         >> > >
    >     >         >> > >
    >     >         >> > >
    >     >         >> > > --
    >     >         >> > > -Regards,
    >     >         >> > > Mayuresh R. Gharat
    >     >         >> > > (862) 250-7125
    >     >         >> > >
    >     >         >> > The information contained in this email is strictly
    >     > confidential and for
    >     >         >> > the use of the addressee only, unless otherwise
    > indicated. If
    >     > you are
    >     >         >> not
    >     >         >> > the intended recipient, please do not read, copy, use or
    >     > disclose to
    >     >         >> others
    >     >         >> > this message or any attachment. Please also notify the
    > sender
    >     > by
    >     >         >> replying
    >     >         >> > to this email or by telephone (+44(020 7896 0011) and
    > then
    >     > delete the
    >     >         >> email
    >     >         >> > and any copies of it. Opinions, conclusion (etc) that
    > do not
    >     > relate to
    >     >         >> the
    >     >         >> > official business of this company shall be understood as
    >     > neither given
    >     >         >> nor
    >     >         >> > endorsed by it. IG is a trading name of IG Markets
    > Limited (a
    >     > company
    >     >         >> > registered in England and Wales, company number
    > 04008957) and
    >     > IG Index
    >     >         >> > Limited (a company registered in England and Wales,
    > company
    >     > number
    >     >         >> > 01190902). Registered address at Cannon Bridge House, 25
    >     > Dowgate Hill,
    >     >         >> > London EC4R 2YA. Both IG Markets Limited (register
    > number
    >     > 195355) and IG
    >     >         >> > Index Limited (register number 114059) are authorised
    > and
    >     > regulated by
    >     >         >> the
    >     >         >> > Financial Conduct Authority.
    >     >         >> >
    >     >         >>
    >     >         >>
    >     >         >>
    >     >         >> --
    >     >         >> -Regards,
    >     >         >> Mayuresh R. Gharat
    >     >         >> (862) 250-7125
    >     >         >> The information contained in this email is strictly
    >     > confidential and for
    >     >         >> the use of the addressee only, unless otherwise
    > indicated. If
    >     > you are not
    >     >         >> the intended recipient, please do not read, copy, use or
    >     > disclose to others
    >     >         >> this message or any attachment. Please also notify the
    > sender
    >     > by replying
    >     >         >> to this email or by telephone (+44(020 7896 0011) and then
    >     > delete the email
    >     >         >> and any copies of it. Opinions, conclusion (etc) that do
    > not
    >     > relate to the
    >     >         >> official business of this company shall be understood as
    >     > neither given nor
    >     >         >> endorsed by it. IG is a trading name of IG Markets
    > Limited (a
    >     > company
    >     >         >> registered in England and Wales, company number 04008957)
    > and
    >     > IG Index
    >     >         >> Limited (a company registered in England and Wales,
    > company
    >     > number
    >     >         >> 01190902). Registered address at Cannon Bridge House, 25
    >     > Dowgate Hill,
    >     >         >> London EC4R 2YA. Both IG Markets Limited (register number
    >     > 195355) and IG
    >     >         >> Index Limited (register number 114059) are authorised and
    >     > regulated by the
    >     >         >> Financial Conduct Authority.
    >     >         >>
    >     >         >
    >     >         >
    >     >         >
    >     >         > --
    >     >         > -Regards,
    >     >         > Mayuresh R. Gharat
    >     >         > (862) 250-7125
    >     >         >
    >     >
    >     >
    >     >
    >     >         --
    >     >         -Regards,
    >     >         Mayuresh R. Gharat
    >     >         (862) 250-7125
    >     >
    >     >
    >     >     The information contained in this email is strictly confidential
    > and
    >     > for the use of the addressee only, unless otherwise indicated. If
    > you are
    >     > not the intended recipient, please do not read, copy, use or
    > disclose to
    >     > others this message or any attachment. Please also notify the sender
    > by
    >     > replying to this email or by telephone (+44(020 7896 0011) and then
    > delete
    >     > the email and any copies of it. Opinions, conclusion (etc) that do
    > not
    >     > relate to the official business of this company shall be understood
    > as
    >     > neither given nor endorsed by it. IG is a trading name of IG Markets
    >     > Limited (a company registered in England and Wales, company number
    >     > 04008957) and IG Index Limited (a company registered in England and
    > Wales,
    >     > company number 01190902). Registered address at Cannon Bridge House,
    > 25
    >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
    > number
    >     > 195355) and IG Index Limited (register number 114059) are authorised
    > and
    >     > regulated by the Financial Conduct Authority.
    >     >
    >     >
    >     >
    >
    >
    > The information contained in this email is strictly confidential and for
    > the use of the addressee only, unless otherwise indicated. If you are not
    > the intended recipient, please do not read, copy, use or disclose to others
    > this message or any attachment. Please also notify the sender by replying
    > to this email or by telephone (+44(020 7896 0011) and then delete the email
    > and any copies of it. Opinions, conclusion (etc) that do not relate to the
    > official business of this company shall be understood as neither given nor
    > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > registered in England and Wales, company number 04008957) and IG Index
    > Limited (a company registered in England and Wales, company number
    > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    > Index Limited (register number 114059) are authorised and regulated by the
    > Financial Conduct Authority.
    >


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jun,

Have updated. Thanks again for the feedback.

Agree yes we should align up when it gets to that, I assume you’ve flagged the same in the other KIP?

I think for now let’s get this KIP approved, then we can start the work to get an initial PR, then we can discuss how to align the two if needed.

Cheers,
Mike

On 02/12/2016, 18:26, "Jun Rao" <ju...@confluent.io> wrote:

    Hi, Michael,

    For 2), this is fine. Could you update the KIP wiki to make this and other
    points clearer? Other than that, the KIP looks good to me.

    An orthogonal thing is that there are other KIPs such as KIP-98 that also
    intend to change the message format. If they all get approved, we should
    think about whether it's better to just bump up the magic byte once to
    incorporate multiple format changes like we did in KIP-31/KIP-32.

    Thanks,

    Jun


    On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <Mi...@ig.com>
    wrote:

    > Hi Jao
    >
    > Thanks for the response. Sorry for slow reply, both with personal sickness
    > and also battling some critical issues encountered since upgrading to
    > 0.10.1.0
    >
    > 1) Thans for spotting, Document error where we branched this KIP from
    > KIP-82, will get that removed.
    > 2) Intent is to do this just at the record message level.
    > 3) Thanks for spotting, Will ensure this is corrected.
    > 4) As per discussion thread we will support tombstone + null value,
    > tombstone + non null value, no tombstone + null value.
    > 5) I believe this was in the discussion thread, @Mayuresh is this
    > something we’ve overlooked? I thought we would down convert and remove the
    > value so the old consumer had existing behavior, or is there something we
    > haven’t thought about?
    >
    > Cheers
    > Mike
    >
    > On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
    >
    >     Hi, Michael,
    >
    >     Thanks for the KIP. A few comments below.
    >
    >     1. The message format change contains "HeadersLength Headers". Is that
    >     intended?
    >
    >     2. For compressed messageset, is the tombstone bit only set at the
    > shallow
    >     level? Do we always leave that bit in the wrapper message unset? An
    >     alternative is to set the tombstone bit in the wrapper if at least one
    >     inner message has the tombstone bit set. This makes things a bit more
    >     complicated, but we could potentially exploit that for optimizing down
    >     conversion. For example, we only need to convert messages with magic 2
    > to
    >     magic 1 if the wrapper's tombstone bit is set (conversion is always
    > needed
    >     from magic 2 to magic 0). Not sure if the optimization is worth the
    >     complexity though.
    >
    >     3. The referencing of the new version of ProducerRequest/FetchRequest
    > is
    >     inconsistent (v4 vs v3). Since our convention starts at version at 0, I
    >     think the new version would be 3.
    >
    >     4. "If the magic byte on message is 2, the broker should use the
    > tombstone
    >     bit for log compaction." What about null value? My understanding is
    > that
    >     null value will be treated the same as setting the tombstone bit.
    >
    >     5. For the migration path, it would be useful to describe the down
    >     conversion path to consumers (i.e., brokers on message format 0.10.2
    > and
    >     consumers on older version).
    >
    >     Thanks,
    >
    >     Jun
    >
    >
    >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <Michael.Pearce@ig.com
    > >
    >     wrote:
    >
    >     > Hi All,
    >     >
    >     > We have been discussing in the below thread and final changes have
    > been
    >     > made to the KIP wiki based on these discussions.
    >     >
    >     > We would now like to put to the vote the following KIP:
    >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
    >     > Add+Compaction+Tombstone+Flag
    >     >
    >     > This kip is for having a distinct compaction attribute “tombstone”
    > flag
    >     > instead of relying on null value, allowing non-null value delete
    > messages.
    >     >
    >     > Many thanks,
    >     > Michael
    >     >
    >     >
    >     >
    >     > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com>
    > wrote:
    >     >
    >     >     Hi Mayuresh,
    >     >
    >     >     LGTM. Ive just made one small adjustment updating the wire
    > protocol to
    >     > show the magic byte bump.
    >     >
    >     >     Do we think we’re good to put to a vote? Is there any other bits
    >     > needing discussion?
    >     >
    >     >     Cheers
    >     >     Mike
    >     >
    >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
    > gharatmayuresh15@gmail.com>
    >     > wrote:
    >     >
    >     >         Hi Michael,
    >     >
    >     >         I have updated the migration section of the KIP. Can you
    > please
    >     > take a look?
    >     >
    >     >         Thanks,
    >     >
    >     >         Mayuresh
    >     >
    >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
    >     > gharatmayuresh15@gmail.com
    >     >         > wrote:
    >     >
    >     >         > Hi Michael,
    >     >         >
    >     >         > That whilst sending tombstone and non null value, the
    > consumer
    >     > can expect
    >     >         > only to receive the non-null message only in step (3) is
    > this
    >     > correct?
    >     >         > ---> I do agree with you here.
    >     >         >
    >     >         > Becket, Ismael : can you guys review the migration plan
    > listed
    >     > above using
    >     >         > magic byte?
    >     >         >
    >     >         > Thanks,
    >     >         >
    >     >         > Mayuresh
    >     >         >
    >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
    >     > Michael.Pearce@ig.com>
    >     >         > wrote:
    >     >         >
    >     >         >> Many thanks for this Mayuresh. I don't have any
    > objections.
    >     >         >>
    >     >         >> I assume we should state:
    >     >         >>
    >     >         >> That whilst sending tombstone and non null value, the
    > consumer
    >     > can expect
    >     >         >> only to receive the non-null message only in step (3) is
    > this
    >     > correct?
    >     >         >>
    >     >         >> Cheers
    >     >         >> Mike
    >     >         >>
    >     >         >>
    >     >         >>
    >     >         >> Sent using OWA for iPhone
    >     >         >> ________________________________________
    >     >         >> From: Mayuresh Gharat <gh...@gmail.com>
    >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
    >     >         >> To: dev@kafka.apache.org
    >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone
    > Flag
    >     >         >>
    >     >         >> Hi Ismael,
    >     >         >>
    >     >         >> Thanks for the explanation.
    >     >         >> Specially I like this part where in you mentioned we can
    > get
    >     > rid of the
    >     >         >> older null value support for log compaction later on,
    > here :
    >     >         >> We can't change semantics of the message format without
    > having
    >     > a long
    >     >         >> transition period. And we can't rely
    >     >         >> on people reading documentation or acting on a warning for
    >     > something so
    >     >         >> fundamental. As such, my take is that we need to bump the
    > magic
    >     > byte. The
    >     >         >> good news is
    >     >         >> that we don't have to support all versions forever. We
    > have
    >     > said that we
    >     >         >> will support direct upgrades for 2 years. That means that
    >     > message format
    >     >         >> version n could, in theory, be removed 2 years after the
    > it's
    >     > introduced.
    >     >         >>
    >     >         >> Just a heads up, I would like to mention that even without
    >     > bumping magic
    >     >         >> byte, we will *NOT* loose zero copy as in the client(x+1)
    > in my
    >     >         >> explanation
    >     >         >> above will convert internally a null value to have a
    > tombstone
    >     > bit set and
    >     >         >> a tombstone bit set to have a null value automatically
    >     > internally and by
    >     >         >> the time we move to version (x+2), the clients would have
    >     > upgraded.
    >     >         >> Obviously if we support a request from consumer(x), we
    > will
    >     > loose zero
    >     >         >> copy
    >     >         >> but that is the same case with magic byte.
    >     >         >>
    >     >         >> But if magic byte bump makes life easier for transition
    > for the
    >     > above
    >     >         >> reasons that you explained, I am OK with it since we are
    > going
    >     > to meet the
    >     >         >> end goal down the road :)
    >     >         >>
    >     >         >> On a side note can we update the doc here on magic byte
    > to say
    >     > that "*it
    >     >         >> should be bumped whenever the message format is changed
    > or the
    >     >         >> interpretation of message format (usage of the reserved
    > bits as
    >     > well) is
    >     >         >> changed*".
    >     >         >>
    >     >         >>
    >     >         >> Hi Michael,
    >     >         >>
    >     >         >> Here is the update plan that we discussed offline
    > yesterday :
    >     >         >>
    >     >         >> Currently the magic-byte which corresponds to the
    >     > "message.format.version"
    >     >         >> is set to 1.
    >     >         >>
    >     >         >> 1) On broker it will be set to 1 initially.
    >     >         >>
    >     >         >> 2) When a producer client sends a message with magic-byte
    > = 2,
    >     > since the
    >     >         >> broker is on magic-byte = 1, we will down convert it,
    > which
    >     > means if the
    >     >         >> tombstone bit is set, the value will be set to null. A
    > consumer
    >     >         >> understanding magic-byte = 1, will still work with this. A
    >     > consumer
    >     >         >> working
    >     >         >> with magic-byte =2 will also be able to understand this,
    > since
    >     > it
    >     >         >> understands the tombstone.
    >     >         >> Now there is still the question of supporting a
    > non-tombstone
    >     > and null
    >     >         >> value from producer client with magic-byte = 2.* (I am
    > not sure
    >     > if we
    >     >         >> should support this. Ismael/Becket can comment here)*
    >     >         >>
    >     >         >> 3) When almost all the clients have upgraded, the
    >     > message.format.version
    >     >         >> on
    >     >         >> the broker can be changed to 2, where in the down
    > conversion in
    >     > the above
    >     >         >> step will not happen. If at this point we get a consumer
    >     > request from a
    >     >         >> older consumer, we might have to down convert where in we
    > loose
    >     > zero copy,
    >     >         >> but these cases should be rare.
    >     >         >>
    >     >         >> Becket can you review this plan and add more details if I
    > have
    >     >         >> missed/wronged something, before we put it on KIP.
    >     >         >>
    >     >         >> Thanks,
    >     >         >>
    >     >         >> Mayuresh
    >     >         >>
    >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
    >     > Michael.Pearce@ig.com>
    >     >         >> wrote:
    >     >         >>
    >     >         >> > Thanks guys, for discussing this offline and getting
    > some
    >     > consensus.
    >     >         >> >
    >     >         >> > So its clear for myself and others what is proposed now
    > (i
    >     > think i
    >     >         >> > understand, but want to make sure)
    >     >         >> >
    >     >         >> > Could i ask either directly update the kip to detail the
    >     > migration
    >     >         >> > strategy, or (re-)state your offline discussed and
    > agreed
    >     > migration
    >     >         >> > strategy based on a magic byte is in this thread.
    >     >         >> >
    >     >         >> >
    >     >         >> > The main original driver for the KIP was to support
    >     > compaction where
    >     >         >> value
    >     >         >> > isn't null, based off the discussions on KIP-82 thread.
    >     >         >> >
    >     >         >> > We should be able to support non-tombstone + null value
    > by the
    >     >         >> completion
    >     >         >> > of the KIP, as we noted when discussing this kip, having
    >     > logic based on
    >     >         >> a
    >     >         >> > null value isn't very clean and also separates the
    > concerns.
    >     >         >> >
    >     >         >> > As discussed already though we can split this into
    > KIP-87a
    >     > and KIP-87b
    >     >         >> >
    >     >         >> > Where we look to deliver KIP-87a on a compacted topic
    > (to
    >     > address the
    >     >         >> > immediate issues)
    >     >         >> > * tombstone + null value
    >     >         >> > * tombstone + non-null value
    >     >         >> > * non-tombstone + non-null value
    >     >         >> >
    >     >         >> > Then we can discuss once KIP-87a is completed options
    > later
    >     > and how we
    >     >         >> > support the second part KIP-87b to deliver:
    >     >         >> > * non-tombstone + null value
    >     >         >> >
    >     >         >> > Cheers
    >     >         >> > Mike
    >     >         >> >
    >     >         >> >
    >     >         >> >
    >     >         >> > ________________________________________
    >     >         >> > From: Becket Qin <be...@gmail.com>
    >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
    >     >         >> > To: dev@kafka.apache.org
    >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
    > Tombstone Flag
    >     >         >> >
    >     >         >> > Renu, Mayuresh and I had an offline discussion, and
    > following
    >     > is a brief
    >     >         >> > summary.
    >     >         >> >
    >     >         >> > 1. We agreed that not bumping up magic value may result
    > in
    >     > losing zero
    >     >         >> copy
    >     >         >> > during migration.
    >     >         >> > 2. Given that bumping up magic value is almost free and
    > has
    >     > benefit of
    >     >         >> > avoiding potential performance issue. It is probably
    > worth
    >     > doing.
    >     >         >> >
    >     >         >> > One issue we still need to think about is whether we
    > want to
    >     > support a
    >     >         >> > non-tombstone message with null value.
    >     >         >> > Currently it is not supported by Kafka. If we allow a
    >     > non-tombstone null
    >     >         >> > value message to exist after KIP-87. The problem is
    > that such
    >     > message
    >     >         >> will
    >     >         >> > not be supported by the consumers prior to KIP-87.
    > Because a
    >     > null value
    >     >         >> > will always be interpreted to a tombstone.
    >     >         >> >
    >     >         >> > One option is that we keep the current way, i.e. do not
    >     > support such
    >     >         >> > message. It would be good to know if there is a
    > concrete use
    >     > case for
    >     >         >> such
    >     >         >> > message. If there is not, we can probably just not
    > support it.
    >     >         >> >
    >     >         >> > Thanks,
    >     >         >> >
    >     >         >> > JIangjie (Becket) Qin
    >     >         >> >
    >     >         >> >
    >     >         >> >
    >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
    >     >         >> > gharatmayuresh15@gmail.com
    >     >         >> > > wrote:
    >     >         >> >
    >     >         >> > > Hi Ismael,
    >     >         >> > >
    >     >         >> > > This is something I can think of for migration plan:
    >     >         >> > > So the migration plan can look something like this,
    > with up
    >     >         >> conversion :
    >     >         >> > >
    >     >         >> > > 1) Currently lets say we have Broker at version x.
    >     >         >> > > 2) Currently we have clients at version x.
    >     >         >> > > 3) a) We move the version to Broker(x+1) : supports
    > both
    >     > tombstone and
    >     >         >> > null
    >     >         >> > > for log compaction.
    >     >         >> > >     b) We upgrade the client to version client(x+1) :
    > if in
    >     > the
    >     >         >> producer
    >     >         >> > > client(x+1) the value is set to null, we will
    > automatically
    >     > set the
    >     >         >> > > Tombstone bit internally. If the producer client(x+1)
    > sets
    >     > the
    >     >         >> tombstone
    >     >         >> > > itself, well and good. For producer client(x), the
    > broker
    >     > will up
    >     >         >> convert
    >     >         >> > > to have the tombstone bit. Broker(x+1) is supporting
    > both.
    >     > Consumer
    >     >         >> > > client(x+1) will be aware of this and should be able
    > to
    >     > handle this.
    >     >         >> For
    >     >         >> > > consumer client(x) we will down convert the message
    > on the
    >     > broker
    >     >         >> side.
    >     >         >> > >     c) At this point we will have to specify a
    > warning or
    >     > clearly
    >     >         >> specify
    >     >         >> > > in docs that this behavior is about to be changed for
    > log
    >     > compaction.
    >     >         >> > > 4) a) In next release of the Broker(x+2), we say that
    > only
    >     > Tombstone
    >     >         >> is
    >     >         >> > > used for log compaction on the Broker side.
    > Clients(x+1)
    >     > still is
    >     >         >> > > supported.
    >     >         >> > >     b) We upgrade the client to version client(x+2) :
    > if
    >     > value is set
    >     >         >> to
    >     >         >> > > null, tombstone will not be set automatically. The
    > client
    >     > will have to
    >     >         >> > call
    >     >         >> > > setTombstone() to actually set the tombstone.
    >     >         >> > >
    >     >         >> > > We should compare this migration plan with the
    > migration
    >     > plan for
    >     >         >> magic
    >     >         >> > > byte bump and do whatever looks good.
    >     >         >> > > I am just worried that if we go down magic byte route,
    >     > unless I am
    >     >         >> > missing
    >     >         >> > > something, it sounds like kafka will be stuck with
    >     > supporting both
    >     >         >> null
    >     >         >> > > value and tombstone bit for log compaction for life
    > long,
    >     > which does
    >     >         >> not
    >     >         >> > > look like a good end state.
    >     >         >> > >
    >     >         >> > > Thanks,
    >     >         >> > >
    >     >         >> > > Mayuresh
    >     >         >> > >
    >     >         >> > >
    >     >         >> > >
    >     >         >> > >
    >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
    >     >         >> > > gharatmayuresh15@gmail.com
    >     >         >> > > > wrote:
    >     >         >> > >
    >     >         >> > > > Hi Ismael,
    >     >         >> > > >
    >     >         >> > > > That's a very good point which I might have not
    >     > considered earlier.
    >     >         >> > > >
    >     >         >> > > > Here is a plan that I can think of:
    >     >         >> > > >
    >     >         >> > > > Stage 1) The broker from now on, up converts the
    > message
    >     > to have the
    >     >         >> > > > tombstone marker. The log compaction thread does log
    >     > compaction
    >     >         >> based
    >     >         >> > on
    >     >         >> > > > both null and tombstone marker. This is our
    > transition
    >     > period.
    >     >         >> > > > Stage 2) The next release we only say that log
    > compaction
    >     > is based
    >     >         >> on
    >     >         >> > > > tombstone marker. (Open source kafka makes this as a
    >     > policy). By
    >     >         >> this
    >     >         >> > > time,
    >     >         >> > > > the organization which is moving to this release
    > will be
    >     > sure that
    >     >         >> they
    >     >         >> > > > have gone through the entire transition period.
    >     >         >> > > >
    >     >         >> > > > My only goal of doing this is that Kafka clearly
    >     > specifies the end
    >     >         >> > state
    >     >         >> > > > about what log compaction means (is it null value
    > or a
    >     > tombstone
    >     >         >> > marker,
    >     >         >> > > > but not both).
    >     >         >> > > >
    >     >         >> > > > What do you think?
    >     >         >> > > >
    >     >         >> > > > Thanks,
    >     >         >> > > >
    >     >         >> > > > Mayuresh
    >     >         >> > > > .
    >     >         >> > > >
    >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
    >     > ismael@juma.me.uk>
    >     >         >> > wrote:
    >     >         >> > > >
    >     >         >> > > >> One comment below.
    >     >         >> > > >>
    >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
    >     >         >> > > >> gharatmayuresh15@gmail.com
    >     >         >> > > >> > wrote:
    >     >         >> > > >>
    >     >         >> > > >> >    - If we don't bump up the magic byte, on the
    > broker
    >     > side, the
    >     >         >> > > broker
    >     >         >> > > >> >    will always have to look at both tombstone
    > bit and
    >     > the value
    >     >         >> when
    >     >         >> > > do
    >     >         >> > > >> the
    >     >         >> > > >> >    compaction. Assuming we do not bump up the
    > magic
    >     > byte,
    >     >         >> > > >> >    imagine the broker sees a message which does
    > not
    >     > have a
    >     >         >> tombstone
    >     >         >> > > bit
    >     >         >> > > >> >    set. The broker does not know when the
    > message was
    >     > produced
    >     >         >> (i.e.
    >     >         >> > > >> > whether
    >     >         >> > > >> >    the message has been up converted or not), it
    > has
    >     > to take a
    >     >         >> > further
    >     >         >> > > >> > look at
    >     >         >> > > >> >    the value to see if it is null or not in
    > order to
    >     > determine
    >     >         >> if it
    >     >         >> > > is
    >     >         >> > > >> a
    >     >         >> > > >> >    tombstone. The same logic has to be put on the
    >     > consumer as
    >     >         >> well
    >     >         >> > > >> because
    >     >         >> > > >> > the
    >     >         >> > > >> >    consumer does not know if the message has
    > been up
    >     > converted or
    >     >         >> > not.
    >     >         >> > > >> >       - If we upconvert while appending, this is
    > not
    >     > the case,
    >     >         >> > right?
    >     >         >> > > >>
    >     >         >> > > >>
    >     >         >> > > >> If I understand you correctly, this is not
    > sufficient
    >     > because the
    >     >         >> log
    >     >         >> > > may
    >     >         >> > > >> have messages appended before it was upgraded to
    > include
    >     > KIP-87.
    >     >         >> > > >>
    >     >         >> > > >> Ismael
    >     >         >> > > >>
    >     >         >> > > >
    >     >         >> > > >
    >     >         >> > > >
    >     >         >> > > > --
    >     >         >> > > > -Regards,
    >     >         >> > > > Mayuresh R. Gharat
    >     >         >> > > > (862) 250-7125
    >     >         >> > > >
    >     >         >> > >
    >     >         >> > >
    >     >         >> > >
    >     >         >> > > --
    >     >         >> > > -Regards,
    >     >         >> > > Mayuresh R. Gharat
    >     >         >> > > (862) 250-7125
    >     >         >> > >
    >     >         >> > The information contained in this email is strictly
    >     > confidential and for
    >     >         >> > the use of the addressee only, unless otherwise
    > indicated. If
    >     > you are
    >     >         >> not
    >     >         >> > the intended recipient, please do not read, copy, use or
    >     > disclose to
    >     >         >> others
    >     >         >> > this message or any attachment. Please also notify the
    > sender
    >     > by
    >     >         >> replying
    >     >         >> > to this email or by telephone (+44(020 7896 0011) and
    > then
    >     > delete the
    >     >         >> email
    >     >         >> > and any copies of it. Opinions, conclusion (etc) that
    > do not
    >     > relate to
    >     >         >> the
    >     >         >> > official business of this company shall be understood as
    >     > neither given
    >     >         >> nor
    >     >         >> > endorsed by it. IG is a trading name of IG Markets
    > Limited (a
    >     > company
    >     >         >> > registered in England and Wales, company number
    > 04008957) and
    >     > IG Index
    >     >         >> > Limited (a company registered in England and Wales,
    > company
    >     > number
    >     >         >> > 01190902). Registered address at Cannon Bridge House, 25
    >     > Dowgate Hill,
    >     >         >> > London EC4R 2YA. Both IG Markets Limited (register
    > number
    >     > 195355) and IG
    >     >         >> > Index Limited (register number 114059) are authorised
    > and
    >     > regulated by
    >     >         >> the
    >     >         >> > Financial Conduct Authority.
    >     >         >> >
    >     >         >>
    >     >         >>
    >     >         >>
    >     >         >> --
    >     >         >> -Regards,
    >     >         >> Mayuresh R. Gharat
    >     >         >> (862) 250-7125
    >     >         >> The information contained in this email is strictly
    >     > confidential and for
    >     >         >> the use of the addressee only, unless otherwise
    > indicated. If
    >     > you are not
    >     >         >> the intended recipient, please do not read, copy, use or
    >     > disclose to others
    >     >         >> this message or any attachment. Please also notify the
    > sender
    >     > by replying
    >     >         >> to this email or by telephone (+44(020 7896 0011) and then
    >     > delete the email
    >     >         >> and any copies of it. Opinions, conclusion (etc) that do
    > not
    >     > relate to the
    >     >         >> official business of this company shall be understood as
    >     > neither given nor
    >     >         >> endorsed by it. IG is a trading name of IG Markets
    > Limited (a
    >     > company
    >     >         >> registered in England and Wales, company number 04008957)
    > and
    >     > IG Index
    >     >         >> Limited (a company registered in England and Wales,
    > company
    >     > number
    >     >         >> 01190902). Registered address at Cannon Bridge House, 25
    >     > Dowgate Hill,
    >     >         >> London EC4R 2YA. Both IG Markets Limited (register number
    >     > 195355) and IG
    >     >         >> Index Limited (register number 114059) are authorised and
    >     > regulated by the
    >     >         >> Financial Conduct Authority.
    >     >         >>
    >     >         >
    >     >         >
    >     >         >
    >     >         > --
    >     >         > -Regards,
    >     >         > Mayuresh R. Gharat
    >     >         > (862) 250-7125
    >     >         >
    >     >
    >     >
    >     >
    >     >         --
    >     >         -Regards,
    >     >         Mayuresh R. Gharat
    >     >         (862) 250-7125
    >     >
    >     >
    >     >     The information contained in this email is strictly confidential
    > and
    >     > for the use of the addressee only, unless otherwise indicated. If
    > you are
    >     > not the intended recipient, please do not read, copy, use or
    > disclose to
    >     > others this message or any attachment. Please also notify the sender
    > by
    >     > replying to this email or by telephone (+44(020 7896 0011) and then
    > delete
    >     > the email and any copies of it. Opinions, conclusion (etc) that do
    > not
    >     > relate to the official business of this company shall be understood
    > as
    >     > neither given nor endorsed by it. IG is a trading name of IG Markets
    >     > Limited (a company registered in England and Wales, company number
    >     > 04008957) and IG Index Limited (a company registered in England and
    > Wales,
    >     > company number 01190902). Registered address at Cannon Bridge House,
    > 25
    >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
    > number
    >     > 195355) and IG Index Limited (register number 114059) are authorised
    > and
    >     > regulated by the Financial Conduct Authority.
    >     >
    >     >
    >     >
    >
    >
    > The information contained in this email is strictly confidential and for
    > the use of the addressee only, unless otherwise indicated. If you are not
    > the intended recipient, please do not read, copy, use or disclose to others
    > this message or any attachment. Please also notify the sender by replying
    > to this email or by telephone (+44(020 7896 0011) and then delete the email
    > and any copies of it. Opinions, conclusion (etc) that do not relate to the
    > official business of this company shall be understood as neither given nor
    > endorsed by it. IG is a trading name of IG Markets Limited (a company
    > registered in England and Wales, company number 04008957) and IG Index
    > Limited (a company registered in England and Wales, company number
    > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
    > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
    > Index Limited (register number 114059) are authorised and regulated by the
    > Financial Conduct Authority.
    >


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jun Rao <ju...@confluent.io>]

Hi, Michael,

For 2), this is fine. Could you update the KIP wiki to make this and other
points clearer? Other than that, the KIP looks good to me.

An orthogonal thing is that there are other KIPs such as KIP-98 that also
intend to change the message format. If they all get approved, we should
think about whether it's better to just bump up the magic byte once to
incorporate multiple format changes like we did in KIP-31/KIP-32.

Thanks,

Jun


On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi Jao
>
> Thanks for the response. Sorry for slow reply, both with personal sickness
> and also battling some critical issues encountered since upgrading to
> 0.10.1.0
>
> 1) Thans for spotting, Document error where we branched this KIP from
> KIP-82, will get that removed.
> 2) Intent is to do this just at the record message level.
> 3) Thanks for spotting, Will ensure this is corrected.
> 4) As per discussion thread we will support tombstone + null value,
> tombstone + non null value, no tombstone + null value.
> 5) I believe this was in the discussion thread, @Mayuresh is this
> something we’ve overlooked? I thought we would down convert and remove the
> value so the old consumer had existing behavior, or is there something we
> haven’t thought about?
>
> Cheers
> Mike
>
> On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
>
>     Hi, Michael,
>
>     Thanks for the KIP. A few comments below.
>
>     1. The message format change contains "HeadersLength Headers". Is that
>     intended?
>
>     2. For compressed messageset, is the tombstone bit only set at the
> shallow
>     level? Do we always leave that bit in the wrapper message unset? An
>     alternative is to set the tombstone bit in the wrapper if at least one
>     inner message has the tombstone bit set. This makes things a bit more
>     complicated, but we could potentially exploit that for optimizing down
>     conversion. For example, we only need to convert messages with magic 2
> to
>     magic 1 if the wrapper's tombstone bit is set (conversion is always
> needed
>     from magic 2 to magic 0). Not sure if the optimization is worth the
>     complexity though.
>
>     3. The referencing of the new version of ProducerRequest/FetchRequest
> is
>     inconsistent (v4 vs v3). Since our convention starts at version at 0, I
>     think the new version would be 3.
>
>     4. "If the magic byte on message is 2, the broker should use the
> tombstone
>     bit for log compaction." What about null value? My understanding is
> that
>     null value will be treated the same as setting the tombstone bit.
>
>     5. For the migration path, it would be useful to describe the down
>     conversion path to consumers (i.e., brokers on message format 0.10.2
> and
>     consumers on older version).
>
>     Thanks,
>
>     Jun
>
>
>     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <Michael.Pearce@ig.com
> >
>     wrote:
>
>     > Hi All,
>     >
>     > We have been discussing in the below thread and final changes have
> been
>     > made to the KIP wiki based on these discussions.
>     >
>     > We would now like to put to the vote the following KIP:
>     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
>     > Add+Compaction+Tombstone+Flag
>     >
>     > This kip is for having a distinct compaction attribute “tombstone”
> flag
>     > instead of relying on null value, allowing non-null value delete
> messages.
>     >
>     > Many thanks,
>     > Michael
>     >
>     >
>     >
>     > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com>
> wrote:
>     >
>     >     Hi Mayuresh,
>     >
>     >     LGTM. Ive just made one small adjustment updating the wire
> protocol to
>     > show the magic byte bump.
>     >
>     >     Do we think we’re good to put to a vote? Is there any other bits
>     > needing discussion?
>     >
>     >     Cheers
>     >     Mike
>     >
>     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> gharatmayuresh15@gmail.com>
>     > wrote:
>     >
>     >         Hi Michael,
>     >
>     >         I have updated the migration section of the KIP. Can you
> please
>     > take a look?
>     >
>     >         Thanks,
>     >
>     >         Mayuresh
>     >
>     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
>     > gharatmayuresh15@gmail.com
>     >         > wrote:
>     >
>     >         > Hi Michael,
>     >         >
>     >         > That whilst sending tombstone and non null value, the
> consumer
>     > can expect
>     >         > only to receive the non-null message only in step (3) is
> this
>     > correct?
>     >         > ---> I do agree with you here.
>     >         >
>     >         > Becket, Ismael : can you guys review the migration plan
> listed
>     > above using
>     >         > magic byte?
>     >         >
>     >         > Thanks,
>     >         >
>     >         > Mayuresh
>     >         >
>     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
>     > Michael.Pearce@ig.com>
>     >         > wrote:
>     >         >
>     >         >> Many thanks for this Mayuresh. I don't have any
> objections.
>     >         >>
>     >         >> I assume we should state:
>     >         >>
>     >         >> That whilst sending tombstone and non null value, the
> consumer
>     > can expect
>     >         >> only to receive the non-null message only in step (3) is
> this
>     > correct?
>     >         >>
>     >         >> Cheers
>     >         >> Mike
>     >         >>
>     >         >>
>     >         >>
>     >         >> Sent using OWA for iPhone
>     >         >> ________________________________________
>     >         >> From: Mayuresh Gharat <gh...@gmail.com>
>     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
>     >         >> To: dev@kafka.apache.org
>     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone
> Flag
>     >         >>
>     >         >> Hi Ismael,
>     >         >>
>     >         >> Thanks for the explanation.
>     >         >> Specially I like this part where in you mentioned we can
> get
>     > rid of the
>     >         >> older null value support for log compaction later on,
> here :
>     >         >> We can't change semantics of the message format without
> having
>     > a long
>     >         >> transition period. And we can't rely
>     >         >> on people reading documentation or acting on a warning for
>     > something so
>     >         >> fundamental. As such, my take is that we need to bump the
> magic
>     > byte. The
>     >         >> good news is
>     >         >> that we don't have to support all versions forever. We
> have
>     > said that we
>     >         >> will support direct upgrades for 2 years. That means that
>     > message format
>     >         >> version n could, in theory, be removed 2 years after the
> it's
>     > introduced.
>     >         >>
>     >         >> Just a heads up, I would like to mention that even without
>     > bumping magic
>     >         >> byte, we will *NOT* loose zero copy as in the client(x+1)
> in my
>     >         >> explanation
>     >         >> above will convert internally a null value to have a
> tombstone
>     > bit set and
>     >         >> a tombstone bit set to have a null value automatically
>     > internally and by
>     >         >> the time we move to version (x+2), the clients would have
>     > upgraded.
>     >         >> Obviously if we support a request from consumer(x), we
> will
>     > loose zero
>     >         >> copy
>     >         >> but that is the same case with magic byte.
>     >         >>
>     >         >> But if magic byte bump makes life easier for transition
> for the
>     > above
>     >         >> reasons that you explained, I am OK with it since we are
> going
>     > to meet the
>     >         >> end goal down the road :)
>     >         >>
>     >         >> On a side note can we update the doc here on magic byte
> to say
>     > that "*it
>     >         >> should be bumped whenever the message format is changed
> or the
>     >         >> interpretation of message format (usage of the reserved
> bits as
>     > well) is
>     >         >> changed*".
>     >         >>
>     >         >>
>     >         >> Hi Michael,
>     >         >>
>     >         >> Here is the update plan that we discussed offline
> yesterday :
>     >         >>
>     >         >> Currently the magic-byte which corresponds to the
>     > "message.format.version"
>     >         >> is set to 1.
>     >         >>
>     >         >> 1) On broker it will be set to 1 initially.
>     >         >>
>     >         >> 2) When a producer client sends a message with magic-byte
> = 2,
>     > since the
>     >         >> broker is on magic-byte = 1, we will down convert it,
> which
>     > means if the
>     >         >> tombstone bit is set, the value will be set to null. A
> consumer
>     >         >> understanding magic-byte = 1, will still work with this. A
>     > consumer
>     >         >> working
>     >         >> with magic-byte =2 will also be able to understand this,
> since
>     > it
>     >         >> understands the tombstone.
>     >         >> Now there is still the question of supporting a
> non-tombstone
>     > and null
>     >         >> value from producer client with magic-byte = 2.* (I am
> not sure
>     > if we
>     >         >> should support this. Ismael/Becket can comment here)*
>     >         >>
>     >         >> 3) When almost all the clients have upgraded, the
>     > message.format.version
>     >         >> on
>     >         >> the broker can be changed to 2, where in the down
> conversion in
>     > the above
>     >         >> step will not happen. If at this point we get a consumer
>     > request from a
>     >         >> older consumer, we might have to down convert where in we
> loose
>     > zero copy,
>     >         >> but these cases should be rare.
>     >         >>
>     >         >> Becket can you review this plan and add more details if I
> have
>     >         >> missed/wronged something, before we put it on KIP.
>     >         >>
>     >         >> Thanks,
>     >         >>
>     >         >> Mayuresh
>     >         >>
>     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
>     > Michael.Pearce@ig.com>
>     >         >> wrote:
>     >         >>
>     >         >> > Thanks guys, for discussing this offline and getting
> some
>     > consensus.
>     >         >> >
>     >         >> > So its clear for myself and others what is proposed now
> (i
>     > think i
>     >         >> > understand, but want to make sure)
>     >         >> >
>     >         >> > Could i ask either directly update the kip to detail the
>     > migration
>     >         >> > strategy, or (re-)state your offline discussed and
> agreed
>     > migration
>     >         >> > strategy based on a magic byte is in this thread.
>     >         >> >
>     >         >> >
>     >         >> > The main original driver for the KIP was to support
>     > compaction where
>     >         >> value
>     >         >> > isn't null, based off the discussions on KIP-82 thread.
>     >         >> >
>     >         >> > We should be able to support non-tombstone + null value
> by the
>     >         >> completion
>     >         >> > of the KIP, as we noted when discussing this kip, having
>     > logic based on
>     >         >> a
>     >         >> > null value isn't very clean and also separates the
> concerns.
>     >         >> >
>     >         >> > As discussed already though we can split this into
> KIP-87a
>     > and KIP-87b
>     >         >> >
>     >         >> > Where we look to deliver KIP-87a on a compacted topic
> (to
>     > address the
>     >         >> > immediate issues)
>     >         >> > * tombstone + null value
>     >         >> > * tombstone + non-null value
>     >         >> > * non-tombstone + non-null value
>     >         >> >
>     >         >> > Then we can discuss once KIP-87a is completed options
> later
>     > and how we
>     >         >> > support the second part KIP-87b to deliver:
>     >         >> > * non-tombstone + null value
>     >         >> >
>     >         >> > Cheers
>     >         >> > Mike
>     >         >> >
>     >         >> >
>     >         >> >
>     >         >> > ________________________________________
>     >         >> > From: Becket Qin <be...@gmail.com>
>     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
>     >         >> > To: dev@kafka.apache.org
>     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> Tombstone Flag
>     >         >> >
>     >         >> > Renu, Mayuresh and I had an offline discussion, and
> following
>     > is a brief
>     >         >> > summary.
>     >         >> >
>     >         >> > 1. We agreed that not bumping up magic value may result
> in
>     > losing zero
>     >         >> copy
>     >         >> > during migration.
>     >         >> > 2. Given that bumping up magic value is almost free and
> has
>     > benefit of
>     >         >> > avoiding potential performance issue. It is probably
> worth
>     > doing.
>     >         >> >
>     >         >> > One issue we still need to think about is whether we
> want to
>     > support a
>     >         >> > non-tombstone message with null value.
>     >         >> > Currently it is not supported by Kafka. If we allow a
>     > non-tombstone null
>     >         >> > value message to exist after KIP-87. The problem is
> that such
>     > message
>     >         >> will
>     >         >> > not be supported by the consumers prior to KIP-87.
> Because a
>     > null value
>     >         >> > will always be interpreted to a tombstone.
>     >         >> >
>     >         >> > One option is that we keep the current way, i.e. do not
>     > support such
>     >         >> > message. It would be good to know if there is a
> concrete use
>     > case for
>     >         >> such
>     >         >> > message. If there is not, we can probably just not
> support it.
>     >         >> >
>     >         >> > Thanks,
>     >         >> >
>     >         >> > JIangjie (Becket) Qin
>     >         >> >
>     >         >> >
>     >         >> >
>     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
>     >         >> > gharatmayuresh15@gmail.com
>     >         >> > > wrote:
>     >         >> >
>     >         >> > > Hi Ismael,
>     >         >> > >
>     >         >> > > This is something I can think of for migration plan:
>     >         >> > > So the migration plan can look something like this,
> with up
>     >         >> conversion :
>     >         >> > >
>     >         >> > > 1) Currently lets say we have Broker at version x.
>     >         >> > > 2) Currently we have clients at version x.
>     >         >> > > 3) a) We move the version to Broker(x+1) : supports
> both
>     > tombstone and
>     >         >> > null
>     >         >> > > for log compaction.
>     >         >> > >     b) We upgrade the client to version client(x+1) :
> if in
>     > the
>     >         >> producer
>     >         >> > > client(x+1) the value is set to null, we will
> automatically
>     > set the
>     >         >> > > Tombstone bit internally. If the producer client(x+1)
> sets
>     > the
>     >         >> tombstone
>     >         >> > > itself, well and good. For producer client(x), the
> broker
>     > will up
>     >         >> convert
>     >         >> > > to have the tombstone bit. Broker(x+1) is supporting
> both.
>     > Consumer
>     >         >> > > client(x+1) will be aware of this and should be able
> to
>     > handle this.
>     >         >> For
>     >         >> > > consumer client(x) we will down convert the message
> on the
>     > broker
>     >         >> side.
>     >         >> > >     c) At this point we will have to specify a
> warning or
>     > clearly
>     >         >> specify
>     >         >> > > in docs that this behavior is about to be changed for
> log
>     > compaction.
>     >         >> > > 4) a) In next release of the Broker(x+2), we say that
> only
>     > Tombstone
>     >         >> is
>     >         >> > > used for log compaction on the Broker side.
> Clients(x+1)
>     > still is
>     >         >> > > supported.
>     >         >> > >     b) We upgrade the client to version client(x+2) :
> if
>     > value is set
>     >         >> to
>     >         >> > > null, tombstone will not be set automatically. The
> client
>     > will have to
>     >         >> > call
>     >         >> > > setTombstone() to actually set the tombstone.
>     >         >> > >
>     >         >> > > We should compare this migration plan with the
> migration
>     > plan for
>     >         >> magic
>     >         >> > > byte bump and do whatever looks good.
>     >         >> > > I am just worried that if we go down magic byte route,
>     > unless I am
>     >         >> > missing
>     >         >> > > something, it sounds like kafka will be stuck with
>     > supporting both
>     >         >> null
>     >         >> > > value and tombstone bit for log compaction for life
> long,
>     > which does
>     >         >> not
>     >         >> > > look like a good end state.
>     >         >> > >
>     >         >> > > Thanks,
>     >         >> > >
>     >         >> > > Mayuresh
>     >         >> > >
>     >         >> > >
>     >         >> > >
>     >         >> > >
>     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
>     >         >> > > gharatmayuresh15@gmail.com
>     >         >> > > > wrote:
>     >         >> > >
>     >         >> > > > Hi Ismael,
>     >         >> > > >
>     >         >> > > > That's a very good point which I might have not
>     > considered earlier.
>     >         >> > > >
>     >         >> > > > Here is a plan that I can think of:
>     >         >> > > >
>     >         >> > > > Stage 1) The broker from now on, up converts the
> message
>     > to have the
>     >         >> > > > tombstone marker. The log compaction thread does log
>     > compaction
>     >         >> based
>     >         >> > on
>     >         >> > > > both null and tombstone marker. This is our
> transition
>     > period.
>     >         >> > > > Stage 2) The next release we only say that log
> compaction
>     > is based
>     >         >> on
>     >         >> > > > tombstone marker. (Open source kafka makes this as a
>     > policy). By
>     >         >> this
>     >         >> > > time,
>     >         >> > > > the organization which is moving to this release
> will be
>     > sure that
>     >         >> they
>     >         >> > > > have gone through the entire transition period.
>     >         >> > > >
>     >         >> > > > My only goal of doing this is that Kafka clearly
>     > specifies the end
>     >         >> > state
>     >         >> > > > about what log compaction means (is it null value
> or a
>     > tombstone
>     >         >> > marker,
>     >         >> > > > but not both).
>     >         >> > > >
>     >         >> > > > What do you think?
>     >         >> > > >
>     >         >> > > > Thanks,
>     >         >> > > >
>     >         >> > > > Mayuresh
>     >         >> > > > .
>     >         >> > > >
>     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
>     > ismael@juma.me.uk>
>     >         >> > wrote:
>     >         >> > > >
>     >         >> > > >> One comment below.
>     >         >> > > >>
>     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
>     >         >> > > >> gharatmayuresh15@gmail.com
>     >         >> > > >> > wrote:
>     >         >> > > >>
>     >         >> > > >> >    - If we don't bump up the magic byte, on the
> broker
>     > side, the
>     >         >> > > broker
>     >         >> > > >> >    will always have to look at both tombstone
> bit and
>     > the value
>     >         >> when
>     >         >> > > do
>     >         >> > > >> the
>     >         >> > > >> >    compaction. Assuming we do not bump up the
> magic
>     > byte,
>     >         >> > > >> >    imagine the broker sees a message which does
> not
>     > have a
>     >         >> tombstone
>     >         >> > > bit
>     >         >> > > >> >    set. The broker does not know when the
> message was
>     > produced
>     >         >> (i.e.
>     >         >> > > >> > whether
>     >         >> > > >> >    the message has been up converted or not), it
> has
>     > to take a
>     >         >> > further
>     >         >> > > >> > look at
>     >         >> > > >> >    the value to see if it is null or not in
> order to
>     > determine
>     >         >> if it
>     >         >> > > is
>     >         >> > > >> a
>     >         >> > > >> >    tombstone. The same logic has to be put on the
>     > consumer as
>     >         >> well
>     >         >> > > >> because
>     >         >> > > >> > the
>     >         >> > > >> >    consumer does not know if the message has
> been up
>     > converted or
>     >         >> > not.
>     >         >> > > >> >       - If we upconvert while appending, this is
> not
>     > the case,
>     >         >> > right?
>     >         >> > > >>
>     >         >> > > >>
>     >         >> > > >> If I understand you correctly, this is not
> sufficient
>     > because the
>     >         >> log
>     >         >> > > may
>     >         >> > > >> have messages appended before it was upgraded to
> include
>     > KIP-87.
>     >         >> > > >>
>     >         >> > > >> Ismael
>     >         >> > > >>
>     >         >> > > >
>     >         >> > > >
>     >         >> > > >
>     >         >> > > > --
>     >         >> > > > -Regards,
>     >         >> > > > Mayuresh R. Gharat
>     >         >> > > > (862) 250-7125
>     >         >> > > >
>     >         >> > >
>     >         >> > >
>     >         >> > >
>     >         >> > > --
>     >         >> > > -Regards,
>     >         >> > > Mayuresh R. Gharat
>     >         >> > > (862) 250-7125
>     >         >> > >
>     >         >> > The information contained in this email is strictly
>     > confidential and for
>     >         >> > the use of the addressee only, unless otherwise
> indicated. If
>     > you are
>     >         >> not
>     >         >> > the intended recipient, please do not read, copy, use or
>     > disclose to
>     >         >> others
>     >         >> > this message or any attachment. Please also notify the
> sender
>     > by
>     >         >> replying
>     >         >> > to this email or by telephone (+44(020 7896 0011) and
> then
>     > delete the
>     >         >> email
>     >         >> > and any copies of it. Opinions, conclusion (etc) that
> do not
>     > relate to
>     >         >> the
>     >         >> > official business of this company shall be understood as
>     > neither given
>     >         >> nor
>     >         >> > endorsed by it. IG is a trading name of IG Markets
> Limited (a
>     > company
>     >         >> > registered in England and Wales, company number
> 04008957) and
>     > IG Index
>     >         >> > Limited (a company registered in England and Wales,
> company
>     > number
>     >         >> > 01190902). Registered address at Cannon Bridge House, 25
>     > Dowgate Hill,
>     >         >> > London EC4R 2YA. Both IG Markets Limited (register
> number
>     > 195355) and IG
>     >         >> > Index Limited (register number 114059) are authorised
> and
>     > regulated by
>     >         >> the
>     >         >> > Financial Conduct Authority.
>     >         >> >
>     >         >>
>     >         >>
>     >         >>
>     >         >> --
>     >         >> -Regards,
>     >         >> Mayuresh R. Gharat
>     >         >> (862) 250-7125
>     >         >> The information contained in this email is strictly
>     > confidential and for
>     >         >> the use of the addressee only, unless otherwise
> indicated. If
>     > you are not
>     >         >> the intended recipient, please do not read, copy, use or
>     > disclose to others
>     >         >> this message or any attachment. Please also notify the
> sender
>     > by replying
>     >         >> to this email or by telephone (+44(020 7896 0011) and then
>     > delete the email
>     >         >> and any copies of it. Opinions, conclusion (etc) that do
> not
>     > relate to the
>     >         >> official business of this company shall be understood as
>     > neither given nor
>     >         >> endorsed by it. IG is a trading name of IG Markets
> Limited (a
>     > company
>     >         >> registered in England and Wales, company number 04008957)
> and
>     > IG Index
>     >         >> Limited (a company registered in England and Wales,
> company
>     > number
>     >         >> 01190902). Registered address at Cannon Bridge House, 25
>     > Dowgate Hill,
>     >         >> London EC4R 2YA. Both IG Markets Limited (register number
>     > 195355) and IG
>     >         >> Index Limited (register number 114059) are authorised and
>     > regulated by the
>     >         >> Financial Conduct Authority.
>     >         >>
>     >         >
>     >         >
>     >         >
>     >         > --
>     >         > -Regards,
>     >         > Mayuresh R. Gharat
>     >         > (862) 250-7125
>     >         >
>     >
>     >
>     >
>     >         --
>     >         -Regards,
>     >         Mayuresh R. Gharat
>     >         (862) 250-7125
>     >
>     >
>     >     The information contained in this email is strictly confidential
> and
>     > for the use of the addressee only, unless otherwise indicated. If
> you are
>     > not the intended recipient, please do not read, copy, use or
> disclose to
>     > others this message or any attachment. Please also notify the sender
> by
>     > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
>     > the email and any copies of it. Opinions, conclusion (etc) that do
> not
>     > relate to the official business of this company shall be understood
> as
>     > neither given nor endorsed by it. IG is a trading name of IG Markets
>     > Limited (a company registered in England and Wales, company number
>     > 04008957) and IG Index Limited (a company registered in England and
> Wales,
>     > company number 01190902). Registered address at Cannon Bridge House,
> 25
>     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> number
>     > 195355) and IG Index Limited (register number 114059) are authorised
> and
>     > regulated by the Financial Conduct Authority.
>     >
>     >
>     >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Mayuresh Gharat <gh...@gmail.com>]

Hi Michael,

5) I believe this was in the discussion thread, @Mayuresh is this something
we’ve overlooked? I thought we would down convert and remove the value so
the old consumer had existing behavior, or is there something we haven’t
thought about?
---> Yea, we would have to down convert and remove the value, so that old
consumer retains existing behavior.

Thanks,

Mayuresh

On Fri, Dec 2, 2016 at 5:55 AM, Michael Pearce <Mi...@ig.com>
wrote:

> *Hi Jun,
>
> Soo sorry for the typo/mistake.
>
> On 02/12/2016, 11:19, "Michael Pearce" <Mi...@ig.com> wrote:
>
>     Hi Jao
>
>     Thanks for the response. Sorry for slow reply, both with personal
> sickness and also battling some critical issues encountered since upgrading
> to 0.10.1.0
>
>     1) Thans for spotting, Document error where we branched this KIP from
> KIP-82, will get that removed.
>     2) Intent is to do this just at the record message level.
>     3) Thanks for spotting, Will ensure this is corrected.
>     4) As per discussion thread we will support tombstone + null value,
> tombstone + non null value, no tombstone + null value.
>     5) I believe this was in the discussion thread, @Mayuresh is this
> something we’ve overlooked? I thought we would down convert and remove the
> value so the old consumer had existing behavior, or is there something we
> haven’t thought about?
>
>     Cheers
>     Mike
>
>     On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
>
>         Hi, Michael,
>
>         Thanks for the KIP. A few comments below.
>
>         1. The message format change contains "HeadersLength Headers". Is
> that
>         intended?
>
>         2. For compressed messageset, is the tombstone bit only set at the
> shallow
>         level? Do we always leave that bit in the wrapper message unset? An
>         alternative is to set the tombstone bit in the wrapper if at least
> one
>         inner message has the tombstone bit set. This makes things a bit
> more
>         complicated, but we could potentially exploit that for optimizing
> down
>         conversion. For example, we only need to convert messages with
> magic 2 to
>         magic 1 if the wrapper's tombstone bit is set (conversion is
> always needed
>         from magic 2 to magic 0). Not sure if the optimization is worth the
>         complexity though.
>
>         3. The referencing of the new version of
> ProducerRequest/FetchRequest is
>         inconsistent (v4 vs v3). Since our convention starts at version at
> 0, I
>         think the new version would be 3.
>
>         4. "If the magic byte on message is 2, the broker should use the
> tombstone
>         bit for log compaction." What about null value? My understanding
> is that
>         null value will be treated the same as setting the tombstone bit.
>
>         5. For the migration path, it would be useful to describe the down
>         conversion path to consumers (i.e., brokers on message format
> 0.10.2 and
>         consumers on older version).
>
>         Thanks,
>
>         Jun
>
>
>         On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>         wrote:
>
>         > Hi All,
>         >
>         > We have been discussing in the below thread and final changes
> have been
>         > made to the KIP wiki based on these discussions.
>         >
>         > We would now like to put to the vote the following KIP:
>         > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
>         > Add+Compaction+Tombstone+Flag
>         >
>         > This kip is for having a distinct compaction attribute
> “tombstone” flag
>         > instead of relying on null value, allowing non-null value delete
> messages.
>         >
>         > Many thanks,
>         > Michael
>         >
>         >
>         >
>         > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com>
> wrote:
>         >
>         >     Hi Mayuresh,
>         >
>         >     LGTM. Ive just made one small adjustment updating the wire
> protocol to
>         > show the magic byte bump.
>         >
>         >     Do we think we’re good to put to a vote? Is there any other
> bits
>         > needing discussion?
>         >
>         >     Cheers
>         >     Mike
>         >
>         >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> gharatmayuresh15@gmail.com>
>         > wrote:
>         >
>         >         Hi Michael,
>         >
>         >         I have updated the migration section of the KIP. Can you
> please
>         > take a look?
>         >
>         >         Thanks,
>         >
>         >         Mayuresh
>         >
>         >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
>         > gharatmayuresh15@gmail.com
>         >         > wrote:
>         >
>         >         > Hi Michael,
>         >         >
>         >         > That whilst sending tombstone and non null value, the
> consumer
>         > can expect
>         >         > only to receive the non-null message only in step (3)
> is this
>         > correct?
>         >         > ---> I do agree with you here.
>         >         >
>         >         > Becket, Ismael : can you guys review the migration
> plan listed
>         > above using
>         >         > magic byte?
>         >         >
>         >         > Thanks,
>         >         >
>         >         > Mayuresh
>         >         >
>         >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
>         > Michael.Pearce@ig.com>
>         >         > wrote:
>         >         >
>         >         >> Many thanks for this Mayuresh. I don't have any
> objections.
>         >         >>
>         >         >> I assume we should state:
>         >         >>
>         >         >> That whilst sending tombstone and non null value, the
> consumer
>         > can expect
>         >         >> only to receive the non-null message only in step (3)
> is this
>         > correct?
>         >         >>
>         >         >> Cheers
>         >         >> Mike
>         >         >>
>         >         >>
>         >         >>
>         >         >> Sent using OWA for iPhone
>         >         >> ________________________________________
>         >         >> From: Mayuresh Gharat <gh...@gmail.com>
>         >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
>         >         >> To: dev@kafka.apache.org
>         >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> Tombstone Flag
>         >         >>
>         >         >> Hi Ismael,
>         >         >>
>         >         >> Thanks for the explanation.
>         >         >> Specially I like this part where in you mentioned we
> can get
>         > rid of the
>         >         >> older null value support for log compaction later on,
> here :
>         >         >> We can't change semantics of the message format
> without having
>         > a long
>         >         >> transition period. And we can't rely
>         >         >> on people reading documentation or acting on a
> warning for
>         > something so
>         >         >> fundamental. As such, my take is that we need to bump
> the magic
>         > byte. The
>         >         >> good news is
>         >         >> that we don't have to support all versions forever.
> We have
>         > said that we
>         >         >> will support direct upgrades for 2 years. That means
> that
>         > message format
>         >         >> version n could, in theory, be removed 2 years after
> the it's
>         > introduced.
>         >         >>
>         >         >> Just a heads up, I would like to mention that even
> without
>         > bumping magic
>         >         >> byte, we will *NOT* loose zero copy as in the
> client(x+1) in my
>         >         >> explanation
>         >         >> above will convert internally a null value to have a
> tombstone
>         > bit set and
>         >         >> a tombstone bit set to have a null value automatically
>         > internally and by
>         >         >> the time we move to version (x+2), the clients would
> have
>         > upgraded.
>         >         >> Obviously if we support a request from consumer(x),
> we will
>         > loose zero
>         >         >> copy
>         >         >> but that is the same case with magic byte.
>         >         >>
>         >         >> But if magic byte bump makes life easier for
> transition for the
>         > above
>         >         >> reasons that you explained, I am OK with it since we
> are going
>         > to meet the
>         >         >> end goal down the road :)
>         >         >>
>         >         >> On a side note can we update the doc here on magic
> byte to say
>         > that "*it
>         >         >> should be bumped whenever the message format is
> changed or the
>         >         >> interpretation of message format (usage of the
> reserved bits as
>         > well) is
>         >         >> changed*".
>         >         >>
>         >         >>
>         >         >> Hi Michael,
>         >         >>
>         >         >> Here is the update plan that we discussed offline
> yesterday :
>         >         >>
>         >         >> Currently the magic-byte which corresponds to the
>         > "message.format.version"
>         >         >> is set to 1.
>         >         >>
>         >         >> 1) On broker it will be set to 1 initially.
>         >         >>
>         >         >> 2) When a producer client sends a message with
> magic-byte = 2,
>         > since the
>         >         >> broker is on magic-byte = 1, we will down convert it,
> which
>         > means if the
>         >         >> tombstone bit is set, the value will be set to null.
> A consumer
>         >         >> understanding magic-byte = 1, will still work with
> this. A
>         > consumer
>         >         >> working
>         >         >> with magic-byte =2 will also be able to understand
> this, since
>         > it
>         >         >> understands the tombstone.
>         >         >> Now there is still the question of supporting a
> non-tombstone
>         > and null
>         >         >> value from producer client with magic-byte = 2.* (I
> am not sure
>         > if we
>         >         >> should support this. Ismael/Becket can comment here)*
>         >         >>
>         >         >> 3) When almost all the clients have upgraded, the
>         > message.format.version
>         >         >> on
>         >         >> the broker can be changed to 2, where in the down
> conversion in
>         > the above
>         >         >> step will not happen. If at this point we get a
> consumer
>         > request from a
>         >         >> older consumer, we might have to down convert where
> in we loose
>         > zero copy,
>         >         >> but these cases should be rare.
>         >         >>
>         >         >> Becket can you review this plan and add more details
> if I have
>         >         >> missed/wronged something, before we put it on KIP.
>         >         >>
>         >         >> Thanks,
>         >         >>
>         >         >> Mayuresh
>         >         >>
>         >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
>         > Michael.Pearce@ig.com>
>         >         >> wrote:
>         >         >>
>         >         >> > Thanks guys, for discussing this offline and
> getting some
>         > consensus.
>         >         >> >
>         >         >> > So its clear for myself and others what is proposed
> now (i
>         > think i
>         >         >> > understand, but want to make sure)
>         >         >> >
>         >         >> > Could i ask either directly update the kip to
> detail the
>         > migration
>         >         >> > strategy, or (re-)state your offline discussed and
> agreed
>         > migration
>         >         >> > strategy based on a magic byte is in this thread.
>         >         >> >
>         >         >> >
>         >         >> > The main original driver for the KIP was to support
>         > compaction where
>         >         >> value
>         >         >> > isn't null, based off the discussions on KIP-82
> thread.
>         >         >> >
>         >         >> > We should be able to support non-tombstone + null
> value by the
>         >         >> completion
>         >         >> > of the KIP, as we noted when discussing this kip,
> having
>         > logic based on
>         >         >> a
>         >         >> > null value isn't very clean and also separates the
> concerns.
>         >         >> >
>         >         >> > As discussed already though we can split this into
> KIP-87a
>         > and KIP-87b
>         >         >> >
>         >         >> > Where we look to deliver KIP-87a on a compacted
> topic (to
>         > address the
>         >         >> > immediate issues)
>         >         >> > * tombstone + null value
>         >         >> > * tombstone + non-null value
>         >         >> > * non-tombstone + non-null value
>         >         >> >
>         >         >> > Then we can discuss once KIP-87a is completed
> options later
>         > and how we
>         >         >> > support the second part KIP-87b to deliver:
>         >         >> > * non-tombstone + null value
>         >         >> >
>         >         >> > Cheers
>         >         >> > Mike
>         >         >> >
>         >         >> >
>         >         >> >
>         >         >> > ________________________________________
>         >         >> > From: Becket Qin <be...@gmail.com>
>         >         >> > Sent: Thursday, November 17, 2016 1:43 AM
>         >         >> > To: dev@kafka.apache.org
>         >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> Tombstone Flag
>         >         >> >
>         >         >> > Renu, Mayuresh and I had an offline discussion, and
> following
>         > is a brief
>         >         >> > summary.
>         >         >> >
>         >         >> > 1. We agreed that not bumping up magic value may
> result in
>         > losing zero
>         >         >> copy
>         >         >> > during migration.
>         >         >> > 2. Given that bumping up magic value is almost free
> and has
>         > benefit of
>         >         >> > avoiding potential performance issue. It is
> probably worth
>         > doing.
>         >         >> >
>         >         >> > One issue we still need to think about is whether
> we want to
>         > support a
>         >         >> > non-tombstone message with null value.
>         >         >> > Currently it is not supported by Kafka. If we allow
> a
>         > non-tombstone null
>         >         >> > value message to exist after KIP-87. The problem is
> that such
>         > message
>         >         >> will
>         >         >> > not be supported by the consumers prior to KIP-87.
> Because a
>         > null value
>         >         >> > will always be interpreted to a tombstone.
>         >         >> >
>         >         >> > One option is that we keep the current way, i.e. do
> not
>         > support such
>         >         >> > message. It would be good to know if there is a
> concrete use
>         > case for
>         >         >> such
>         >         >> > message. If there is not, we can probably just not
> support it.
>         >         >> >
>         >         >> > Thanks,
>         >         >> >
>         >         >> > JIangjie (Becket) Qin
>         >         >> >
>         >         >> >
>         >         >> >
>         >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
>         >         >> > gharatmayuresh15@gmail.com
>         >         >> > > wrote:
>         >         >> >
>         >         >> > > Hi Ismael,
>         >         >> > >
>         >         >> > > This is something I can think of for migration
> plan:
>         >         >> > > So the migration plan can look something like
> this, with up
>         >         >> conversion :
>         >         >> > >
>         >         >> > > 1) Currently lets say we have Broker at version x.
>         >         >> > > 2) Currently we have clients at version x.
>         >         >> > > 3) a) We move the version to Broker(x+1) :
> supports both
>         > tombstone and
>         >         >> > null
>         >         >> > > for log compaction.
>         >         >> > >     b) We upgrade the client to version
> client(x+1) : if in
>         > the
>         >         >> producer
>         >         >> > > client(x+1) the value is set to null, we will
> automatically
>         > set the
>         >         >> > > Tombstone bit internally. If the producer
> client(x+1) sets
>         > the
>         >         >> tombstone
>         >         >> > > itself, well and good. For producer client(x),
> the broker
>         > will up
>         >         >> convert
>         >         >> > > to have the tombstone bit. Broker(x+1) is
> supporting both.
>         > Consumer
>         >         >> > > client(x+1) will be aware of this and should be
> able to
>         > handle this.
>         >         >> For
>         >         >> > > consumer client(x) we will down convert the
> message on the
>         > broker
>         >         >> side.
>         >         >> > >     c) At this point we will have to specify a
> warning or
>         > clearly
>         >         >> specify
>         >         >> > > in docs that this behavior is about to be changed
> for log
>         > compaction.
>         >         >> > > 4) a) In next release of the Broker(x+2), we say
> that only
>         > Tombstone
>         >         >> is
>         >         >> > > used for log compaction on the Broker side.
> Clients(x+1)
>         > still is
>         >         >> > > supported.
>         >         >> > >     b) We upgrade the client to version
> client(x+2) : if
>         > value is set
>         >         >> to
>         >         >> > > null, tombstone will not be set automatically.
> The client
>         > will have to
>         >         >> > call
>         >         >> > > setTombstone() to actually set the tombstone.
>         >         >> > >
>         >         >> > > We should compare this migration plan with the
> migration
>         > plan for
>         >         >> magic
>         >         >> > > byte bump and do whatever looks good.
>         >         >> > > I am just worried that if we go down magic byte
> route,
>         > unless I am
>         >         >> > missing
>         >         >> > > something, it sounds like kafka will be stuck with
>         > supporting both
>         >         >> null
>         >         >> > > value and tombstone bit for log compaction for
> life long,
>         > which does
>         >         >> not
>         >         >> > > look like a good end state.
>         >         >> > >
>         >         >> > > Thanks,
>         >         >> > >
>         >         >> > > Mayuresh
>         >         >> > >
>         >         >> > >
>         >         >> > >
>         >         >> > >
>         >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
>         >         >> > > gharatmayuresh15@gmail.com
>         >         >> > > > wrote:
>         >         >> > >
>         >         >> > > > Hi Ismael,
>         >         >> > > >
>         >         >> > > > That's a very good point which I might have not
>         > considered earlier.
>         >         >> > > >
>         >         >> > > > Here is a plan that I can think of:
>         >         >> > > >
>         >         >> > > > Stage 1) The broker from now on, up converts
> the message
>         > to have the
>         >         >> > > > tombstone marker. The log compaction thread
> does log
>         > compaction
>         >         >> based
>         >         >> > on
>         >         >> > > > both null and tombstone marker. This is our
> transition
>         > period.
>         >         >> > > > Stage 2) The next release we only say that log
> compaction
>         > is based
>         >         >> on
>         >         >> > > > tombstone marker. (Open source kafka makes this
> as a
>         > policy). By
>         >         >> this
>         >         >> > > time,
>         >         >> > > > the organization which is moving to this
> release will be
>         > sure that
>         >         >> they
>         >         >> > > > have gone through the entire transition period.
>         >         >> > > >
>         >         >> > > > My only goal of doing this is that Kafka clearly
>         > specifies the end
>         >         >> > state
>         >         >> > > > about what log compaction means (is it null
> value or a
>         > tombstone
>         >         >> > marker,
>         >         >> > > > but not both).
>         >         >> > > >
>         >         >> > > > What do you think?
>         >         >> > > >
>         >         >> > > > Thanks,
>         >         >> > > >
>         >         >> > > > Mayuresh
>         >         >> > > > .
>         >         >> > > >
>         >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
>         > ismael@juma.me.uk>
>         >         >> > wrote:
>         >         >> > > >
>         >         >> > > >> One comment below.
>         >         >> > > >>
>         >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
> Gharat <
>         >         >> > > >> gharatmayuresh15@gmail.com
>         >         >> > > >> > wrote:
>         >         >> > > >>
>         >         >> > > >> >    - If we don't bump up the magic byte, on
> the broker
>         > side, the
>         >         >> > > broker
>         >         >> > > >> >    will always have to look at both
> tombstone bit and
>         > the value
>         >         >> when
>         >         >> > > do
>         >         >> > > >> the
>         >         >> > > >> >    compaction. Assuming we do not bump up
> the magic
>         > byte,
>         >         >> > > >> >    imagine the broker sees a message which
> does not
>         > have a
>         >         >> tombstone
>         >         >> > > bit
>         >         >> > > >> >    set. The broker does not know when the
> message was
>         > produced
>         >         >> (i.e.
>         >         >> > > >> > whether
>         >         >> > > >> >    the message has been up converted or
> not), it has
>         > to take a
>         >         >> > further
>         >         >> > > >> > look at
>         >         >> > > >> >    the value to see if it is null or not in
> order to
>         > determine
>         >         >> if it
>         >         >> > > is
>         >         >> > > >> a
>         >         >> > > >> >    tombstone. The same logic has to be put
> on the
>         > consumer as
>         >         >> well
>         >         >> > > >> because
>         >         >> > > >> > the
>         >         >> > > >> >    consumer does not know if the message has
> been up
>         > converted or
>         >         >> > not.
>         >         >> > > >> >       - If we upconvert while appending,
> this is not
>         > the case,
>         >         >> > right?
>         >         >> > > >>
>         >         >> > > >>
>         >         >> > > >> If I understand you correctly, this is not
> sufficient
>         > because the
>         >         >> log
>         >         >> > > may
>         >         >> > > >> have messages appended before it was upgraded
> to include
>         > KIP-87.
>         >         >> > > >>
>         >         >> > > >> Ismael
>         >         >> > > >>
>         >         >> > > >
>         >         >> > > >
>         >         >> > > >
>         >         >> > > > --
>         >         >> > > > -Regards,
>         >         >> > > > Mayuresh R. Gharat
>         >         >> > > > (862) 250-7125
>         >         >> > > >
>         >         >> > >
>         >         >> > >
>         >         >> > >
>         >         >> > > --
>         >         >> > > -Regards,
>         >         >> > > Mayuresh R. Gharat
>         >         >> > > (862) 250-7125
>         >         >> > >
>         >         >> > The information contained in this email is strictly
>         > confidential and for
>         >         >> > the use of the addressee only, unless otherwise
> indicated. If
>         > you are
>         >         >> not
>         >         >> > the intended recipient, please do not read, copy,
> use or
>         > disclose to
>         >         >> others
>         >         >> > this message or any attachment. Please also notify
> the sender
>         > by
>         >         >> replying
>         >         >> > to this email or by telephone (+44(020 7896 0011)
> and then
>         > delete the
>         >         >> email
>         >         >> > and any copies of it. Opinions, conclusion (etc)
> that do not
>         > relate to
>         >         >> the
>         >         >> > official business of this company shall be
> understood as
>         > neither given
>         >         >> nor
>         >         >> > endorsed by it. IG is a trading name of IG Markets
> Limited (a
>         > company
>         >         >> > registered in England and Wales, company number
> 04008957) and
>         > IG Index
>         >         >> > Limited (a company registered in England and Wales,
> company
>         > number
>         >         >> > 01190902). Registered address at Cannon Bridge
> House, 25
>         > Dowgate Hill,
>         >         >> > London EC4R 2YA. Both IG Markets Limited (register
> number
>         > 195355) and IG
>         >         >> > Index Limited (register number 114059) are
> authorised and
>         > regulated by
>         >         >> the
>         >         >> > Financial Conduct Authority.
>         >         >> >
>         >         >>
>         >         >>
>         >         >>
>         >         >> --
>         >         >> -Regards,
>         >         >> Mayuresh R. Gharat
>         >         >> (862) 250-7125
>         >         >> The information contained in this email is strictly
>         > confidential and for
>         >         >> the use of the addressee only, unless otherwise
> indicated. If
>         > you are not
>         >         >> the intended recipient, please do not read, copy, use
> or
>         > disclose to others
>         >         >> this message or any attachment. Please also notify
> the sender
>         > by replying
>         >         >> to this email or by telephone (+44(020 7896 0011) and
> then
>         > delete the email
>         >         >> and any copies of it. Opinions, conclusion (etc) that
> do not
>         > relate to the
>         >         >> official business of this company shall be understood
> as
>         > neither given nor
>         >         >> endorsed by it. IG is a trading name of IG Markets
> Limited (a
>         > company
>         >         >> registered in England and Wales, company number
> 04008957) and
>         > IG Index
>         >         >> Limited (a company registered in England and Wales,
> company
>         > number
>         >         >> 01190902). Registered address at Cannon Bridge House,
> 25
>         > Dowgate Hill,
>         >         >> London EC4R 2YA. Both IG Markets Limited (register
> number
>         > 195355) and IG
>         >         >> Index Limited (register number 114059) are authorised
> and
>         > regulated by the
>         >         >> Financial Conduct Authority.
>         >         >>
>         >         >
>         >         >
>         >         >
>         >         > --
>         >         > -Regards,
>         >         > Mayuresh R. Gharat
>         >         > (862) 250-7125
>         >         >
>         >
>         >
>         >
>         >         --
>         >         -Regards,
>         >         Mayuresh R. Gharat
>         >         (862) 250-7125
>         >
>         >
>         >     The information contained in this email is strictly
> confidential and
>         > for the use of the addressee only, unless otherwise indicated.
> If you are
>         > not the intended recipient, please do not read, copy, use or
> disclose to
>         > others this message or any attachment. Please also notify the
> sender by
>         > replying to this email or by telephone (+44(020 7896 0011) and
> then delete
>         > the email and any copies of it. Opinions, conclusion (etc) that
> do not
>         > relate to the official business of this company shall be
> understood as
>         > neither given nor endorsed by it. IG is a trading name of IG
> Markets
>         > Limited (a company registered in England and Wales, company
> number
>         > 04008957) and IG Index Limited (a company registered in England
> and Wales,
>         > company number 01190902). Registered address at Cannon Bridge
> House, 25
>         > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> number
>         > 195355) and IG Index Limited (register number 114059) are
> authorised and
>         > regulated by the Financial Conduct Authority.
>         >
>         >
>         >
>
>
>     The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>


-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

*Hi Jun, 

Soo sorry for the typo/mistake.

On 02/12/2016, 11:19, "Michael Pearce" <Mi...@ig.com> wrote:

    Hi Jao
    
    Thanks for the response. Sorry for slow reply, both with personal sickness and also battling some critical issues encountered since upgrading to 0.10.1.0
    
    1) Thans for spotting, Document error where we branched this KIP from KIP-82, will get that removed.
    2) Intent is to do this just at the record message level.
    3) Thanks for spotting, Will ensure this is corrected.
    4) As per discussion thread we will support tombstone + null value, tombstone + non null value, no tombstone + null value.
    5) I believe this was in the discussion thread, @Mayuresh is this something we’ve overlooked? I thought we would down convert and remove the value so the old consumer had existing behavior, or is there something we haven’t thought about?
    
    Cheers
    Mike
    
    On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:
    
        Hi, Michael,
    
        Thanks for the KIP. A few comments below.
    
        1. The message format change contains "HeadersLength Headers". Is that
        intended?
    
        2. For compressed messageset, is the tombstone bit only set at the shallow
        level? Do we always leave that bit in the wrapper message unset? An
        alternative is to set the tombstone bit in the wrapper if at least one
        inner message has the tombstone bit set. This makes things a bit more
        complicated, but we could potentially exploit that for optimizing down
        conversion. For example, we only need to convert messages with magic 2 to
        magic 1 if the wrapper's tombstone bit is set (conversion is always needed
        from magic 2 to magic 0). Not sure if the optimization is worth the
        complexity though.
    
        3. The referencing of the new version of ProducerRequest/FetchRequest is
        inconsistent (v4 vs v3). Since our convention starts at version at 0, I
        think the new version would be 3.
    
        4. "If the magic byte on message is 2, the broker should use the tombstone
        bit for log compaction." What about null value? My understanding is that
        null value will be treated the same as setting the tombstone bit.
    
        5. For the migration path, it would be useful to describe the down
        conversion path to consumers (i.e., brokers on message format 0.10.2 and
        consumers on older version).
    
        Thanks,
    
        Jun
    
    
        On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <Mi...@ig.com>
        wrote:
    
        > Hi All,
        >
        > We have been discussing in the below thread and final changes have been
        > made to the KIP wiki based on these discussions.
        >
        > We would now like to put to the vote the following KIP:
        > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
        > Add+Compaction+Tombstone+Flag
        >
        > This kip is for having a distinct compaction attribute “tombstone” flag
        > instead of relying on null value, allowing non-null value delete messages.
        >
        > Many thanks,
        > Michael
        >
        >
        >
        > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com> wrote:
        >
        >     Hi Mayuresh,
        >
        >     LGTM. Ive just made one small adjustment updating the wire protocol to
        > show the magic byte bump.
        >
        >     Do we think we’re good to put to a vote? Is there any other bits
        > needing discussion?
        >
        >     Cheers
        >     Mike
        >
        >     On 21/11/2016, 18:26, "Mayuresh Gharat" <gh...@gmail.com>
        > wrote:
        >
        >         Hi Michael,
        >
        >         I have updated the migration section of the KIP. Can you please
        > take a look?
        >
        >         Thanks,
        >
        >         Mayuresh
        >
        >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
        > gharatmayuresh15@gmail.com
        >         > wrote:
        >
        >         > Hi Michael,
        >         >
        >         > That whilst sending tombstone and non null value, the consumer
        > can expect
        >         > only to receive the non-null message only in step (3) is this
        > correct?
        >         > ---> I do agree with you here.
        >         >
        >         > Becket, Ismael : can you guys review the migration plan listed
        > above using
        >         > magic byte?
        >         >
        >         > Thanks,
        >         >
        >         > Mayuresh
        >         >
        >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
        > Michael.Pearce@ig.com>
        >         > wrote:
        >         >
        >         >> Many thanks for this Mayuresh. I don't have any objections.
        >         >>
        >         >> I assume we should state:
        >         >>
        >         >> That whilst sending tombstone and non null value, the consumer
        > can expect
        >         >> only to receive the non-null message only in step (3) is this
        > correct?
        >         >>
        >         >> Cheers
        >         >> Mike
        >         >>
        >         >>
        >         >>
        >         >> Sent using OWA for iPhone
        >         >> ________________________________________
        >         >> From: Mayuresh Gharat <gh...@gmail.com>
        >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
        >         >> To: dev@kafka.apache.org
        >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
        >         >>
        >         >> Hi Ismael,
        >         >>
        >         >> Thanks for the explanation.
        >         >> Specially I like this part where in you mentioned we can get
        > rid of the
        >         >> older null value support for log compaction later on, here :
        >         >> We can't change semantics of the message format without having
        > a long
        >         >> transition period. And we can't rely
        >         >> on people reading documentation or acting on a warning for
        > something so
        >         >> fundamental. As such, my take is that we need to bump the magic
        > byte. The
        >         >> good news is
        >         >> that we don't have to support all versions forever. We have
        > said that we
        >         >> will support direct upgrades for 2 years. That means that
        > message format
        >         >> version n could, in theory, be removed 2 years after the it's
        > introduced.
        >         >>
        >         >> Just a heads up, I would like to mention that even without
        > bumping magic
        >         >> byte, we will *NOT* loose zero copy as in the client(x+1) in my
        >         >> explanation
        >         >> above will convert internally a null value to have a tombstone
        > bit set and
        >         >> a tombstone bit set to have a null value automatically
        > internally and by
        >         >> the time we move to version (x+2), the clients would have
        > upgraded.
        >         >> Obviously if we support a request from consumer(x), we will
        > loose zero
        >         >> copy
        >         >> but that is the same case with magic byte.
        >         >>
        >         >> But if magic byte bump makes life easier for transition for the
        > above
        >         >> reasons that you explained, I am OK with it since we are going
        > to meet the
        >         >> end goal down the road :)
        >         >>
        >         >> On a side note can we update the doc here on magic byte to say
        > that "*it
        >         >> should be bumped whenever the message format is changed or the
        >         >> interpretation of message format (usage of the reserved bits as
        > well) is
        >         >> changed*".
        >         >>
        >         >>
        >         >> Hi Michael,
        >         >>
        >         >> Here is the update plan that we discussed offline yesterday :
        >         >>
        >         >> Currently the magic-byte which corresponds to the
        > "message.format.version"
        >         >> is set to 1.
        >         >>
        >         >> 1) On broker it will be set to 1 initially.
        >         >>
        >         >> 2) When a producer client sends a message with magic-byte = 2,
        > since the
        >         >> broker is on magic-byte = 1, we will down convert it, which
        > means if the
        >         >> tombstone bit is set, the value will be set to null. A consumer
        >         >> understanding magic-byte = 1, will still work with this. A
        > consumer
        >         >> working
        >         >> with magic-byte =2 will also be able to understand this, since
        > it
        >         >> understands the tombstone.
        >         >> Now there is still the question of supporting a non-tombstone
        > and null
        >         >> value from producer client with magic-byte = 2.* (I am not sure
        > if we
        >         >> should support this. Ismael/Becket can comment here)*
        >         >>
        >         >> 3) When almost all the clients have upgraded, the
        > message.format.version
        >         >> on
        >         >> the broker can be changed to 2, where in the down conversion in
        > the above
        >         >> step will not happen. If at this point we get a consumer
        > request from a
        >         >> older consumer, we might have to down convert where in we loose
        > zero copy,
        >         >> but these cases should be rare.
        >         >>
        >         >> Becket can you review this plan and add more details if I have
        >         >> missed/wronged something, before we put it on KIP.
        >         >>
        >         >> Thanks,
        >         >>
        >         >> Mayuresh
        >         >>
        >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
        > Michael.Pearce@ig.com>
        >         >> wrote:
        >         >>
        >         >> > Thanks guys, for discussing this offline and getting some
        > consensus.
        >         >> >
        >         >> > So its clear for myself and others what is proposed now (i
        > think i
        >         >> > understand, but want to make sure)
        >         >> >
        >         >> > Could i ask either directly update the kip to detail the
        > migration
        >         >> > strategy, or (re-)state your offline discussed and agreed
        > migration
        >         >> > strategy based on a magic byte is in this thread.
        >         >> >
        >         >> >
        >         >> > The main original driver for the KIP was to support
        > compaction where
        >         >> value
        >         >> > isn't null, based off the discussions on KIP-82 thread.
        >         >> >
        >         >> > We should be able to support non-tombstone + null value by the
        >         >> completion
        >         >> > of the KIP, as we noted when discussing this kip, having
        > logic based on
        >         >> a
        >         >> > null value isn't very clean and also separates the concerns.
        >         >> >
        >         >> > As discussed already though we can split this into KIP-87a
        > and KIP-87b
        >         >> >
        >         >> > Where we look to deliver KIP-87a on a compacted topic (to
        > address the
        >         >> > immediate issues)
        >         >> > * tombstone + null value
        >         >> > * tombstone + non-null value
        >         >> > * non-tombstone + non-null value
        >         >> >
        >         >> > Then we can discuss once KIP-87a is completed options later
        > and how we
        >         >> > support the second part KIP-87b to deliver:
        >         >> > * non-tombstone + null value
        >         >> >
        >         >> > Cheers
        >         >> > Mike
        >         >> >
        >         >> >
        >         >> >
        >         >> > ________________________________________
        >         >> > From: Becket Qin <be...@gmail.com>
        >         >> > Sent: Thursday, November 17, 2016 1:43 AM
        >         >> > To: dev@kafka.apache.org
        >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
        >         >> >
        >         >> > Renu, Mayuresh and I had an offline discussion, and following
        > is a brief
        >         >> > summary.
        >         >> >
        >         >> > 1. We agreed that not bumping up magic value may result in
        > losing zero
        >         >> copy
        >         >> > during migration.
        >         >> > 2. Given that bumping up magic value is almost free and has
        > benefit of
        >         >> > avoiding potential performance issue. It is probably worth
        > doing.
        >         >> >
        >         >> > One issue we still need to think about is whether we want to
        > support a
        >         >> > non-tombstone message with null value.
        >         >> > Currently it is not supported by Kafka. If we allow a
        > non-tombstone null
        >         >> > value message to exist after KIP-87. The problem is that such
        > message
        >         >> will
        >         >> > not be supported by the consumers prior to KIP-87. Because a
        > null value
        >         >> > will always be interpreted to a tombstone.
        >         >> >
        >         >> > One option is that we keep the current way, i.e. do not
        > support such
        >         >> > message. It would be good to know if there is a concrete use
        > case for
        >         >> such
        >         >> > message. If there is not, we can probably just not support it.
        >         >> >
        >         >> > Thanks,
        >         >> >
        >         >> > JIangjie (Becket) Qin
        >         >> >
        >         >> >
        >         >> >
        >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
        >         >> > gharatmayuresh15@gmail.com
        >         >> > > wrote:
        >         >> >
        >         >> > > Hi Ismael,
        >         >> > >
        >         >> > > This is something I can think of for migration plan:
        >         >> > > So the migration plan can look something like this, with up
        >         >> conversion :
        >         >> > >
        >         >> > > 1) Currently lets say we have Broker at version x.
        >         >> > > 2) Currently we have clients at version x.
        >         >> > > 3) a) We move the version to Broker(x+1) : supports both
        > tombstone and
        >         >> > null
        >         >> > > for log compaction.
        >         >> > >     b) We upgrade the client to version client(x+1) : if in
        > the
        >         >> producer
        >         >> > > client(x+1) the value is set to null, we will automatically
        > set the
        >         >> > > Tombstone bit internally. If the producer client(x+1) sets
        > the
        >         >> tombstone
        >         >> > > itself, well and good. For producer client(x), the broker
        > will up
        >         >> convert
        >         >> > > to have the tombstone bit. Broker(x+1) is supporting both.
        > Consumer
        >         >> > > client(x+1) will be aware of this and should be able to
        > handle this.
        >         >> For
        >         >> > > consumer client(x) we will down convert the message on the
        > broker
        >         >> side.
        >         >> > >     c) At this point we will have to specify a warning or
        > clearly
        >         >> specify
        >         >> > > in docs that this behavior is about to be changed for log
        > compaction.
        >         >> > > 4) a) In next release of the Broker(x+2), we say that only
        > Tombstone
        >         >> is
        >         >> > > used for log compaction on the Broker side. Clients(x+1)
        > still is
        >         >> > > supported.
        >         >> > >     b) We upgrade the client to version client(x+2) : if
        > value is set
        >         >> to
        >         >> > > null, tombstone will not be set automatically. The client
        > will have to
        >         >> > call
        >         >> > > setTombstone() to actually set the tombstone.
        >         >> > >
        >         >> > > We should compare this migration plan with the migration
        > plan for
        >         >> magic
        >         >> > > byte bump and do whatever looks good.
        >         >> > > I am just worried that if we go down magic byte route,
        > unless I am
        >         >> > missing
        >         >> > > something, it sounds like kafka will be stuck with
        > supporting both
        >         >> null
        >         >> > > value and tombstone bit for log compaction for life long,
        > which does
        >         >> not
        >         >> > > look like a good end state.
        >         >> > >
        >         >> > > Thanks,
        >         >> > >
        >         >> > > Mayuresh
        >         >> > >
        >         >> > >
        >         >> > >
        >         >> > >
        >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
        >         >> > > gharatmayuresh15@gmail.com
        >         >> > > > wrote:
        >         >> > >
        >         >> > > > Hi Ismael,
        >         >> > > >
        >         >> > > > That's a very good point which I might have not
        > considered earlier.
        >         >> > > >
        >         >> > > > Here is a plan that I can think of:
        >         >> > > >
        >         >> > > > Stage 1) The broker from now on, up converts the message
        > to have the
        >         >> > > > tombstone marker. The log compaction thread does log
        > compaction
        >         >> based
        >         >> > on
        >         >> > > > both null and tombstone marker. This is our transition
        > period.
        >         >> > > > Stage 2) The next release we only say that log compaction
        > is based
        >         >> on
        >         >> > > > tombstone marker. (Open source kafka makes this as a
        > policy). By
        >         >> this
        >         >> > > time,
        >         >> > > > the organization which is moving to this release will be
        > sure that
        >         >> they
        >         >> > > > have gone through the entire transition period.
        >         >> > > >
        >         >> > > > My only goal of doing this is that Kafka clearly
        > specifies the end
        >         >> > state
        >         >> > > > about what log compaction means (is it null value or a
        > tombstone
        >         >> > marker,
        >         >> > > > but not both).
        >         >> > > >
        >         >> > > > What do you think?
        >         >> > > >
        >         >> > > > Thanks,
        >         >> > > >
        >         >> > > > Mayuresh
        >         >> > > > .
        >         >> > > >
        >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
        > ismael@juma.me.uk>
        >         >> > wrote:
        >         >> > > >
        >         >> > > >> One comment below.
        >         >> > > >>
        >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
        >         >> > > >> gharatmayuresh15@gmail.com
        >         >> > > >> > wrote:
        >         >> > > >>
        >         >> > > >> >    - If we don't bump up the magic byte, on the broker
        > side, the
        >         >> > > broker
        >         >> > > >> >    will always have to look at both tombstone bit and
        > the value
        >         >> when
        >         >> > > do
        >         >> > > >> the
        >         >> > > >> >    compaction. Assuming we do not bump up the magic
        > byte,
        >         >> > > >> >    imagine the broker sees a message which does not
        > have a
        >         >> tombstone
        >         >> > > bit
        >         >> > > >> >    set. The broker does not know when the message was
        > produced
        >         >> (i.e.
        >         >> > > >> > whether
        >         >> > > >> >    the message has been up converted or not), it has
        > to take a
        >         >> > further
        >         >> > > >> > look at
        >         >> > > >> >    the value to see if it is null or not in order to
        > determine
        >         >> if it
        >         >> > > is
        >         >> > > >> a
        >         >> > > >> >    tombstone. The same logic has to be put on the
        > consumer as
        >         >> well
        >         >> > > >> because
        >         >> > > >> > the
        >         >> > > >> >    consumer does not know if the message has been up
        > converted or
        >         >> > not.
        >         >> > > >> >       - If we upconvert while appending, this is not
        > the case,
        >         >> > right?
        >         >> > > >>
        >         >> > > >>
        >         >> > > >> If I understand you correctly, this is not sufficient
        > because the
        >         >> log
        >         >> > > may
        >         >> > > >> have messages appended before it was upgraded to include
        > KIP-87.
        >         >> > > >>
        >         >> > > >> Ismael
        >         >> > > >>
        >         >> > > >
        >         >> > > >
        >         >> > > >
        >         >> > > > --
        >         >> > > > -Regards,
        >         >> > > > Mayuresh R. Gharat
        >         >> > > > (862) 250-7125
        >         >> > > >
        >         >> > >
        >         >> > >
        >         >> > >
        >         >> > > --
        >         >> > > -Regards,
        >         >> > > Mayuresh R. Gharat
        >         >> > > (862) 250-7125
        >         >> > >
        >         >> > The information contained in this email is strictly
        > confidential and for
        >         >> > the use of the addressee only, unless otherwise indicated. If
        > you are
        >         >> not
        >         >> > the intended recipient, please do not read, copy, use or
        > disclose to
        >         >> others
        >         >> > this message or any attachment. Please also notify the sender
        > by
        >         >> replying
        >         >> > to this email or by telephone (+44(020 7896 0011) and then
        > delete the
        >         >> email
        >         >> > and any copies of it. Opinions, conclusion (etc) that do not
        > relate to
        >         >> the
        >         >> > official business of this company shall be understood as
        > neither given
        >         >> nor
        >         >> > endorsed by it. IG is a trading name of IG Markets Limited (a
        > company
        >         >> > registered in England and Wales, company number 04008957) and
        > IG Index
        >         >> > Limited (a company registered in England and Wales, company
        > number
        >         >> > 01190902). Registered address at Cannon Bridge House, 25
        > Dowgate Hill,
        >         >> > London EC4R 2YA. Both IG Markets Limited (register number
        > 195355) and IG
        >         >> > Index Limited (register number 114059) are authorised and
        > regulated by
        >         >> the
        >         >> > Financial Conduct Authority.
        >         >> >
        >         >>
        >         >>
        >         >>
        >         >> --
        >         >> -Regards,
        >         >> Mayuresh R. Gharat
        >         >> (862) 250-7125
        >         >> The information contained in this email is strictly
        > confidential and for
        >         >> the use of the addressee only, unless otherwise indicated. If
        > you are not
        >         >> the intended recipient, please do not read, copy, use or
        > disclose to others
        >         >> this message or any attachment. Please also notify the sender
        > by replying
        >         >> to this email or by telephone (+44(020 7896 0011) and then
        > delete the email
        >         >> and any copies of it. Opinions, conclusion (etc) that do not
        > relate to the
        >         >> official business of this company shall be understood as
        > neither given nor
        >         >> endorsed by it. IG is a trading name of IG Markets Limited (a
        > company
        >         >> registered in England and Wales, company number 04008957) and
        > IG Index
        >         >> Limited (a company registered in England and Wales, company
        > number
        >         >> 01190902). Registered address at Cannon Bridge House, 25
        > Dowgate Hill,
        >         >> London EC4R 2YA. Both IG Markets Limited (register number
        > 195355) and IG
        >         >> Index Limited (register number 114059) are authorised and
        > regulated by the
        >         >> Financial Conduct Authority.
        >         >>
        >         >
        >         >
        >         >
        >         > --
        >         > -Regards,
        >         > Mayuresh R. Gharat
        >         > (862) 250-7125
        >         >
        >
        >
        >
        >         --
        >         -Regards,
        >         Mayuresh R. Gharat
        >         (862) 250-7125
        >
        >
        >     The information contained in this email is strictly confidential and
        > for the use of the addressee only, unless otherwise indicated. If you are
        > not the intended recipient, please do not read, copy, use or disclose to
        > others this message or any attachment. Please also notify the sender by
        > replying to this email or by telephone (+44(020 7896 0011) and then delete
        > the email and any copies of it. Opinions, conclusion (etc) that do not
        > relate to the official business of this company shall be understood as
        > neither given nor endorsed by it. IG is a trading name of IG Markets
        > Limited (a company registered in England and Wales, company number
        > 04008957) and IG Index Limited (a company registered in England and Wales,
        > company number 01190902). Registered address at Cannon Bridge House, 25
        > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
        > 195355) and IG Index Limited (register number 114059) are authorised and
        > regulated by the Financial Conduct Authority.
        >
        >
        >
    
    
    The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
    

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Hi Jao

Thanks for the response. Sorry for slow reply, both with personal sickness and also battling some critical issues encountered since upgrading to 0.10.1.0

1) Thans for spotting, Document error where we branched this KIP from KIP-82, will get that removed.
2) Intent is to do this just at the record message level.
3) Thanks for spotting, Will ensure this is corrected.
4) As per discussion thread we will support tombstone + null value, tombstone + non null value, no tombstone + null value.
5) I believe this was in the discussion thread, @Mayuresh is this something we’ve overlooked? I thought we would down convert and remove the value so the old consumer had existing behavior, or is there something we haven’t thought about?

Cheers
Mike

On 30/11/2016, 18:12, "Jun Rao" <ju...@confluent.io> wrote:

    Hi, Michael,

    Thanks for the KIP. A few comments below.

    1. The message format change contains "HeadersLength Headers". Is that
    intended?

    2. For compressed messageset, is the tombstone bit only set at the shallow
    level? Do we always leave that bit in the wrapper message unset? An
    alternative is to set the tombstone bit in the wrapper if at least one
    inner message has the tombstone bit set. This makes things a bit more
    complicated, but we could potentially exploit that for optimizing down
    conversion. For example, we only need to convert messages with magic 2 to
    magic 1 if the wrapper's tombstone bit is set (conversion is always needed
    from magic 2 to magic 0). Not sure if the optimization is worth the
    complexity though.

    3. The referencing of the new version of ProducerRequest/FetchRequest is
    inconsistent (v4 vs v3). Since our convention starts at version at 0, I
    think the new version would be 3.

    4. "If the magic byte on message is 2, the broker should use the tombstone
    bit for log compaction." What about null value? My understanding is that
    null value will be treated the same as setting the tombstone bit.

    5. For the migration path, it would be useful to describe the down
    conversion path to consumers (i.e., brokers on message format 0.10.2 and
    consumers on older version).

    Thanks,

    Jun


    On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <Mi...@ig.com>
    wrote:

    > Hi All,
    >
    > We have been discussing in the below thread and final changes have been
    > made to the KIP wiki based on these discussions.
    >
    > We would now like to put to the vote the following KIP:
    > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
    > Add+Compaction+Tombstone+Flag
    >
    > This kip is for having a distinct compaction attribute “tombstone” flag
    > instead of relying on null value, allowing non-null value delete messages.
    >
    > Many thanks,
    > Michael
    >
    >
    >
    > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com> wrote:
    >
    >     Hi Mayuresh,
    >
    >     LGTM. Ive just made one small adjustment updating the wire protocol to
    > show the magic byte bump.
    >
    >     Do we think we’re good to put to a vote? Is there any other bits
    > needing discussion?
    >
    >     Cheers
    >     Mike
    >
    >     On 21/11/2016, 18:26, "Mayuresh Gharat" <gh...@gmail.com>
    > wrote:
    >
    >         Hi Michael,
    >
    >         I have updated the migration section of the KIP. Can you please
    > take a look?
    >
    >         Thanks,
    >
    >         Mayuresh
    >
    >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
    > gharatmayuresh15@gmail.com
    >         > wrote:
    >
    >         > Hi Michael,
    >         >
    >         > That whilst sending tombstone and non null value, the consumer
    > can expect
    >         > only to receive the non-null message only in step (3) is this
    > correct?
    >         > ---> I do agree with you here.
    >         >
    >         > Becket, Ismael : can you guys review the migration plan listed
    > above using
    >         > magic byte?
    >         >
    >         > Thanks,
    >         >
    >         > Mayuresh
    >         >
    >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
    > Michael.Pearce@ig.com>
    >         > wrote:
    >         >
    >         >> Many thanks for this Mayuresh. I don't have any objections.
    >         >>
    >         >> I assume we should state:
    >         >>
    >         >> That whilst sending tombstone and non null value, the consumer
    > can expect
    >         >> only to receive the non-null message only in step (3) is this
    > correct?
    >         >>
    >         >> Cheers
    >         >> Mike
    >         >>
    >         >>
    >         >>
    >         >> Sent using OWA for iPhone
    >         >> ________________________________________
    >         >> From: Mayuresh Gharat <gh...@gmail.com>
    >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
    >         >> To: dev@kafka.apache.org
    >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
    >         >>
    >         >> Hi Ismael,
    >         >>
    >         >> Thanks for the explanation.
    >         >> Specially I like this part where in you mentioned we can get
    > rid of the
    >         >> older null value support for log compaction later on, here :
    >         >> We can't change semantics of the message format without having
    > a long
    >         >> transition period. And we can't rely
    >         >> on people reading documentation or acting on a warning for
    > something so
    >         >> fundamental. As such, my take is that we need to bump the magic
    > byte. The
    >         >> good news is
    >         >> that we don't have to support all versions forever. We have
    > said that we
    >         >> will support direct upgrades for 2 years. That means that
    > message format
    >         >> version n could, in theory, be removed 2 years after the it's
    > introduced.
    >         >>
    >         >> Just a heads up, I would like to mention that even without
    > bumping magic
    >         >> byte, we will *NOT* loose zero copy as in the client(x+1) in my
    >         >> explanation
    >         >> above will convert internally a null value to have a tombstone
    > bit set and
    >         >> a tombstone bit set to have a null value automatically
    > internally and by
    >         >> the time we move to version (x+2), the clients would have
    > upgraded.
    >         >> Obviously if we support a request from consumer(x), we will
    > loose zero
    >         >> copy
    >         >> but that is the same case with magic byte.
    >         >>
    >         >> But if magic byte bump makes life easier for transition for the
    > above
    >         >> reasons that you explained, I am OK with it since we are going
    > to meet the
    >         >> end goal down the road :)
    >         >>
    >         >> On a side note can we update the doc here on magic byte to say
    > that "*it
    >         >> should be bumped whenever the message format is changed or the
    >         >> interpretation of message format (usage of the reserved bits as
    > well) is
    >         >> changed*".
    >         >>
    >         >>
    >         >> Hi Michael,
    >         >>
    >         >> Here is the update plan that we discussed offline yesterday :
    >         >>
    >         >> Currently the magic-byte which corresponds to the
    > "message.format.version"
    >         >> is set to 1.
    >         >>
    >         >> 1) On broker it will be set to 1 initially.
    >         >>
    >         >> 2) When a producer client sends a message with magic-byte = 2,
    > since the
    >         >> broker is on magic-byte = 1, we will down convert it, which
    > means if the
    >         >> tombstone bit is set, the value will be set to null. A consumer
    >         >> understanding magic-byte = 1, will still work with this. A
    > consumer
    >         >> working
    >         >> with magic-byte =2 will also be able to understand this, since
    > it
    >         >> understands the tombstone.
    >         >> Now there is still the question of supporting a non-tombstone
    > and null
    >         >> value from producer client with magic-byte = 2.* (I am not sure
    > if we
    >         >> should support this. Ismael/Becket can comment here)*
    >         >>
    >         >> 3) When almost all the clients have upgraded, the
    > message.format.version
    >         >> on
    >         >> the broker can be changed to 2, where in the down conversion in
    > the above
    >         >> step will not happen. If at this point we get a consumer
    > request from a
    >         >> older consumer, we might have to down convert where in we loose
    > zero copy,
    >         >> but these cases should be rare.
    >         >>
    >         >> Becket can you review this plan and add more details if I have
    >         >> missed/wronged something, before we put it on KIP.
    >         >>
    >         >> Thanks,
    >         >>
    >         >> Mayuresh
    >         >>
    >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
    > Michael.Pearce@ig.com>
    >         >> wrote:
    >         >>
    >         >> > Thanks guys, for discussing this offline and getting some
    > consensus.
    >         >> >
    >         >> > So its clear for myself and others what is proposed now (i
    > think i
    >         >> > understand, but want to make sure)
    >         >> >
    >         >> > Could i ask either directly update the kip to detail the
    > migration
    >         >> > strategy, or (re-)state your offline discussed and agreed
    > migration
    >         >> > strategy based on a magic byte is in this thread.
    >         >> >
    >         >> >
    >         >> > The main original driver for the KIP was to support
    > compaction where
    >         >> value
    >         >> > isn't null, based off the discussions on KIP-82 thread.
    >         >> >
    >         >> > We should be able to support non-tombstone + null value by the
    >         >> completion
    >         >> > of the KIP, as we noted when discussing this kip, having
    > logic based on
    >         >> a
    >         >> > null value isn't very clean and also separates the concerns.
    >         >> >
    >         >> > As discussed already though we can split this into KIP-87a
    > and KIP-87b
    >         >> >
    >         >> > Where we look to deliver KIP-87a on a compacted topic (to
    > address the
    >         >> > immediate issues)
    >         >> > * tombstone + null value
    >         >> > * tombstone + non-null value
    >         >> > * non-tombstone + non-null value
    >         >> >
    >         >> > Then we can discuss once KIP-87a is completed options later
    > and how we
    >         >> > support the second part KIP-87b to deliver:
    >         >> > * non-tombstone + null value
    >         >> >
    >         >> > Cheers
    >         >> > Mike
    >         >> >
    >         >> >
    >         >> >
    >         >> > ________________________________________
    >         >> > From: Becket Qin <be...@gmail.com>
    >         >> > Sent: Thursday, November 17, 2016 1:43 AM
    >         >> > To: dev@kafka.apache.org
    >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
    >         >> >
    >         >> > Renu, Mayuresh and I had an offline discussion, and following
    > is a brief
    >         >> > summary.
    >         >> >
    >         >> > 1. We agreed that not bumping up magic value may result in
    > losing zero
    >         >> copy
    >         >> > during migration.
    >         >> > 2. Given that bumping up magic value is almost free and has
    > benefit of
    >         >> > avoiding potential performance issue. It is probably worth
    > doing.
    >         >> >
    >         >> > One issue we still need to think about is whether we want to
    > support a
    >         >> > non-tombstone message with null value.
    >         >> > Currently it is not supported by Kafka. If we allow a
    > non-tombstone null
    >         >> > value message to exist after KIP-87. The problem is that such
    > message
    >         >> will
    >         >> > not be supported by the consumers prior to KIP-87. Because a
    > null value
    >         >> > will always be interpreted to a tombstone.
    >         >> >
    >         >> > One option is that we keep the current way, i.e. do not
    > support such
    >         >> > message. It would be good to know if there is a concrete use
    > case for
    >         >> such
    >         >> > message. If there is not, we can probably just not support it.
    >         >> >
    >         >> > Thanks,
    >         >> >
    >         >> > JIangjie (Becket) Qin
    >         >> >
    >         >> >
    >         >> >
    >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
    >         >> > gharatmayuresh15@gmail.com
    >         >> > > wrote:
    >         >> >
    >         >> > > Hi Ismael,
    >         >> > >
    >         >> > > This is something I can think of for migration plan:
    >         >> > > So the migration plan can look something like this, with up
    >         >> conversion :
    >         >> > >
    >         >> > > 1) Currently lets say we have Broker at version x.
    >         >> > > 2) Currently we have clients at version x.
    >         >> > > 3) a) We move the version to Broker(x+1) : supports both
    > tombstone and
    >         >> > null
    >         >> > > for log compaction.
    >         >> > >     b) We upgrade the client to version client(x+1) : if in
    > the
    >         >> producer
    >         >> > > client(x+1) the value is set to null, we will automatically
    > set the
    >         >> > > Tombstone bit internally. If the producer client(x+1) sets
    > the
    >         >> tombstone
    >         >> > > itself, well and good. For producer client(x), the broker
    > will up
    >         >> convert
    >         >> > > to have the tombstone bit. Broker(x+1) is supporting both.
    > Consumer
    >         >> > > client(x+1) will be aware of this and should be able to
    > handle this.
    >         >> For
    >         >> > > consumer client(x) we will down convert the message on the
    > broker
    >         >> side.
    >         >> > >     c) At this point we will have to specify a warning or
    > clearly
    >         >> specify
    >         >> > > in docs that this behavior is about to be changed for log
    > compaction.
    >         >> > > 4) a) In next release of the Broker(x+2), we say that only
    > Tombstone
    >         >> is
    >         >> > > used for log compaction on the Broker side. Clients(x+1)
    > still is
    >         >> > > supported.
    >         >> > >     b) We upgrade the client to version client(x+2) : if
    > value is set
    >         >> to
    >         >> > > null, tombstone will not be set automatically. The client
    > will have to
    >         >> > call
    >         >> > > setTombstone() to actually set the tombstone.
    >         >> > >
    >         >> > > We should compare this migration plan with the migration
    > plan for
    >         >> magic
    >         >> > > byte bump and do whatever looks good.
    >         >> > > I am just worried that if we go down magic byte route,
    > unless I am
    >         >> > missing
    >         >> > > something, it sounds like kafka will be stuck with
    > supporting both
    >         >> null
    >         >> > > value and tombstone bit for log compaction for life long,
    > which does
    >         >> not
    >         >> > > look like a good end state.
    >         >> > >
    >         >> > > Thanks,
    >         >> > >
    >         >> > > Mayuresh
    >         >> > >
    >         >> > >
    >         >> > >
    >         >> > >
    >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
    >         >> > > gharatmayuresh15@gmail.com
    >         >> > > > wrote:
    >         >> > >
    >         >> > > > Hi Ismael,
    >         >> > > >
    >         >> > > > That's a very good point which I might have not
    > considered earlier.
    >         >> > > >
    >         >> > > > Here is a plan that I can think of:
    >         >> > > >
    >         >> > > > Stage 1) The broker from now on, up converts the message
    > to have the
    >         >> > > > tombstone marker. The log compaction thread does log
    > compaction
    >         >> based
    >         >> > on
    >         >> > > > both null and tombstone marker. This is our transition
    > period.
    >         >> > > > Stage 2) The next release we only say that log compaction
    > is based
    >         >> on
    >         >> > > > tombstone marker. (Open source kafka makes this as a
    > policy). By
    >         >> this
    >         >> > > time,
    >         >> > > > the organization which is moving to this release will be
    > sure that
    >         >> they
    >         >> > > > have gone through the entire transition period.
    >         >> > > >
    >         >> > > > My only goal of doing this is that Kafka clearly
    > specifies the end
    >         >> > state
    >         >> > > > about what log compaction means (is it null value or a
    > tombstone
    >         >> > marker,
    >         >> > > > but not both).
    >         >> > > >
    >         >> > > > What do you think?
    >         >> > > >
    >         >> > > > Thanks,
    >         >> > > >
    >         >> > > > Mayuresh
    >         >> > > > .
    >         >> > > >
    >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
    > ismael@juma.me.uk>
    >         >> > wrote:
    >         >> > > >
    >         >> > > >> One comment below.
    >         >> > > >>
    >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
    >         >> > > >> gharatmayuresh15@gmail.com
    >         >> > > >> > wrote:
    >         >> > > >>
    >         >> > > >> >    - If we don't bump up the magic byte, on the broker
    > side, the
    >         >> > > broker
    >         >> > > >> >    will always have to look at both tombstone bit and
    > the value
    >         >> when
    >         >> > > do
    >         >> > > >> the
    >         >> > > >> >    compaction. Assuming we do not bump up the magic
    > byte,
    >         >> > > >> >    imagine the broker sees a message which does not
    > have a
    >         >> tombstone
    >         >> > > bit
    >         >> > > >> >    set. The broker does not know when the message was
    > produced
    >         >> (i.e.
    >         >> > > >> > whether
    >         >> > > >> >    the message has been up converted or not), it has
    > to take a
    >         >> > further
    >         >> > > >> > look at
    >         >> > > >> >    the value to see if it is null or not in order to
    > determine
    >         >> if it
    >         >> > > is
    >         >> > > >> a
    >         >> > > >> >    tombstone. The same logic has to be put on the
    > consumer as
    >         >> well
    >         >> > > >> because
    >         >> > > >> > the
    >         >> > > >> >    consumer does not know if the message has been up
    > converted or
    >         >> > not.
    >         >> > > >> >       - If we upconvert while appending, this is not
    > the case,
    >         >> > right?
    >         >> > > >>
    >         >> > > >>
    >         >> > > >> If I understand you correctly, this is not sufficient
    > because the
    >         >> log
    >         >> > > may
    >         >> > > >> have messages appended before it was upgraded to include
    > KIP-87.
    >         >> > > >>
    >         >> > > >> Ismael
    >         >> > > >>
    >         >> > > >
    >         >> > > >
    >         >> > > >
    >         >> > > > --
    >         >> > > > -Regards,
    >         >> > > > Mayuresh R. Gharat
    >         >> > > > (862) 250-7125
    >         >> > > >
    >         >> > >
    >         >> > >
    >         >> > >
    >         >> > > --
    >         >> > > -Regards,
    >         >> > > Mayuresh R. Gharat
    >         >> > > (862) 250-7125
    >         >> > >
    >         >> > The information contained in this email is strictly
    > confidential and for
    >         >> > the use of the addressee only, unless otherwise indicated. If
    > you are
    >         >> not
    >         >> > the intended recipient, please do not read, copy, use or
    > disclose to
    >         >> others
    >         >> > this message or any attachment. Please also notify the sender
    > by
    >         >> replying
    >         >> > to this email or by telephone (+44(020 7896 0011) and then
    > delete the
    >         >> email
    >         >> > and any copies of it. Opinions, conclusion (etc) that do not
    > relate to
    >         >> the
    >         >> > official business of this company shall be understood as
    > neither given
    >         >> nor
    >         >> > endorsed by it. IG is a trading name of IG Markets Limited (a
    > company
    >         >> > registered in England and Wales, company number 04008957) and
    > IG Index
    >         >> > Limited (a company registered in England and Wales, company
    > number
    >         >> > 01190902). Registered address at Cannon Bridge House, 25
    > Dowgate Hill,
    >         >> > London EC4R 2YA. Both IG Markets Limited (register number
    > 195355) and IG
    >         >> > Index Limited (register number 114059) are authorised and
    > regulated by
    >         >> the
    >         >> > Financial Conduct Authority.
    >         >> >
    >         >>
    >         >>
    >         >>
    >         >> --
    >         >> -Regards,
    >         >> Mayuresh R. Gharat
    >         >> (862) 250-7125
    >         >> The information contained in this email is strictly
    > confidential and for
    >         >> the use of the addressee only, unless otherwise indicated. If
    > you are not
    >         >> the intended recipient, please do not read, copy, use or
    > disclose to others
    >         >> this message or any attachment. Please also notify the sender
    > by replying
    >         >> to this email or by telephone (+44(020 7896 0011) and then
    > delete the email
    >         >> and any copies of it. Opinions, conclusion (etc) that do not
    > relate to the
    >         >> official business of this company shall be understood as
    > neither given nor
    >         >> endorsed by it. IG is a trading name of IG Markets Limited (a
    > company
    >         >> registered in England and Wales, company number 04008957) and
    > IG Index
    >         >> Limited (a company registered in England and Wales, company
    > number
    >         >> 01190902). Registered address at Cannon Bridge House, 25
    > Dowgate Hill,
    >         >> London EC4R 2YA. Both IG Markets Limited (register number
    > 195355) and IG
    >         >> Index Limited (register number 114059) are authorised and
    > regulated by the
    >         >> Financial Conduct Authority.
    >         >>
    >         >
    >         >
    >         >
    >         > --
    >         > -Regards,
    >         > Mayuresh R. Gharat
    >         > (862) 250-7125
    >         >
    >
    >
    >
    >         --
    >         -Regards,
    >         Mayuresh R. Gharat
    >         (862) 250-7125
    >
    >
    >     The information contained in this email is strictly confidential and
    > for the use of the addressee only, unless otherwise indicated. If you are
    > not the intended recipient, please do not read, copy, use or disclose to
    > others this message or any attachment. Please also notify the sender by
    > replying to this email or by telephone (+44(020 7896 0011) and then delete
    > the email and any copies of it. Opinions, conclusion (etc) that do not
    > relate to the official business of this company shall be understood as
    > neither given nor endorsed by it. IG is a trading name of IG Markets
    > Limited (a company registered in England and Wales, company number
    > 04008957) and IG Index Limited (a company registered in England and Wales,
    > company number 01190902). Registered address at Cannon Bridge House, 25
    > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
    > 195355) and IG Index Limited (register number 114059) are authorised and
    > regulated by the Financial Conduct Authority.
    >
    >
    >


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Jun Rao <ju...@confluent.io>]

Hi, Michael,

Thanks for the KIP. A few comments below.

1. The message format change contains "HeadersLength Headers". Is that
intended?

2. For compressed messageset, is the tombstone bit only set at the shallow
level? Do we always leave that bit in the wrapper message unset? An
alternative is to set the tombstone bit in the wrapper if at least one
inner message has the tombstone bit set. This makes things a bit more
complicated, but we could potentially exploit that for optimizing down
conversion. For example, we only need to convert messages with magic 2 to
magic 1 if the wrapper's tombstone bit is set (conversion is always needed
from magic 2 to magic 0). Not sure if the optimization is worth the
complexity though.

3. The referencing of the new version of ProducerRequest/FetchRequest is
inconsistent (v4 vs v3). Since our convention starts at version at 0, I
think the new version would be 3.

4. "If the magic byte on message is 2, the broker should use the tombstone
bit for log compaction." What about null value? My understanding is that
null value will be treated the same as setting the tombstone bit.

5. For the migration path, it would be useful to describe the down
conversion path to consumers (i.e., brokers on message format 0.10.2 and
consumers on older version).

Thanks,

Jun


On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <Mi...@ig.com>
wrote:

> Hi All,
>
> We have been discussing in the below thread and final changes have been
> made to the KIP wiki based on these discussions.
>
> We would now like to put to the vote the following KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
> Add+Compaction+Tombstone+Flag
>
> This kip is for having a distinct compaction attribute “tombstone” flag
> instead of relying on null value, allowing non-null value delete messages.
>
> Many thanks,
> Michael
>
>
>
> On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com> wrote:
>
>     Hi Mayuresh,
>
>     LGTM. Ive just made one small adjustment updating the wire protocol to
> show the magic byte bump.
>
>     Do we think we’re good to put to a vote? Is there any other bits
> needing discussion?
>
>     Cheers
>     Mike
>
>     On 21/11/2016, 18:26, "Mayuresh Gharat" <gh...@gmail.com>
> wrote:
>
>         Hi Michael,
>
>         I have updated the migration section of the KIP. Can you please
> take a look?
>
>         Thanks,
>
>         Mayuresh
>
>         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com
>         > wrote:
>
>         > Hi Michael,
>         >
>         > That whilst sending tombstone and non null value, the consumer
> can expect
>         > only to receive the non-null message only in step (3) is this
> correct?
>         > ---> I do agree with you here.
>         >
>         > Becket, Ismael : can you guys review the migration plan listed
> above using
>         > magic byte?
>         >
>         > Thanks,
>         >
>         > Mayuresh
>         >
>         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>         > wrote:
>         >
>         >> Many thanks for this Mayuresh. I don't have any objections.
>         >>
>         >> I assume we should state:
>         >>
>         >> That whilst sending tombstone and non null value, the consumer
> can expect
>         >> only to receive the non-null message only in step (3) is this
> correct?
>         >>
>         >> Cheers
>         >> Mike
>         >>
>         >>
>         >>
>         >> Sent using OWA for iPhone
>         >> ________________________________________
>         >> From: Mayuresh Gharat <gh...@gmail.com>
>         >> Sent: Thursday, November 17, 2016 5:18:41 PM
>         >> To: dev@kafka.apache.org
>         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>         >>
>         >> Hi Ismael,
>         >>
>         >> Thanks for the explanation.
>         >> Specially I like this part where in you mentioned we can get
> rid of the
>         >> older null value support for log compaction later on, here :
>         >> We can't change semantics of the message format without having
> a long
>         >> transition period. And we can't rely
>         >> on people reading documentation or acting on a warning for
> something so
>         >> fundamental. As such, my take is that we need to bump the magic
> byte. The
>         >> good news is
>         >> that we don't have to support all versions forever. We have
> said that we
>         >> will support direct upgrades for 2 years. That means that
> message format
>         >> version n could, in theory, be removed 2 years after the it's
> introduced.
>         >>
>         >> Just a heads up, I would like to mention that even without
> bumping magic
>         >> byte, we will *NOT* loose zero copy as in the client(x+1) in my
>         >> explanation
>         >> above will convert internally a null value to have a tombstone
> bit set and
>         >> a tombstone bit set to have a null value automatically
> internally and by
>         >> the time we move to version (x+2), the clients would have
> upgraded.
>         >> Obviously if we support a request from consumer(x), we will
> loose zero
>         >> copy
>         >> but that is the same case with magic byte.
>         >>
>         >> But if magic byte bump makes life easier for transition for the
> above
>         >> reasons that you explained, I am OK with it since we are going
> to meet the
>         >> end goal down the road :)
>         >>
>         >> On a side note can we update the doc here on magic byte to say
> that "*it
>         >> should be bumped whenever the message format is changed or the
>         >> interpretation of message format (usage of the reserved bits as
> well) is
>         >> changed*".
>         >>
>         >>
>         >> Hi Michael,
>         >>
>         >> Here is the update plan that we discussed offline yesterday :
>         >>
>         >> Currently the magic-byte which corresponds to the
> "message.format.version"
>         >> is set to 1.
>         >>
>         >> 1) On broker it will be set to 1 initially.
>         >>
>         >> 2) When a producer client sends a message with magic-byte = 2,
> since the
>         >> broker is on magic-byte = 1, we will down convert it, which
> means if the
>         >> tombstone bit is set, the value will be set to null. A consumer
>         >> understanding magic-byte = 1, will still work with this. A
> consumer
>         >> working
>         >> with magic-byte =2 will also be able to understand this, since
> it
>         >> understands the tombstone.
>         >> Now there is still the question of supporting a non-tombstone
> and null
>         >> value from producer client with magic-byte = 2.* (I am not sure
> if we
>         >> should support this. Ismael/Becket can comment here)*
>         >>
>         >> 3) When almost all the clients have upgraded, the
> message.format.version
>         >> on
>         >> the broker can be changed to 2, where in the down conversion in
> the above
>         >> step will not happen. If at this point we get a consumer
> request from a
>         >> older consumer, we might have to down convert where in we loose
> zero copy,
>         >> but these cases should be rare.
>         >>
>         >> Becket can you review this plan and add more details if I have
>         >> missed/wronged something, before we put it on KIP.
>         >>
>         >> Thanks,
>         >>
>         >> Mayuresh
>         >>
>         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
> Michael.Pearce@ig.com>
>         >> wrote:
>         >>
>         >> > Thanks guys, for discussing this offline and getting some
> consensus.
>         >> >
>         >> > So its clear for myself and others what is proposed now (i
> think i
>         >> > understand, but want to make sure)
>         >> >
>         >> > Could i ask either directly update the kip to detail the
> migration
>         >> > strategy, or (re-)state your offline discussed and agreed
> migration
>         >> > strategy based on a magic byte is in this thread.
>         >> >
>         >> >
>         >> > The main original driver for the KIP was to support
> compaction where
>         >> value
>         >> > isn't null, based off the discussions on KIP-82 thread.
>         >> >
>         >> > We should be able to support non-tombstone + null value by the
>         >> completion
>         >> > of the KIP, as we noted when discussing this kip, having
> logic based on
>         >> a
>         >> > null value isn't very clean and also separates the concerns.
>         >> >
>         >> > As discussed already though we can split this into KIP-87a
> and KIP-87b
>         >> >
>         >> > Where we look to deliver KIP-87a on a compacted topic (to
> address the
>         >> > immediate issues)
>         >> > * tombstone + null value
>         >> > * tombstone + non-null value
>         >> > * non-tombstone + non-null value
>         >> >
>         >> > Then we can discuss once KIP-87a is completed options later
> and how we
>         >> > support the second part KIP-87b to deliver:
>         >> > * non-tombstone + null value
>         >> >
>         >> > Cheers
>         >> > Mike
>         >> >
>         >> >
>         >> >
>         >> > ________________________________________
>         >> > From: Becket Qin <be...@gmail.com>
>         >> > Sent: Thursday, November 17, 2016 1:43 AM
>         >> > To: dev@kafka.apache.org
>         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>         >> >
>         >> > Renu, Mayuresh and I had an offline discussion, and following
> is a brief
>         >> > summary.
>         >> >
>         >> > 1. We agreed that not bumping up magic value may result in
> losing zero
>         >> copy
>         >> > during migration.
>         >> > 2. Given that bumping up magic value is almost free and has
> benefit of
>         >> > avoiding potential performance issue. It is probably worth
> doing.
>         >> >
>         >> > One issue we still need to think about is whether we want to
> support a
>         >> > non-tombstone message with null value.
>         >> > Currently it is not supported by Kafka. If we allow a
> non-tombstone null
>         >> > value message to exist after KIP-87. The problem is that such
> message
>         >> will
>         >> > not be supported by the consumers prior to KIP-87. Because a
> null value
>         >> > will always be interpreted to a tombstone.
>         >> >
>         >> > One option is that we keep the current way, i.e. do not
> support such
>         >> > message. It would be good to know if there is a concrete use
> case for
>         >> such
>         >> > message. If there is not, we can probably just not support it.
>         >> >
>         >> > Thanks,
>         >> >
>         >> > JIangjie (Becket) Qin
>         >> >
>         >> >
>         >> >
>         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
>         >> > gharatmayuresh15@gmail.com
>         >> > > wrote:
>         >> >
>         >> > > Hi Ismael,
>         >> > >
>         >> > > This is something I can think of for migration plan:
>         >> > > So the migration plan can look something like this, with up
>         >> conversion :
>         >> > >
>         >> > > 1) Currently lets say we have Broker at version x.
>         >> > > 2) Currently we have clients at version x.
>         >> > > 3) a) We move the version to Broker(x+1) : supports both
> tombstone and
>         >> > null
>         >> > > for log compaction.
>         >> > >     b) We upgrade the client to version client(x+1) : if in
> the
>         >> producer
>         >> > > client(x+1) the value is set to null, we will automatically
> set the
>         >> > > Tombstone bit internally. If the producer client(x+1) sets
> the
>         >> tombstone
>         >> > > itself, well and good. For producer client(x), the broker
> will up
>         >> convert
>         >> > > to have the tombstone bit. Broker(x+1) is supporting both.
> Consumer
>         >> > > client(x+1) will be aware of this and should be able to
> handle this.
>         >> For
>         >> > > consumer client(x) we will down convert the message on the
> broker
>         >> side.
>         >> > >     c) At this point we will have to specify a warning or
> clearly
>         >> specify
>         >> > > in docs that this behavior is about to be changed for log
> compaction.
>         >> > > 4) a) In next release of the Broker(x+2), we say that only
> Tombstone
>         >> is
>         >> > > used for log compaction on the Broker side. Clients(x+1)
> still is
>         >> > > supported.
>         >> > >     b) We upgrade the client to version client(x+2) : if
> value is set
>         >> to
>         >> > > null, tombstone will not be set automatically. The client
> will have to
>         >> > call
>         >> > > setTombstone() to actually set the tombstone.
>         >> > >
>         >> > > We should compare this migration plan with the migration
> plan for
>         >> magic
>         >> > > byte bump and do whatever looks good.
>         >> > > I am just worried that if we go down magic byte route,
> unless I am
>         >> > missing
>         >> > > something, it sounds like kafka will be stuck with
> supporting both
>         >> null
>         >> > > value and tombstone bit for log compaction for life long,
> which does
>         >> not
>         >> > > look like a good end state.
>         >> > >
>         >> > > Thanks,
>         >> > >
>         >> > > Mayuresh
>         >> > >
>         >> > >
>         >> > >
>         >> > >
>         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
>         >> > > gharatmayuresh15@gmail.com
>         >> > > > wrote:
>         >> > >
>         >> > > > Hi Ismael,
>         >> > > >
>         >> > > > That's a very good point which I might have not
> considered earlier.
>         >> > > >
>         >> > > > Here is a plan that I can think of:
>         >> > > >
>         >> > > > Stage 1) The broker from now on, up converts the message
> to have the
>         >> > > > tombstone marker. The log compaction thread does log
> compaction
>         >> based
>         >> > on
>         >> > > > both null and tombstone marker. This is our transition
> period.
>         >> > > > Stage 2) The next release we only say that log compaction
> is based
>         >> on
>         >> > > > tombstone marker. (Open source kafka makes this as a
> policy). By
>         >> this
>         >> > > time,
>         >> > > > the organization which is moving to this release will be
> sure that
>         >> they
>         >> > > > have gone through the entire transition period.
>         >> > > >
>         >> > > > My only goal of doing this is that Kafka clearly
> specifies the end
>         >> > state
>         >> > > > about what log compaction means (is it null value or a
> tombstone
>         >> > marker,
>         >> > > > but not both).
>         >> > > >
>         >> > > > What do you think?
>         >> > > >
>         >> > > > Thanks,
>         >> > > >
>         >> > > > Mayuresh
>         >> > > > .
>         >> > > >
>         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
> ismael@juma.me.uk>
>         >> > wrote:
>         >> > > >
>         >> > > >> One comment below.
>         >> > > >>
>         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
>         >> > > >> gharatmayuresh15@gmail.com
>         >> > > >> > wrote:
>         >> > > >>
>         >> > > >> >    - If we don't bump up the magic byte, on the broker
> side, the
>         >> > > broker
>         >> > > >> >    will always have to look at both tombstone bit and
> the value
>         >> when
>         >> > > do
>         >> > > >> the
>         >> > > >> >    compaction. Assuming we do not bump up the magic
> byte,
>         >> > > >> >    imagine the broker sees a message which does not
> have a
>         >> tombstone
>         >> > > bit
>         >> > > >> >    set. The broker does not know when the message was
> produced
>         >> (i.e.
>         >> > > >> > whether
>         >> > > >> >    the message has been up converted or not), it has
> to take a
>         >> > further
>         >> > > >> > look at
>         >> > > >> >    the value to see if it is null or not in order to
> determine
>         >> if it
>         >> > > is
>         >> > > >> a
>         >> > > >> >    tombstone. The same logic has to be put on the
> consumer as
>         >> well
>         >> > > >> because
>         >> > > >> > the
>         >> > > >> >    consumer does not know if the message has been up
> converted or
>         >> > not.
>         >> > > >> >       - If we upconvert while appending, this is not
> the case,
>         >> > right?
>         >> > > >>
>         >> > > >>
>         >> > > >> If I understand you correctly, this is not sufficient
> because the
>         >> log
>         >> > > may
>         >> > > >> have messages appended before it was upgraded to include
> KIP-87.
>         >> > > >>
>         >> > > >> Ismael
>         >> > > >>
>         >> > > >
>         >> > > >
>         >> > > >
>         >> > > > --
>         >> > > > -Regards,
>         >> > > > Mayuresh R. Gharat
>         >> > > > (862) 250-7125
>         >> > > >
>         >> > >
>         >> > >
>         >> > >
>         >> > > --
>         >> > > -Regards,
>         >> > > Mayuresh R. Gharat
>         >> > > (862) 250-7125
>         >> > >
>         >> > The information contained in this email is strictly
> confidential and for
>         >> > the use of the addressee only, unless otherwise indicated. If
> you are
>         >> not
>         >> > the intended recipient, please do not read, copy, use or
> disclose to
>         >> others
>         >> > this message or any attachment. Please also notify the sender
> by
>         >> replying
>         >> > to this email or by telephone (+44(020 7896 0011) and then
> delete the
>         >> email
>         >> > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>         >> the
>         >> > official business of this company shall be understood as
> neither given
>         >> nor
>         >> > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>         >> > registered in England and Wales, company number 04008957) and
> IG Index
>         >> > Limited (a company registered in England and Wales, company
> number
>         >> > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
>         >> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
>         >> > Index Limited (register number 114059) are authorised and
> regulated by
>         >> the
>         >> > Financial Conduct Authority.
>         >> >
>         >>
>         >>
>         >>
>         >> --
>         >> -Regards,
>         >> Mayuresh R. Gharat
>         >> (862) 250-7125
>         >> The information contained in this email is strictly
> confidential and for
>         >> the use of the addressee only, unless otherwise indicated. If
> you are not
>         >> the intended recipient, please do not read, copy, use or
> disclose to others
>         >> this message or any attachment. Please also notify the sender
> by replying
>         >> to this email or by telephone (+44(020 7896 0011) and then
> delete the email
>         >> and any copies of it. Opinions, conclusion (etc) that do not
> relate to the
>         >> official business of this company shall be understood as
> neither given nor
>         >> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>         >> registered in England and Wales, company number 04008957) and
> IG Index
>         >> Limited (a company registered in England and Wales, company
> number
>         >> 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
>         >> London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
>         >> Index Limited (register number 114059) are authorised and
> regulated by the
>         >> Financial Conduct Authority.
>         >>
>         >
>         >
>         >
>         > --
>         > -Regards,
>         > Mayuresh R. Gharat
>         > (862) 250-7125
>         >
>
>
>
>         --
>         -Regards,
>         Mayuresh R. Gharat
>         (862) 250-7125
>
>
>     The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce <Mi...@ig.com>]

Thanks Becket, The intent was it would be immutable. This didn’t get picked up on during the discussion phase but a very good point about setter methods and make it constructor parameter instead. Have updated KIP, thanks for the feedback.


On 29/11/2016, 18:30, "Becket Qin" <be...@gmail.com> wrote:

    +1 (binding) with a minor comment below:

    The wiki suggests we are going to have a new interface setTombstone() in
    the ProducerRecord. I am wondering if there is any considerations given to
    just having the constructor take an additional field so the ProducerRecord
    remains not modifiable?

    Thanks,

    Jiangjie (Becket) Qin

    On Tue, Nov 29, 2016 at 8:33 AM, radai <ra...@gmail.com> wrote:

    > +1 (non-binding)
    >
    > On Tue, Nov 29, 2016 at 8:08 AM, <gh...@gmail.com> wrote:
    >
    > > +1 (non-binding)
    > >
    > > Thanks,
    > >
    > > Mayuresh
    > >
    > >
    > > > On Nov 29, 2016, at 3:18 AM, Michael Pearce <Mi...@ig.com>
    > > wrote:
    > > >
    > > > Hi All,
    > > >
    > > > We have been discussing in the below thread and final changes have been
    > > made to the KIP wiki based on these discussions.
    > > >
    > > > We would now like to put to the vote the following KIP:
    > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
    > > 87+-+Add+Compaction+Tombstone+Flag
    > > >
    > > > This kip is for having a distinct compaction attribute “tombstone” flag
    > > instead of relying on null value, allowing non-null value delete
    > messages.
    > > >
    > > > Many thanks,
    > > > Michael
    > > >
    > > >
    > > >
    > > > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com> wrote:
    > > >
    > > >    Hi Mayuresh,
    > > >
    > > >    LGTM. Ive just made one small adjustment updating the wire protocol
    > > to show the magic byte bump.
    > > >
    > > >    Do we think we’re good to put to a vote? Is there any other bits
    > > needing discussion?
    > > >
    > > >    Cheers
    > > >    Mike
    > > >
    > > >    On 21/11/2016, 18:26, "Mayuresh Gharat" <gharatmayuresh15@gmail.com
    > >
    > > wrote:
    > > >
    > > >        Hi Michael,
    > > >
    > > >        I have updated the migration section of the KIP. Can you please
    > > take a look?
    > > >
    > > >        Thanks,
    > > >
    > > >        Mayuresh
    > > >
    > > >        On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
    > > gharatmayuresh15@gmail.com
    > > >> wrote:
    > > >
    > > >> Hi Michael,
    > > >>
    > > >> That whilst sending tombstone and non null value, the consumer can
    > > expect
    > > >> only to receive the non-null message only in step (3) is this correct?
    > > >> ---> I do agree with you here.
    > > >>
    > > >> Becket, Ismael : can you guys review the migration plan listed above
    > > using
    > > >> magic byte?
    > > >>
    > > >> Thanks,
    > > >>
    > > >> Mayuresh
    > > >>
    > > >> On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
    > Michael.Pearce@ig.com>
    > > >> wrote:
    > > >>
    > > >>> Many thanks for this Mayuresh. I don't have any objections.
    > > >>>
    > > >>> I assume we should state:
    > > >>>
    > > >>> That whilst sending tombstone and non null value, the consumer can
    > > expect
    > > >>> only to receive the non-null message only in step (3) is this
    > correct?
    > > >>>
    > > >>> Cheers
    > > >>> Mike
    > > >>>
    > > >>>
    > > >>>
    > > >>> Sent using OWA for iPhone
    > > >>> ________________________________________
    > > >>> From: Mayuresh Gharat <gh...@gmail.com>
    > > >>> Sent: Thursday, November 17, 2016 5:18:41 PM
    > > >>> To: dev@kafka.apache.org
    > > >>> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
    > > >>>
    > > >>> Hi Ismael,
    > > >>>
    > > >>> Thanks for the explanation.
    > > >>> Specially I like this part where in you mentioned we can get rid of
    > the
    > > >>> older null value support for log compaction later on, here :
    > > >>> We can't change semantics of the message format without having a long
    > > >>> transition period. And we can't rely
    > > >>> on people reading documentation or acting on a warning for something
    > so
    > > >>> fundamental. As such, my take is that we need to bump the magic byte.
    > > The
    > > >>> good news is
    > > >>> that we don't have to support all versions forever. We have said that
    > > we
    > > >>> will support direct upgrades for 2 years. That means that message
    > > format
    > > >>> version n could, in theory, be removed 2 years after the it's
    > > introduced.
    > > >>>
    > > >>> Just a heads up, I would like to mention that even without bumping
    > > magic
    > > >>> byte, we will *NOT* loose zero copy as in the client(x+1) in my
    > > >>> explanation
    > > >>> above will convert internally a null value to have a tombstone bit
    > set
    > > and
    > > >>> a tombstone bit set to have a null value automatically internally and
    > > by
    > > >>> the time we move to version (x+2), the clients would have upgraded.
    > > >>> Obviously if we support a request from consumer(x), we will loose
    > zero
    > > >>> copy
    > > >>> but that is the same case with magic byte.
    > > >>>
    > > >>> But if magic byte bump makes life easier for transition for the above
    > > >>> reasons that you explained, I am OK with it since we are going to
    > meet
    > > the
    > > >>> end goal down the road :)
    > > >>>
    > > >>> On a side note can we update the doc here on magic byte to say that
    > > "*it
    > > >>> should be bumped whenever the message format is changed or the
    > > >>> interpretation of message format (usage of the reserved bits as well)
    > > is
    > > >>> changed*".
    > > >>>
    > > >>>
    > > >>> Hi Michael,
    > > >>>
    > > >>> Here is the update plan that we discussed offline yesterday :
    > > >>>
    > > >>> Currently the magic-byte which corresponds to the
    > > "message.format.version"
    > > >>> is set to 1.
    > > >>>
    > > >>> 1) On broker it will be set to 1 initially.
    > > >>>
    > > >>> 2) When a producer client sends a message with magic-byte = 2, since
    > > the
    > > >>> broker is on magic-byte = 1, we will down convert it, which means if
    > > the
    > > >>> tombstone bit is set, the value will be set to null. A consumer
    > > >>> understanding magic-byte = 1, will still work with this. A consumer
    > > >>> working
    > > >>> with magic-byte =2 will also be able to understand this, since it
    > > >>> understands the tombstone.
    > > >>> Now there is still the question of supporting a non-tombstone and
    > null
    > > >>> value from producer client with magic-byte = 2.* (I am not sure if we
    > > >>> should support this. Ismael/Becket can comment here)*
    > > >>>
    > > >>> 3) When almost all the clients have upgraded, the
    > > message.format.version
    > > >>> on
    > > >>> the broker can be changed to 2, where in the down conversion in the
    > > above
    > > >>> step will not happen. If at this point we get a consumer request
    > from a
    > > >>> older consumer, we might have to down convert where in we loose zero
    > > copy,
    > > >>> but these cases should be rare.
    > > >>>
    > > >>> Becket can you review this plan and add more details if I have
    > > >>> missed/wronged something, before we put it on KIP.
    > > >>>
    > > >>> Thanks,
    > > >>>
    > > >>> Mayuresh
    > > >>>
    > > >>> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
    > > Michael.Pearce@ig.com>
    > > >>> wrote:
    > > >>>
    > > >>>> Thanks guys, for discussing this offline and getting some consensus.
    > > >>>>
    > > >>>> So its clear for myself and others what is proposed now (i think i
    > > >>>> understand, but want to make sure)
    > > >>>>
    > > >>>> Could i ask either directly update the kip to detail the migration
    > > >>>> strategy, or (re-)state your offline discussed and agreed migration
    > > >>>> strategy based on a magic byte is in this thread.
    > > >>>>
    > > >>>>
    > > >>>> The main original driver for the KIP was to support compaction where
    > > >>> value
    > > >>>> isn't null, based off the discussions on KIP-82 thread.
    > > >>>>
    > > >>>> We should be able to support non-tombstone + null value by the
    > > >>> completion
    > > >>>> of the KIP, as we noted when discussing this kip, having logic based
    > > on
    > > >>> a
    > > >>>> null value isn't very clean and also separates the concerns.
    > > >>>>
    > > >>>> As discussed already though we can split this into KIP-87a and
    > KIP-87b
    > > >>>>
    > > >>>> Where we look to deliver KIP-87a on a compacted topic (to address
    > the
    > > >>>> immediate issues)
    > > >>>> * tombstone + null value
    > > >>>> * tombstone + non-null value
    > > >>>> * non-tombstone + non-null value
    > > >>>>
    > > >>>> Then we can discuss once KIP-87a is completed options later and how
    > we
    > > >>>> support the second part KIP-87b to deliver:
    > > >>>> * non-tombstone + null value
    > > >>>>
    > > >>>> Cheers
    > > >>>> Mike
    > > >>>>
    > > >>>>
    > > >>>>
    > > >>>> ________________________________________
    > > >>>> From: Becket Qin <be...@gmail.com>
    > > >>>> Sent: Thursday, November 17, 2016 1:43 AM
    > > >>>> To: dev@kafka.apache.org
    > > >>>> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
    > > >>>>
    > > >>>> Renu, Mayuresh and I had an offline discussion, and following is a
    > > brief
    > > >>>> summary.
    > > >>>>
    > > >>>> 1. We agreed that not bumping up magic value may result in losing
    > zero
    > > >>> copy
    > > >>>> during migration.
    > > >>>> 2. Given that bumping up magic value is almost free and has benefit
    > of
    > > >>>> avoiding potential performance issue. It is probably worth doing.
    > > >>>>
    > > >>>> One issue we still need to think about is whether we want to
    > support a
    > > >>>> non-tombstone message with null value.
    > > >>>> Currently it is not supported by Kafka. If we allow a non-tombstone
    > > null
    > > >>>> value message to exist after KIP-87. The problem is that such
    > message
    > > >>> will
    > > >>>> not be supported by the consumers prior to KIP-87. Because a null
    > > value
    > > >>>> will always be interpreted to a tombstone.
    > > >>>>
    > > >>>> One option is that we keep the current way, i.e. do not support such
    > > >>>> message. It would be good to know if there is a concrete use case
    > for
    > > >>> such
    > > >>>> message. If there is not, we can probably just not support it.
    > > >>>>
    > > >>>> Thanks,
    > > >>>>
    > > >>>> JIangjie (Becket) Qin
    > > >>>>
    > > >>>>
    > > >>>>
    > > >>>> On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
    > > >>>> gharatmayuresh15@gmail.com
    > > >>>>> wrote:
    > > >>>>
    > > >>>>> Hi Ismael,
    > > >>>>>
    > > >>>>> This is something I can think of for migration plan:
    > > >>>>> So the migration plan can look something like this, with up
    > > >>> conversion :
    > > >>>>>
    > > >>>>> 1) Currently lets say we have Broker at version x.
    > > >>>>> 2) Currently we have clients at version x.
    > > >>>>> 3) a) We move the version to Broker(x+1) : supports both tombstone
    > > and
    > > >>>> null
    > > >>>>> for log compaction.
    > > >>>>>    b) We upgrade the client to version client(x+1) : if in the
    > > >>> producer
    > > >>>>> client(x+1) the value is set to null, we will automatically set the
    > > >>>>> Tombstone bit internally. If the producer client(x+1) sets the
    > > >>> tombstone
    > > >>>>> itself, well and good. For producer client(x), the broker will up
    > > >>> convert
    > > >>>>> to have the tombstone bit. Broker(x+1) is supporting both. Consumer
    > > >>>>> client(x+1) will be aware of this and should be able to handle
    > this.
    > > >>> For
    > > >>>>> consumer client(x) we will down convert the message on the broker
    > > >>> side.
    > > >>>>>    c) At this point we will have to specify a warning or clearly
    > > >>> specify
    > > >>>>> in docs that this behavior is about to be changed for log
    > compaction.
    > > >>>>> 4) a) In next release of the Broker(x+2), we say that only
    > Tombstone
    > > >>> is
    > > >>>>> used for log compaction on the Broker side. Clients(x+1) still is
    > > >>>>> supported.
    > > >>>>>    b) We upgrade the client to version client(x+2) : if value is
    > set
    > > >>> to
    > > >>>>> null, tombstone will not be set automatically. The client will have
    > > to
    > > >>>> call
    > > >>>>> setTombstone() to actually set the tombstone.
    > > >>>>>
    > > >>>>> We should compare this migration plan with the migration plan for
    > > >>> magic
    > > >>>>> byte bump and do whatever looks good.
    > > >>>>> I am just worried that if we go down magic byte route, unless I am
    > > >>>> missing
    > > >>>>> something, it sounds like kafka will be stuck with supporting both
    > > >>> null
    > > >>>>> value and tombstone bit for log compaction for life long, which
    > does
    > > >>> not
    > > >>>>> look like a good end state.
    > > >>>>>
    > > >>>>> Thanks,
    > > >>>>>
    > > >>>>> Mayuresh
    > > >>>>>
    > > >>>>>
    > > >>>>>
    > > >>>>>
    > > >>>>> On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
    > > >>>>> gharatmayuresh15@gmail.com
    > > >>>>>> wrote:
    > > >>>>>
    > > >>>>>> Hi Ismael,
    > > >>>>>>
    > > >>>>>> That's a very good point which I might have not considered
    > earlier.
    > > >>>>>>
    > > >>>>>> Here is a plan that I can think of:
    > > >>>>>>
    > > >>>>>> Stage 1) The broker from now on, up converts the message to have
    > the
    > > >>>>>> tombstone marker. The log compaction thread does log compaction
    > > >>> based
    > > >>>> on
    > > >>>>>> both null and tombstone marker. This is our transition period.
    > > >>>>>> Stage 2) The next release we only say that log compaction is based
    > > >>> on
    > > >>>>>> tombstone marker. (Open source kafka makes this as a policy). By
    > > >>> this
    > > >>>>> time,
    > > >>>>>> the organization which is moving to this release will be sure that
    > > >>> they
    > > >>>>>> have gone through the entire transition period.
    > > >>>>>>
    > > >>>>>> My only goal of doing this is that Kafka clearly specifies the end
    > > >>>> state
    > > >>>>>> about what log compaction means (is it null value or a tombstone
    > > >>>> marker,
    > > >>>>>> but not both).
    > > >>>>>>
    > > >>>>>> What do you think?
    > > >>>>>>
    > > >>>>>> Thanks,
    > > >>>>>>
    > > >>>>>> Mayuresh
    > > >>>>>> .
    > > >>>>>>
    > > >>>>>> On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <is...@juma.me.uk>
    > > >>>> wrote:
    > > >>>>>>
    > > >>>>>>> One comment below.
    > > >>>>>>>
    > > >>>>>>> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
    > > >>>>>>> gharatmayuresh15@gmail.com
    > > >>>>>>>> wrote:
    > > >>>>>>>
    > > >>>>>>>>   - If we don't bump up the magic byte, on the broker side, the
    > > >>>>> broker
    > > >>>>>>>>   will always have to look at both tombstone bit and the value
    > > >>> when
    > > >>>>> do
    > > >>>>>>> the
    > > >>>>>>>>   compaction. Assuming we do not bump up the magic byte,
    > > >>>>>>>>   imagine the broker sees a message which does not have a
    > > >>> tombstone
    > > >>>>> bit
    > > >>>>>>>>   set. The broker does not know when the message was produced
    > > >>> (i.e.
    > > >>>>>>>> whether
    > > >>>>>>>>   the message has been up converted or not), it has to take a
    > > >>>> further
    > > >>>>>>>> look at
    > > >>>>>>>>   the value to see if it is null or not in order to determine
    > > >>> if it
    > > >>>>> is
    > > >>>>>>> a
    > > >>>>>>>>   tombstone. The same logic has to be put on the consumer as
    > > >>> well
    > > >>>>>>> because
    > > >>>>>>>> the
    > > >>>>>>>>   consumer does not know if the message has been up converted or
    > > >>>> not.
    > > >>>>>>>>      - If we upconvert while appending, this is not the case,
    > > >>>> right?
    > > >>>>>>>
    > > >>>>>>>
    > > >>>>>>> If I understand you correctly, this is not sufficient because the
    > > >>> log
    > > >>>>> may
    > > >>>>>>> have messages appended before it was upgraded to include KIP-87.
    > > >>>>>>>
    > > >>>>>>> Ismael
    > > >>>>>>
    > > >>>>>>
    > > >>>>>>
    > > >>>>>> --
    > > >>>>>> -Regards,
    > > >>>>>> Mayuresh R. Gharat
    > > >>>>>> (862) 250-7125
    > > >>>>>
    > > >>>>>
    > > >>>>>
    > > >>>>> --
    > > >>>>> -Regards,
    > > >>>>> Mayuresh R. Gharat
    > > >>>>> (862) 250-7125
    > > >>>> The information contained in this email is strictly confidential and
    > > for
    > > >>>> the use of the addressee only, unless otherwise indicated. If you
    > are
    > > >>> not
    > > >>>> the intended recipient, please do not read, copy, use or disclose to
    > > >>> others
    > > >>>> this message or any attachment. Please also notify the sender by
    > > >>> replying
    > > >>>> to this email or by telephone (+44(020 7896 0011) and then delete
    > the
    > > >>> email
    > > >>>> and any copies of it. Opinions, conclusion (etc) that do not relate
    > to
    > > >>> the
    > > >>>> official business of this company shall be understood as neither
    > given
    > > >>> nor
    > > >>>> endorsed by it. IG is a trading name of IG Markets Limited (a
    > company
    > > >>>> registered in England and Wales, company number 04008957) and IG
    > Index
    > > >>>> Limited (a company registered in England and Wales, company number
    > > >>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > Hill,
    > > >>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
    > and
    > > IG
    > > >>>> Index Limited (register number 114059) are authorised and regulated
    > by
    > > >>> the
    > > >>>> Financial Conduct Authority.
    > > >>>
    > > >>>
    > > >>>
    > > >>> --
    > > >>> -Regards,
    > > >>> Mayuresh R. Gharat
    > > >>> (862) 250-7125
    > > >>> The information contained in this email is strictly confidential and
    > > for
    > > >>> the use of the addressee only, unless otherwise indicated. If you are
    > > not
    > > >>> the intended recipient, please do not read, copy, use or disclose to
    > > others
    > > >>> this message or any attachment. Please also notify the sender by
    > > replying
    > > >>> to this email or by telephone (+44(020 7896 0011) and then delete the
    > > email
    > > >>> and any copies of it. Opinions, conclusion (etc) that do not relate
    > to
    > > the
    > > >>> official business of this company shall be understood as neither
    > given
    > > nor
    > > >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
    > > >>> registered in England and Wales, company number 04008957) and IG
    > Index
    > > >>> Limited (a company registered in England and Wales, company number
    > > >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
    > Hill,
    > > >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
    > > IG
    > > >>> Index Limited (register number 114059) are authorised and regulated
    > by
    > > the
    > > >>> Financial Conduct Authority.
    > > >>
    > > >>
    > > >>
    > > >> --
    > > >> -Regards,
    > > >> Mayuresh R. Gharat
    > > >> (862) 250-7125
    > > >
    > > >
    > > >
    > > >        --
    > > >        -Regards,
    > > >        Mayuresh R. Gharat
    > > >        (862) 250-7125
    > > >
    > > >
    > > >    The information contained in this email is strictly confidential and
    > > for the use of the addressee only, unless otherwise indicated. If you are
    > > not the intended recipient, please do not read, copy, use or disclose to
    > > others this message or any attachment. Please also notify the sender by
    > > replying to this email or by telephone (+44(020 7896 0011) and then
    > delete
    > > the email and any copies of it. Opinions, conclusion (etc) that do not
    > > relate to the official business of this company shall be understood as
    > > neither given nor endorsed by it. IG is a trading name of IG Markets
    > > Limited (a company registered in England and Wales, company number
    > > 04008957) and IG Index Limited (a company registered in England and
    > Wales,
    > > company number 01190902). Registered address at Cannon Bridge House, 25
    > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
    > > 195355) and IG Index Limited (register number 114059) are authorised and
    > > regulated by the Financial Conduct Authority.
    > > >
    > > >
    > >
    >


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[Becket Qin <be...@gmail.com>]

+1 (binding) with a minor comment below:

The wiki suggests we are going to have a new interface setTombstone() in
the ProducerRecord. I am wondering if there is any considerations given to
just having the constructor take an additional field so the ProducerRecord
remains not modifiable?

Thanks,

Jiangjie (Becket) Qin

On Tue, Nov 29, 2016 at 8:33 AM, radai <ra...@gmail.com> wrote:

> +1 (non-binding)
>
> On Tue, Nov 29, 2016 at 8:08 AM, <gh...@gmail.com> wrote:
>
> > +1 (non-binding)
> >
> > Thanks,
> >
> > Mayuresh
> >
> >
> > > On Nov 29, 2016, at 3:18 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> > >
> > > Hi All,
> > >
> > > We have been discussing in the below thread and final changes have been
> > made to the KIP wiki based on these discussions.
> > >
> > > We would now like to put to the vote the following KIP:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 87+-+Add+Compaction+Tombstone+Flag
> > >
> > > This kip is for having a distinct compaction attribute “tombstone” flag
> > instead of relying on null value, allowing non-null value delete
> messages.
> > >
> > > Many thanks,
> > > Michael
> > >
> > >
> > >
> > > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com> wrote:
> > >
> > >    Hi Mayuresh,
> > >
> > >    LGTM. Ive just made one small adjustment updating the wire protocol
> > to show the magic byte bump.
> > >
> > >    Do we think we’re good to put to a vote? Is there any other bits
> > needing discussion?
> > >
> > >    Cheers
> > >    Mike
> > >
> > >    On 21/11/2016, 18:26, "Mayuresh Gharat" <gharatmayuresh15@gmail.com
> >
> > wrote:
> > >
> > >        Hi Michael,
> > >
> > >        I have updated the migration section of the KIP. Can you please
> > take a look?
> > >
> > >        Thanks,
> > >
> > >        Mayuresh
> > >
> > >        On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com
> > >> wrote:
> > >
> > >> Hi Michael,
> > >>
> > >> That whilst sending tombstone and non null value, the consumer can
> > expect
> > >> only to receive the non-null message only in step (3) is this correct?
> > >> ---> I do agree with you here.
> > >>
> > >> Becket, Ismael : can you guys review the migration plan listed above
> > using
> > >> magic byte?
> > >>
> > >> Thanks,
> > >>
> > >> Mayuresh
> > >>
> > >> On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > >> wrote:
> > >>
> > >>> Many thanks for this Mayuresh. I don't have any objections.
> > >>>
> > >>> I assume we should state:
> > >>>
> > >>> That whilst sending tombstone and non null value, the consumer can
> > expect
> > >>> only to receive the non-null message only in step (3) is this
> correct?
> > >>>
> > >>> Cheers
> > >>> Mike
> > >>>
> > >>>
> > >>>
> > >>> Sent using OWA for iPhone
> > >>> ________________________________________
> > >>> From: Mayuresh Gharat <gh...@gmail.com>
> > >>> Sent: Thursday, November 17, 2016 5:18:41 PM
> > >>> To: dev@kafka.apache.org
> > >>> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> > >>>
> > >>> Hi Ismael,
> > >>>
> > >>> Thanks for the explanation.
> > >>> Specially I like this part where in you mentioned we can get rid of
> the
> > >>> older null value support for log compaction later on, here :
> > >>> We can't change semantics of the message format without having a long
> > >>> transition period. And we can't rely
> > >>> on people reading documentation or acting on a warning for something
> so
> > >>> fundamental. As such, my take is that we need to bump the magic byte.
> > The
> > >>> good news is
> > >>> that we don't have to support all versions forever. We have said that
> > we
> > >>> will support direct upgrades for 2 years. That means that message
> > format
> > >>> version n could, in theory, be removed 2 years after the it's
> > introduced.
> > >>>
> > >>> Just a heads up, I would like to mention that even without bumping
> > magic
> > >>> byte, we will *NOT* loose zero copy as in the client(x+1) in my
> > >>> explanation
> > >>> above will convert internally a null value to have a tombstone bit
> set
> > and
> > >>> a tombstone bit set to have a null value automatically internally and
> > by
> > >>> the time we move to version (x+2), the clients would have upgraded.
> > >>> Obviously if we support a request from consumer(x), we will loose
> zero
> > >>> copy
> > >>> but that is the same case with magic byte.
> > >>>
> > >>> But if magic byte bump makes life easier for transition for the above
> > >>> reasons that you explained, I am OK with it since we are going to
> meet
> > the
> > >>> end goal down the road :)
> > >>>
> > >>> On a side note can we update the doc here on magic byte to say that
> > "*it
> > >>> should be bumped whenever the message format is changed or the
> > >>> interpretation of message format (usage of the reserved bits as well)
> > is
> > >>> changed*".
> > >>>
> > >>>
> > >>> Hi Michael,
> > >>>
> > >>> Here is the update plan that we discussed offline yesterday :
> > >>>
> > >>> Currently the magic-byte which corresponds to the
> > "message.format.version"
> > >>> is set to 1.
> > >>>
> > >>> 1) On broker it will be set to 1 initially.
> > >>>
> > >>> 2) When a producer client sends a message with magic-byte = 2, since
> > the
> > >>> broker is on magic-byte = 1, we will down convert it, which means if
> > the
> > >>> tombstone bit is set, the value will be set to null. A consumer
> > >>> understanding magic-byte = 1, will still work with this. A consumer
> > >>> working
> > >>> with magic-byte =2 will also be able to understand this, since it
> > >>> understands the tombstone.
> > >>> Now there is still the question of supporting a non-tombstone and
> null
> > >>> value from producer client with magic-byte = 2.* (I am not sure if we
> > >>> should support this. Ismael/Becket can comment here)*
> > >>>
> > >>> 3) When almost all the clients have upgraded, the
> > message.format.version
> > >>> on
> > >>> the broker can be changed to 2, where in the down conversion in the
> > above
> > >>> step will not happen. If at this point we get a consumer request
> from a
> > >>> older consumer, we might have to down convert where in we loose zero
> > copy,
> > >>> but these cases should be rare.
> > >>>
> > >>> Becket can you review this plan and add more details if I have
> > >>> missed/wronged something, before we put it on KIP.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Mayuresh
> > >>>
> > >>> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > >>> wrote:
> > >>>
> > >>>> Thanks guys, for discussing this offline and getting some consensus.
> > >>>>
> > >>>> So its clear for myself and others what is proposed now (i think i
> > >>>> understand, but want to make sure)
> > >>>>
> > >>>> Could i ask either directly update the kip to detail the migration
> > >>>> strategy, or (re-)state your offline discussed and agreed migration
> > >>>> strategy based on a magic byte is in this thread.
> > >>>>
> > >>>>
> > >>>> The main original driver for the KIP was to support compaction where
> > >>> value
> > >>>> isn't null, based off the discussions on KIP-82 thread.
> > >>>>
> > >>>> We should be able to support non-tombstone + null value by the
> > >>> completion
> > >>>> of the KIP, as we noted when discussing this kip, having logic based
> > on
> > >>> a
> > >>>> null value isn't very clean and also separates the concerns.
> > >>>>
> > >>>> As discussed already though we can split this into KIP-87a and
> KIP-87b
> > >>>>
> > >>>> Where we look to deliver KIP-87a on a compacted topic (to address
> the
> > >>>> immediate issues)
> > >>>> * tombstone + null value
> > >>>> * tombstone + non-null value
> > >>>> * non-tombstone + non-null value
> > >>>>
> > >>>> Then we can discuss once KIP-87a is completed options later and how
> we
> > >>>> support the second part KIP-87b to deliver:
> > >>>> * non-tombstone + null value
> > >>>>
> > >>>> Cheers
> > >>>> Mike
> > >>>>
> > >>>>
> > >>>>
> > >>>> ________________________________________
> > >>>> From: Becket Qin <be...@gmail.com>
> > >>>> Sent: Thursday, November 17, 2016 1:43 AM
> > >>>> To: dev@kafka.apache.org
> > >>>> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> > >>>>
> > >>>> Renu, Mayuresh and I had an offline discussion, and following is a
> > brief
> > >>>> summary.
> > >>>>
> > >>>> 1. We agreed that not bumping up magic value may result in losing
> zero
> > >>> copy
> > >>>> during migration.
> > >>>> 2. Given that bumping up magic value is almost free and has benefit
> of
> > >>>> avoiding potential performance issue. It is probably worth doing.
> > >>>>
> > >>>> One issue we still need to think about is whether we want to
> support a
> > >>>> non-tombstone message with null value.
> > >>>> Currently it is not supported by Kafka. If we allow a non-tombstone
> > null
> > >>>> value message to exist after KIP-87. The problem is that such
> message
> > >>> will
> > >>>> not be supported by the consumers prior to KIP-87. Because a null
> > value
> > >>>> will always be interpreted to a tombstone.
> > >>>>
> > >>>> One option is that we keep the current way, i.e. do not support such
> > >>>> message. It would be good to know if there is a concrete use case
> for
> > >>> such
> > >>>> message. If there is not, we can probably just not support it.
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> JIangjie (Becket) Qin
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
> > >>>> gharatmayuresh15@gmail.com
> > >>>>> wrote:
> > >>>>
> > >>>>> Hi Ismael,
> > >>>>>
> > >>>>> This is something I can think of for migration plan:
> > >>>>> So the migration plan can look something like this, with up
> > >>> conversion :
> > >>>>>
> > >>>>> 1) Currently lets say we have Broker at version x.
> > >>>>> 2) Currently we have clients at version x.
> > >>>>> 3) a) We move the version to Broker(x+1) : supports both tombstone
> > and
> > >>>> null
> > >>>>> for log compaction.
> > >>>>>    b) We upgrade the client to version client(x+1) : if in the
> > >>> producer
> > >>>>> client(x+1) the value is set to null, we will automatically set the
> > >>>>> Tombstone bit internally. If the producer client(x+1) sets the
> > >>> tombstone
> > >>>>> itself, well and good. For producer client(x), the broker will up
> > >>> convert
> > >>>>> to have the tombstone bit. Broker(x+1) is supporting both. Consumer
> > >>>>> client(x+1) will be aware of this and should be able to handle
> this.
> > >>> For
> > >>>>> consumer client(x) we will down convert the message on the broker
> > >>> side.
> > >>>>>    c) At this point we will have to specify a warning or clearly
> > >>> specify
> > >>>>> in docs that this behavior is about to be changed for log
> compaction.
> > >>>>> 4) a) In next release of the Broker(x+2), we say that only
> Tombstone
> > >>> is
> > >>>>> used for log compaction on the Broker side. Clients(x+1) still is
> > >>>>> supported.
> > >>>>>    b) We upgrade the client to version client(x+2) : if value is
> set
> > >>> to
> > >>>>> null, tombstone will not be set automatically. The client will have
> > to
> > >>>> call
> > >>>>> setTombstone() to actually set the tombstone.
> > >>>>>
> > >>>>> We should compare this migration plan with the migration plan for
> > >>> magic
> > >>>>> byte bump and do whatever looks good.
> > >>>>> I am just worried that if we go down magic byte route, unless I am
> > >>>> missing
> > >>>>> something, it sounds like kafka will be stuck with supporting both
> > >>> null
> > >>>>> value and tombstone bit for log compaction for life long, which
> does
> > >>> not
> > >>>>> look like a good end state.
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Mayuresh
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
> > >>>>> gharatmayuresh15@gmail.com
> > >>>>>> wrote:
> > >>>>>
> > >>>>>> Hi Ismael,
> > >>>>>>
> > >>>>>> That's a very good point which I might have not considered
> earlier.
> > >>>>>>
> > >>>>>> Here is a plan that I can think of:
> > >>>>>>
> > >>>>>> Stage 1) The broker from now on, up converts the message to have
> the
> > >>>>>> tombstone marker. The log compaction thread does log compaction
> > >>> based
> > >>>> on
> > >>>>>> both null and tombstone marker. This is our transition period.
> > >>>>>> Stage 2) The next release we only say that log compaction is based
> > >>> on
> > >>>>>> tombstone marker. (Open source kafka makes this as a policy). By
> > >>> this
> > >>>>> time,
> > >>>>>> the organization which is moving to this release will be sure that
> > >>> they
> > >>>>>> have gone through the entire transition period.
> > >>>>>>
> > >>>>>> My only goal of doing this is that Kafka clearly specifies the end
> > >>>> state
> > >>>>>> about what log compaction means (is it null value or a tombstone
> > >>>> marker,
> > >>>>>> but not both).
> > >>>>>>
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Mayuresh
> > >>>>>> .
> > >>>>>>
> > >>>>>> On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <is...@juma.me.uk>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> One comment below.
> > >>>>>>>
> > >>>>>>> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
> > >>>>>>> gharatmayuresh15@gmail.com
> > >>>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>>   - If we don't bump up the magic byte, on the broker side, the
> > >>>>> broker
> > >>>>>>>>   will always have to look at both tombstone bit and the value
> > >>> when
> > >>>>> do
> > >>>>>>> the
> > >>>>>>>>   compaction. Assuming we do not bump up the magic byte,
> > >>>>>>>>   imagine the broker sees a message which does not have a
> > >>> tombstone
> > >>>>> bit
> > >>>>>>>>   set. The broker does not know when the message was produced
> > >>> (i.e.
> > >>>>>>>> whether
> > >>>>>>>>   the message has been up converted or not), it has to take a
> > >>>> further
> > >>>>>>>> look at
> > >>>>>>>>   the value to see if it is null or not in order to determine
> > >>> if it
> > >>>>> is
> > >>>>>>> a
> > >>>>>>>>   tombstone. The same logic has to be put on the consumer as
> > >>> well
> > >>>>>>> because
> > >>>>>>>> the
> > >>>>>>>>   consumer does not know if the message has been up converted or
> > >>>> not.
> > >>>>>>>>      - If we upconvert while appending, this is not the case,
> > >>>> right?
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> If I understand you correctly, this is not sufficient because the
> > >>> log
> > >>>>> may
> > >>>>>>> have messages appended before it was upgraded to include KIP-87.
> > >>>>>>>
> > >>>>>>> Ismael
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> -Regards,
> > >>>>>> Mayuresh R. Gharat
> > >>>>>> (862) 250-7125
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> -Regards,
> > >>>>> Mayuresh R. Gharat
> > >>>>> (862) 250-7125
> > >>>> The information contained in this email is strictly confidential and
> > for
> > >>>> the use of the addressee only, unless otherwise indicated. If you
> are
> > >>> not
> > >>>> the intended recipient, please do not read, copy, use or disclose to
> > >>> others
> > >>>> this message or any attachment. Please also notify the sender by
> > >>> replying
> > >>>> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > >>> email
> > >>>> and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > >>> the
> > >>>> official business of this company shall be understood as neither
> given
> > >>> nor
> > >>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > >>>> registered in England and Wales, company number 04008957) and IG
> Index
> > >>>> Limited (a company registered in England and Wales, company number
> > >>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > >>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > IG
> > >>>> Index Limited (register number 114059) are authorised and regulated
> by
> > >>> the
> > >>>> Financial Conduct Authority.
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> -Regards,
> > >>> Mayuresh R. Gharat
> > >>> (862) 250-7125
> > >>> The information contained in this email is strictly confidential and
> > for
> > >>> the use of the addressee only, unless otherwise indicated. If you are
> > not
> > >>> the intended recipient, please do not read, copy, use or disclose to
> > others
> > >>> this message or any attachment. Please also notify the sender by
> > replying
> > >>> to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > >>> and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > the
> > >>> official business of this company shall be understood as neither
> given
> > nor
> > >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
> > >>> registered in England and Wales, company number 04008957) and IG
> Index
> > >>> Limited (a company registered in England and Wales, company number
> > >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> > IG
> > >>> Index Limited (register number 114059) are authorised and regulated
> by
> > the
> > >>> Financial Conduct Authority.
> > >>
> > >>
> > >>
> > >> --
> > >> -Regards,
> > >> Mayuresh R. Gharat
> > >> (862) 250-7125
> > >
> > >
> > >
> > >        --
> > >        -Regards,
> > >        Mayuresh R. Gharat
> > >        (862) 250-7125
> > >
> > >
> > >    The information contained in this email is strictly confidential and
> > for the use of the addressee only, unless otherwise indicated. If you are
> > not the intended recipient, please do not read, copy, use or disclose to
> > others this message or any attachment. Please also notify the sender by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do not
> > relate to the official business of this company shall be understood as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG Index Limited (register number 114059) are authorised and
> > regulated by the Financial Conduct Authority.
> > >
> > >
> >
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[radai <ra...@gmail.com>]

+1 (non-binding)

On Tue, Nov 29, 2016 at 8:08 AM, <gh...@gmail.com> wrote:

> +1 (non-binding)
>
> Thanks,
>
> Mayuresh
>
>
> > On Nov 29, 2016, at 3:18 AM, Michael Pearce <Mi...@ig.com>
> wrote:
> >
> > Hi All,
> >
> > We have been discussing in the below thread and final changes have been
> made to the KIP wiki based on these discussions.
> >
> > We would now like to put to the vote the following KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 87+-+Add+Compaction+Tombstone+Flag
> >
> > This kip is for having a distinct compaction attribute “tombstone” flag
> instead of relying on null value, allowing non-null value delete messages.
> >
> > Many thanks,
> > Michael
> >
> >
> >
> > On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> >    Hi Mayuresh,
> >
> >    LGTM. Ive just made one small adjustment updating the wire protocol
> to show the magic byte bump.
> >
> >    Do we think we’re good to put to a vote? Is there any other bits
> needing discussion?
> >
> >    Cheers
> >    Mike
> >
> >    On 21/11/2016, 18:26, "Mayuresh Gharat" <gh...@gmail.com>
> wrote:
> >
> >        Hi Michael,
> >
> >        I have updated the migration section of the KIP. Can you please
> take a look?
> >
> >        Thanks,
> >
> >        Mayuresh
> >
> >        On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com
> >> wrote:
> >
> >> Hi Michael,
> >>
> >> That whilst sending tombstone and non null value, the consumer can
> expect
> >> only to receive the non-null message only in step (3) is this correct?
> >> ---> I do agree with you here.
> >>
> >> Becket, Ismael : can you guys review the migration plan listed above
> using
> >> magic byte?
> >>
> >> Thanks,
> >>
> >> Mayuresh
> >>
> >> On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <Mi...@ig.com>
> >> wrote:
> >>
> >>> Many thanks for this Mayuresh. I don't have any objections.
> >>>
> >>> I assume we should state:
> >>>
> >>> That whilst sending tombstone and non null value, the consumer can
> expect
> >>> only to receive the non-null message only in step (3) is this correct?
> >>>
> >>> Cheers
> >>> Mike
> >>>
> >>>
> >>>
> >>> Sent using OWA for iPhone
> >>> ________________________________________
> >>> From: Mayuresh Gharat <gh...@gmail.com>
> >>> Sent: Thursday, November 17, 2016 5:18:41 PM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> >>>
> >>> Hi Ismael,
> >>>
> >>> Thanks for the explanation.
> >>> Specially I like this part where in you mentioned we can get rid of the
> >>> older null value support for log compaction later on, here :
> >>> We can't change semantics of the message format without having a long
> >>> transition period. And we can't rely
> >>> on people reading documentation or acting on a warning for something so
> >>> fundamental. As such, my take is that we need to bump the magic byte.
> The
> >>> good news is
> >>> that we don't have to support all versions forever. We have said that
> we
> >>> will support direct upgrades for 2 years. That means that message
> format
> >>> version n could, in theory, be removed 2 years after the it's
> introduced.
> >>>
> >>> Just a heads up, I would like to mention that even without bumping
> magic
> >>> byte, we will *NOT* loose zero copy as in the client(x+1) in my
> >>> explanation
> >>> above will convert internally a null value to have a tombstone bit set
> and
> >>> a tombstone bit set to have a null value automatically internally and
> by
> >>> the time we move to version (x+2), the clients would have upgraded.
> >>> Obviously if we support a request from consumer(x), we will loose zero
> >>> copy
> >>> but that is the same case with magic byte.
> >>>
> >>> But if magic byte bump makes life easier for transition for the above
> >>> reasons that you explained, I am OK with it since we are going to meet
> the
> >>> end goal down the road :)
> >>>
> >>> On a side note can we update the doc here on magic byte to say that
> "*it
> >>> should be bumped whenever the message format is changed or the
> >>> interpretation of message format (usage of the reserved bits as well)
> is
> >>> changed*".
> >>>
> >>>
> >>> Hi Michael,
> >>>
> >>> Here is the update plan that we discussed offline yesterday :
> >>>
> >>> Currently the magic-byte which corresponds to the
> "message.format.version"
> >>> is set to 1.
> >>>
> >>> 1) On broker it will be set to 1 initially.
> >>>
> >>> 2) When a producer client sends a message with magic-byte = 2, since
> the
> >>> broker is on magic-byte = 1, we will down convert it, which means if
> the
> >>> tombstone bit is set, the value will be set to null. A consumer
> >>> understanding magic-byte = 1, will still work with this. A consumer
> >>> working
> >>> with magic-byte =2 will also be able to understand this, since it
> >>> understands the tombstone.
> >>> Now there is still the question of supporting a non-tombstone and null
> >>> value from producer client with magic-byte = 2.* (I am not sure if we
> >>> should support this. Ismael/Becket can comment here)*
> >>>
> >>> 3) When almost all the clients have upgraded, the
> message.format.version
> >>> on
> >>> the broker can be changed to 2, where in the down conversion in the
> above
> >>> step will not happen. If at this point we get a consumer request from a
> >>> older consumer, we might have to down convert where in we loose zero
> copy,
> >>> but these cases should be rare.
> >>>
> >>> Becket can you review this plan and add more details if I have
> >>> missed/wronged something, before we put it on KIP.
> >>>
> >>> Thanks,
> >>>
> >>> Mayuresh
> >>>
> >>> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> >>> wrote:
> >>>
> >>>> Thanks guys, for discussing this offline and getting some consensus.
> >>>>
> >>>> So its clear for myself and others what is proposed now (i think i
> >>>> understand, but want to make sure)
> >>>>
> >>>> Could i ask either directly update the kip to detail the migration
> >>>> strategy, or (re-)state your offline discussed and agreed migration
> >>>> strategy based on a magic byte is in this thread.
> >>>>
> >>>>
> >>>> The main original driver for the KIP was to support compaction where
> >>> value
> >>>> isn't null, based off the discussions on KIP-82 thread.
> >>>>
> >>>> We should be able to support non-tombstone + null value by the
> >>> completion
> >>>> of the KIP, as we noted when discussing this kip, having logic based
> on
> >>> a
> >>>> null value isn't very clean and also separates the concerns.
> >>>>
> >>>> As discussed already though we can split this into KIP-87a and KIP-87b
> >>>>
> >>>> Where we look to deliver KIP-87a on a compacted topic (to address the
> >>>> immediate issues)
> >>>> * tombstone + null value
> >>>> * tombstone + non-null value
> >>>> * non-tombstone + non-null value
> >>>>
> >>>> Then we can discuss once KIP-87a is completed options later and how we
> >>>> support the second part KIP-87b to deliver:
> >>>> * non-tombstone + null value
> >>>>
> >>>> Cheers
> >>>> Mike
> >>>>
> >>>>
> >>>>
> >>>> ________________________________________
> >>>> From: Becket Qin <be...@gmail.com>
> >>>> Sent: Thursday, November 17, 2016 1:43 AM
> >>>> To: dev@kafka.apache.org
> >>>> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> >>>>
> >>>> Renu, Mayuresh and I had an offline discussion, and following is a
> brief
> >>>> summary.
> >>>>
> >>>> 1. We agreed that not bumping up magic value may result in losing zero
> >>> copy
> >>>> during migration.
> >>>> 2. Given that bumping up magic value is almost free and has benefit of
> >>>> avoiding potential performance issue. It is probably worth doing.
> >>>>
> >>>> One issue we still need to think about is whether we want to support a
> >>>> non-tombstone message with null value.
> >>>> Currently it is not supported by Kafka. If we allow a non-tombstone
> null
> >>>> value message to exist after KIP-87. The problem is that such message
> >>> will
> >>>> not be supported by the consumers prior to KIP-87. Because a null
> value
> >>>> will always be interpreted to a tombstone.
> >>>>
> >>>> One option is that we keep the current way, i.e. do not support such
> >>>> message. It would be good to know if there is a concrete use case for
> >>> such
> >>>> message. If there is not, we can probably just not support it.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> JIangjie (Becket) Qin
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
> >>>> gharatmayuresh15@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> Hi Ismael,
> >>>>>
> >>>>> This is something I can think of for migration plan:
> >>>>> So the migration plan can look something like this, with up
> >>> conversion :
> >>>>>
> >>>>> 1) Currently lets say we have Broker at version x.
> >>>>> 2) Currently we have clients at version x.
> >>>>> 3) a) We move the version to Broker(x+1) : supports both tombstone
> and
> >>>> null
> >>>>> for log compaction.
> >>>>>    b) We upgrade the client to version client(x+1) : if in the
> >>> producer
> >>>>> client(x+1) the value is set to null, we will automatically set the
> >>>>> Tombstone bit internally. If the producer client(x+1) sets the
> >>> tombstone
> >>>>> itself, well and good. For producer client(x), the broker will up
> >>> convert
> >>>>> to have the tombstone bit. Broker(x+1) is supporting both. Consumer
> >>>>> client(x+1) will be aware of this and should be able to handle this.
> >>> For
> >>>>> consumer client(x) we will down convert the message on the broker
> >>> side.
> >>>>>    c) At this point we will have to specify a warning or clearly
> >>> specify
> >>>>> in docs that this behavior is about to be changed for log compaction.
> >>>>> 4) a) In next release of the Broker(x+2), we say that only Tombstone
> >>> is
> >>>>> used for log compaction on the Broker side. Clients(x+1) still is
> >>>>> supported.
> >>>>>    b) We upgrade the client to version client(x+2) : if value is set
> >>> to
> >>>>> null, tombstone will not be set automatically. The client will have
> to
> >>>> call
> >>>>> setTombstone() to actually set the tombstone.
> >>>>>
> >>>>> We should compare this migration plan with the migration plan for
> >>> magic
> >>>>> byte bump and do whatever looks good.
> >>>>> I am just worried that if we go down magic byte route, unless I am
> >>>> missing
> >>>>> something, it sounds like kafka will be stuck with supporting both
> >>> null
> >>>>> value and tombstone bit for log compaction for life long, which does
> >>> not
> >>>>> look like a good end state.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Mayuresh
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
> >>>>> gharatmayuresh15@gmail.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Hi Ismael,
> >>>>>>
> >>>>>> That's a very good point which I might have not considered earlier.
> >>>>>>
> >>>>>> Here is a plan that I can think of:
> >>>>>>
> >>>>>> Stage 1) The broker from now on, up converts the message to have the
> >>>>>> tombstone marker. The log compaction thread does log compaction
> >>> based
> >>>> on
> >>>>>> both null and tombstone marker. This is our transition period.
> >>>>>> Stage 2) The next release we only say that log compaction is based
> >>> on
> >>>>>> tombstone marker. (Open source kafka makes this as a policy). By
> >>> this
> >>>>> time,
> >>>>>> the organization which is moving to this release will be sure that
> >>> they
> >>>>>> have gone through the entire transition period.
> >>>>>>
> >>>>>> My only goal of doing this is that Kafka clearly specifies the end
> >>>> state
> >>>>>> about what log compaction means (is it null value or a tombstone
> >>>> marker,
> >>>>>> but not both).
> >>>>>>
> >>>>>> What do you think?
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Mayuresh
> >>>>>> .
> >>>>>>
> >>>>>> On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <is...@juma.me.uk>
> >>>> wrote:
> >>>>>>
> >>>>>>> One comment below.
> >>>>>>>
> >>>>>>> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
> >>>>>>> gharatmayuresh15@gmail.com
> >>>>>>>> wrote:
> >>>>>>>
> >>>>>>>>   - If we don't bump up the magic byte, on the broker side, the
> >>>>> broker
> >>>>>>>>   will always have to look at both tombstone bit and the value
> >>> when
> >>>>> do
> >>>>>>> the
> >>>>>>>>   compaction. Assuming we do not bump up the magic byte,
> >>>>>>>>   imagine the broker sees a message which does not have a
> >>> tombstone
> >>>>> bit
> >>>>>>>>   set. The broker does not know when the message was produced
> >>> (i.e.
> >>>>>>>> whether
> >>>>>>>>   the message has been up converted or not), it has to take a
> >>>> further
> >>>>>>>> look at
> >>>>>>>>   the value to see if it is null or not in order to determine
> >>> if it
> >>>>> is
> >>>>>>> a
> >>>>>>>>   tombstone. The same logic has to be put on the consumer as
> >>> well
> >>>>>>> because
> >>>>>>>> the
> >>>>>>>>   consumer does not know if the message has been up converted or
> >>>> not.
> >>>>>>>>      - If we upconvert while appending, this is not the case,
> >>>> right?
> >>>>>>>
> >>>>>>>
> >>>>>>> If I understand you correctly, this is not sufficient because the
> >>> log
> >>>>> may
> >>>>>>> have messages appended before it was upgraded to include KIP-87.
> >>>>>>>
> >>>>>>> Ismael
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> -Regards,
> >>>>>> Mayuresh R. Gharat
> >>>>>> (862) 250-7125
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -Regards,
> >>>>> Mayuresh R. Gharat
> >>>>> (862) 250-7125
> >>>> The information contained in this email is strictly confidential and
> for
> >>>> the use of the addressee only, unless otherwise indicated. If you are
> >>> not
> >>>> the intended recipient, please do not read, copy, use or disclose to
> >>> others
> >>>> this message or any attachment. Please also notify the sender by
> >>> replying
> >>>> to this email or by telephone (+44(020 7896 0011) and then delete the
> >>> email
> >>>> and any copies of it. Opinions, conclusion (etc) that do not relate to
> >>> the
> >>>> official business of this company shall be understood as neither given
> >>> nor
> >>>> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >>>> registered in England and Wales, company number 04008957) and IG Index
> >>>> Limited (a company registered in England and Wales, company number
> >>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >>>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >>>> Index Limited (register number 114059) are authorised and regulated by
> >>> the
> >>>> Financial Conduct Authority.
> >>>
> >>>
> >>>
> >>> --
> >>> -Regards,
> >>> Mayuresh R. Gharat
> >>> (862) 250-7125
> >>> The information contained in this email is strictly confidential and
> for
> >>> the use of the addressee only, unless otherwise indicated. If you are
> not
> >>> the intended recipient, please do not read, copy, use or disclose to
> others
> >>> this message or any attachment. Please also notify the sender by
> replying
> >>> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >>> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >>> official business of this company shall be understood as neither given
> nor
> >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >>> registered in England and Wales, company number 04008957) and IG Index
> >>> Limited (a company registered in England and Wales, company number
> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >>> Index Limited (register number 114059) are authorised and regulated by
> the
> >>> Financial Conduct Authority.
> >>
> >>
> >>
> >> --
> >> -Regards,
> >> Mayuresh R. Gharat
> >> (862) 250-7125
> >
> >
> >
> >        --
> >        -Regards,
> >        Mayuresh R. Gharat
> >        (862) 250-7125
> >
> >
> >    The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
> >
> >
>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

[gh...@gmail.com]

+1 (non-binding)

Thanks,

Mayuresh


> On Nov 29, 2016, at 3:18 AM, Michael Pearce <Mi...@ig.com> wrote:
> 
> Hi All,
> 
> We have been discussing in the below thread and final changes have been made to the KIP wiki based on these discussions.
> 
> We would now like to put to the vote the following KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+Add+Compaction+Tombstone+Flag
> 
> This kip is for having a distinct compaction attribute “tombstone” flag instead of relying on null value, allowing non-null value delete messages.
> 
> Many thanks,
> Michael
> 
> 
> 
> On 22/11/2016, 15:52, "Michael Pearce" <Mi...@ig.com> wrote:
> 
>    Hi Mayuresh,
> 
>    LGTM. Ive just made one small adjustment updating the wire protocol to show the magic byte bump.
> 
>    Do we think we’re good to put to a vote? Is there any other bits needing discussion?
> 
>    Cheers
>    Mike
> 
>    On 21/11/2016, 18:26, "Mayuresh Gharat" <gh...@gmail.com> wrote:
> 
>        Hi Michael,
> 
>        I have updated the migration section of the KIP. Can you please take a look?
> 
>        Thanks,
> 
>        Mayuresh
> 
>        On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com
>> wrote:
> 
>> Hi Michael,
>> 
>> That whilst sending tombstone and non null value, the consumer can expect
>> only to receive the non-null message only in step (3) is this correct?
>> ---> I do agree with you here.
>> 
>> Becket, Ismael : can you guys review the migration plan listed above using
>> magic byte?
>> 
>> Thanks,
>> 
>> Mayuresh
>> 
>> On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <Mi...@ig.com>
>> wrote:
>> 
>>> Many thanks for this Mayuresh. I don't have any objections.
>>> 
>>> I assume we should state:
>>> 
>>> That whilst sending tombstone and non null value, the consumer can expect
>>> only to receive the non-null message only in step (3) is this correct?
>>> 
>>> Cheers
>>> Mike
>>> 
>>> 
>>> 
>>> Sent using OWA for iPhone
>>> ________________________________________
>>> From: Mayuresh Gharat <gh...@gmail.com>
>>> Sent: Thursday, November 17, 2016 5:18:41 PM
>>> To: dev@kafka.apache.org
>>> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>>> 
>>> Hi Ismael,
>>> 
>>> Thanks for the explanation.
>>> Specially I like this part where in you mentioned we can get rid of the
>>> older null value support for log compaction later on, here :
>>> We can't change semantics of the message format without having a long
>>> transition period. And we can't rely
>>> on people reading documentation or acting on a warning for something so
>>> fundamental. As such, my take is that we need to bump the magic byte. The
>>> good news is
>>> that we don't have to support all versions forever. We have said that we
>>> will support direct upgrades for 2 years. That means that message format
>>> version n could, in theory, be removed 2 years after the it's introduced.
>>> 
>>> Just a heads up, I would like to mention that even without bumping magic
>>> byte, we will *NOT* loose zero copy as in the client(x+1) in my
>>> explanation
>>> above will convert internally a null value to have a tombstone bit set and
>>> a tombstone bit set to have a null value automatically internally and by
>>> the time we move to version (x+2), the clients would have upgraded.
>>> Obviously if we support a request from consumer(x), we will loose zero
>>> copy
>>> but that is the same case with magic byte.
>>> 
>>> But if magic byte bump makes life easier for transition for the above
>>> reasons that you explained, I am OK with it since we are going to meet the
>>> end goal down the road :)
>>> 
>>> On a side note can we update the doc here on magic byte to say that "*it
>>> should be bumped whenever the message format is changed or the
>>> interpretation of message format (usage of the reserved bits as well) is
>>> changed*".
>>> 
>>> 
>>> Hi Michael,
>>> 
>>> Here is the update plan that we discussed offline yesterday :
>>> 
>>> Currently the magic-byte which corresponds to the "message.format.version"
>>> is set to 1.
>>> 
>>> 1) On broker it will be set to 1 initially.
>>> 
>>> 2) When a producer client sends a message with magic-byte = 2, since the
>>> broker is on magic-byte = 1, we will down convert it, which means if the
>>> tombstone bit is set, the value will be set to null. A consumer
>>> understanding magic-byte = 1, will still work with this. A consumer
>>> working
>>> with magic-byte =2 will also be able to understand this, since it
>>> understands the tombstone.
>>> Now there is still the question of supporting a non-tombstone and null
>>> value from producer client with magic-byte = 2.* (I am not sure if we
>>> should support this. Ismael/Becket can comment here)*
>>> 
>>> 3) When almost all the clients have upgraded, the message.format.version
>>> on
>>> the broker can be changed to 2, where in the down conversion in the above
>>> step will not happen. If at this point we get a consumer request from a
>>> older consumer, we might have to down convert where in we loose zero copy,
>>> but these cases should be rare.
>>> 
>>> Becket can you review this plan and add more details if I have
>>> missed/wronged something, before we put it on KIP.
>>> 
>>> Thanks,
>>> 
>>> Mayuresh
>>> 
>>> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <Mi...@ig.com>
>>> wrote:
>>> 
>>>> Thanks guys, for discussing this offline and getting some consensus.
>>>> 
>>>> So its clear for myself and others what is proposed now (i think i
>>>> understand, but want to make sure)
>>>> 
>>>> Could i ask either directly update the kip to detail the migration
>>>> strategy, or (re-)state your offline discussed and agreed migration
>>>> strategy based on a magic byte is in this thread.
>>>> 
>>>> 
>>>> The main original driver for the KIP was to support compaction where
>>> value
>>>> isn't null, based off the discussions on KIP-82 thread.
>>>> 
>>>> We should be able to support non-tombstone + null value by the
>>> completion
>>>> of the KIP, as we noted when discussing this kip, having logic based on
>>> a
>>>> null value isn't very clean and also separates the concerns.
>>>> 
>>>> As discussed already though we can split this into KIP-87a and KIP-87b
>>>> 
>>>> Where we look to deliver KIP-87a on a compacted topic (to address the
>>>> immediate issues)
>>>> * tombstone + null value
>>>> * tombstone + non-null value
>>>> * non-tombstone + non-null value
>>>> 
>>>> Then we can discuss once KIP-87a is completed options later and how we
>>>> support the second part KIP-87b to deliver:
>>>> * non-tombstone + null value
>>>> 
>>>> Cheers
>>>> Mike
>>>> 
>>>> 
>>>> 
>>>> ________________________________________
>>>> From: Becket Qin <be...@gmail.com>
>>>> Sent: Thursday, November 17, 2016 1:43 AM
>>>> To: dev@kafka.apache.org
>>>> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>>>> 
>>>> Renu, Mayuresh and I had an offline discussion, and following is a brief
>>>> summary.
>>>> 
>>>> 1. We agreed that not bumping up magic value may result in losing zero
>>> copy
>>>> during migration.
>>>> 2. Given that bumping up magic value is almost free and has benefit of
>>>> avoiding potential performance issue. It is probably worth doing.
>>>> 
>>>> One issue we still need to think about is whether we want to support a
>>>> non-tombstone message with null value.
>>>> Currently it is not supported by Kafka. If we allow a non-tombstone null
>>>> value message to exist after KIP-87. The problem is that such message
>>> will
>>>> not be supported by the consumers prior to KIP-87. Because a null value
>>>> will always be interpreted to a tombstone.
>>>> 
>>>> One option is that we keep the current way, i.e. do not support such
>>>> message. It would be good to know if there is a concrete use case for
>>> such
>>>> message. If there is not, we can probably just not support it.
>>>> 
>>>> Thanks,
>>>> 
>>>> JIangjie (Becket) Qin
>>>> 
>>>> 
>>>> 
>>>> On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
>>>> gharatmayuresh15@gmail.com
>>>>> wrote:
>>>> 
>>>>> Hi Ismael,
>>>>> 
>>>>> This is something I can think of for migration plan:
>>>>> So the migration plan can look something like this, with up
>>> conversion :
>>>>> 
>>>>> 1) Currently lets say we have Broker at version x.
>>>>> 2) Currently we have clients at version x.
>>>>> 3) a) We move the version to Broker(x+1) : supports both tombstone and
>>>> null
>>>>> for log compaction.
>>>>>    b) We upgrade the client to version client(x+1) : if in the
>>> producer
>>>>> client(x+1) the value is set to null, we will automatically set the
>>>>> Tombstone bit internally. If the producer client(x+1) sets the
>>> tombstone
>>>>> itself, well and good. For producer client(x), the broker will up
>>> convert
>>>>> to have the tombstone bit. Broker(x+1) is supporting both. Consumer
>>>>> client(x+1) will be aware of this and should be able to handle this.
>>> For
>>>>> consumer client(x) we will down convert the message on the broker
>>> side.
>>>>>    c) At this point we will have to specify a warning or clearly
>>> specify
>>>>> in docs that this behavior is about to be changed for log compaction.
>>>>> 4) a) In next release of the Broker(x+2), we say that only Tombstone
>>> is
>>>>> used for log compaction on the Broker side. Clients(x+1) still is
>>>>> supported.
>>>>>    b) We upgrade the client to version client(x+2) : if value is set
>>> to
>>>>> null, tombstone will not be set automatically. The client will have to
>>>> call
>>>>> setTombstone() to actually set the tombstone.
>>>>> 
>>>>> We should compare this migration plan with the migration plan for
>>> magic
>>>>> byte bump and do whatever looks good.
>>>>> I am just worried that if we go down magic byte route, unless I am
>>>> missing
>>>>> something, it sounds like kafka will be stuck with supporting both
>>> null
>>>>> value and tombstone bit for log compaction for life long, which does
>>> not
>>>>> look like a good end state.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Mayuresh
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
>>>>> gharatmayuresh15@gmail.com
>>>>>> wrote:
>>>>> 
>>>>>> Hi Ismael,
>>>>>> 
>>>>>> That's a very good point which I might have not considered earlier.
>>>>>> 
>>>>>> Here is a plan that I can think of:
>>>>>> 
>>>>>> Stage 1) The broker from now on, up converts the message to have the
>>>>>> tombstone marker. The log compaction thread does log compaction
>>> based
>>>> on
>>>>>> both null and tombstone marker. This is our transition period.
>>>>>> Stage 2) The next release we only say that log compaction is based
>>> on
>>>>>> tombstone marker. (Open source kafka makes this as a policy). By
>>> this
>>>>> time,
>>>>>> the organization which is moving to this release will be sure that
>>> they
>>>>>> have gone through the entire transition period.
>>>>>> 
>>>>>> My only goal of doing this is that Kafka clearly specifies the end
>>>> state
>>>>>> about what log compaction means (is it null value or a tombstone
>>>> marker,
>>>>>> but not both).
>>>>>> 
>>>>>> What do you think?
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Mayuresh
>>>>>> .
>>>>>> 
>>>>>> On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <is...@juma.me.uk>
>>>> wrote:
>>>>>> 
>>>>>>> One comment below.
>>>>>>> 
>>>>>>> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
>>>>>>> gharatmayuresh15@gmail.com
>>>>>>>> wrote:
>>>>>>> 
>>>>>>>>   - If we don't bump up the magic byte, on the broker side, the
>>>>> broker
>>>>>>>>   will always have to look at both tombstone bit and the value
>>> when
>>>>> do
>>>>>>> the
>>>>>>>>   compaction. Assuming we do not bump up the magic byte,
>>>>>>>>   imagine the broker sees a message which does not have a
>>> tombstone
>>>>> bit
>>>>>>>>   set. The broker does not know when the message was produced
>>> (i.e.
>>>>>>>> whether
>>>>>>>>   the message has been up converted or not), it has to take a
>>>> further
>>>>>>>> look at
>>>>>>>>   the value to see if it is null or not in order to determine
>>> if it
>>>>> is
>>>>>>> a
>>>>>>>>   tombstone. The same logic has to be put on the consumer as
>>> well
>>>>>>> because
>>>>>>>> the
>>>>>>>>   consumer does not know if the message has been up converted or
>>>> not.
>>>>>>>>      - If we upconvert while appending, this is not the case,
>>>> right?
>>>>>>> 
>>>>>>> 
>>>>>>> If I understand you correctly, this is not sufficient because the
>>> log
>>>>> may
>>>>>>> have messages appended before it was upgraded to include KIP-87.
>>>>>>> 
>>>>>>> Ismael
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> -Regards,
>>>>>> Mayuresh R. Gharat
>>>>>> (862) 250-7125
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> -Regards,
>>>>> Mayuresh R. Gharat
>>>>> (862) 250-7125
>>>> The information contained in this email is strictly confidential and for
>>>> the use of the addressee only, unless otherwise indicated. If you are
>>> not
>>>> the intended recipient, please do not read, copy, use or disclose to
>>> others
>>>> this message or any attachment. Please also notify the sender by
>>> replying
>>>> to this email or by telephone (+44(020 7896 0011) and then delete the
>>> email
>>>> and any copies of it. Opinions, conclusion (etc) that do not relate to
>>> the
>>>> official business of this company shall be understood as neither given
>>> nor
>>>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>>>> registered in England and Wales, company number 04008957) and IG Index
>>>> Limited (a company registered in England and Wales, company number
>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>>>> Index Limited (register number 114059) are authorised and regulated by
>>> the
>>>> Financial Conduct Authority.
>>> 
>>> 
>>> 
>>> --
>>> -Regards,
>>> Mayuresh R. Gharat
>>> (862) 250-7125
>>> The information contained in this email is strictly confidential and for
>>> the use of the addressee only, unless otherwise indicated. If you are not
>>> the intended recipient, please do not read, copy, use or disclose to others
>>> this message or any attachment. Please also notify the sender by replying
>>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>>> official business of this company shall be understood as neither given nor
>>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>>> registered in England and Wales, company number 04008957) and IG Index
>>> Limited (a company registered in England and Wales, company number
>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>>> Index Limited (register number 114059) are authorised and regulated by the
>>> Financial Conduct Authority.
>> 
>> 
>> 
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
> 
> 
> 
>        --
>        -Regards,
>        Mayuresh R. Gharat
>        (862) 250-7125
> 
> 
>    The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
> 
>