Re: [kafka-clients] Re: Heads up: KAFKA-1697 - remove code related to ack>1 on the broker

[Mark Roberts <wi...@gmail.com>]

This would sting a whole lot less if there was a programmatic way to get what server version is in use. Also, how will this work in mixed version clusters (during an upgrade, for example)?


> On Jan 15, 2015, at 10:10, Joe Stein <jo...@stealth.ly> wrote:
> 
> Looping in the mailing list that the client developers live on because they are all not on dev (though they should be if they want to be helping to build the best client libraries they can).
> 
> I whole hardily believe that we need to not break existing functionality of the client protocol, ever.
> 
> There are many reasons for this and we have other threads on the mailing list where we are discussing that topic (no pun intended) that I don't want to re-hash here.
> 
> If we change wire protocol functionality OR the binary format (either) we must bump version AND treat version as a feature flag with backward compatibility support until it is deprecated for some time for folks to deal with it.
> 
> match version = {
> case 0: keepDoingWhatWeWereDoing()
> case 1: doNewStuff()
> case 2: doEvenMoreNewStuff()
> }
> 
> has to be a practice we adopt imho ... I know feature flags can be construed as "messy code" but I am eager to hear another (better? different?) solution to this.
> 
> If we don't do a feature flag like this specifically with this change then what happens is that someone upgrades their brokers with a rolling restart in 0.8.3 and every single one of their producer requests start to fail and they have a major production outage. eeeek!!!!
> 
> I do 100% agree that > 1 makes no sense and we *REALLY* need people to start using 0,1,-1 but we need to-do that in a way that is going to work for everyone.
> 
> Old producers and consumers must keep working with new brokers and if we are not going to support that then I am unclear what the use of "version" is based on our original intentions of having it because of the 0.7=>-0.8. We said no more breaking changes when we did that.
> 
> - Joe Stein
> 
>> On Thu, Jan 15, 2015 at 12:38 PM, Ewen Cheslack-Postava <ew...@confluent.io> wrote:
>> Right, so this looks like it could create an issue similar to what's
>> currently being discussed in
>> https://issues.apache.org/jira/browse/KAFKA-1649 where users now get errors
>> under conditions when they previously wouldn't. Old clients won't even know
>> about the error code, so besides failing they won't even be able to log any
>> meaningful error messages.
>> 
>> I think there are two options for compatibility:
>> 
>> 1. An alternative change is to remove the ack > 1 code, but silently
>> "upgrade" requests with acks > 1 to acks = -1. This isn't the same as other
>> changes to behavior since the interaction between the client and server
>> remains the same, no error codes change, etc. The client might just see
>> some increased latency since the message might need to be replicated to
>> more brokers than they requested.
>> 2. Split this into two patches, one that bumps the protocol version on that
>> message to include the new error code but maintains both old (now
>> deprecated) and new behavior, then a second that would be applied in a
>> later release that removes the old protocol + code for handling acks > 1.
>> 
>> 2 is probably the right thing to do. If we specify the release when we'll
>> remove the deprecated protocol at the time of deprecation it makes things a
>> lot easier for people writing non-java clients and could give users better
>> predictability (e.g. if clients are at most 1 major release behind brokers,
>> they'll remain compatible but possibly use deprecated features).
>> 
>> 
>> On Wed, Jan 14, 2015 at 3:51 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>> 
>> > Hi Kafka Devs,
>> >
>> > We are working on KAFKA-1697 - remove code related to ack>1 on the
>> > broker. Per Neha's suggestion, I'd like to give everyone a heads up on
>> > what these changes mean.
>> >
>> > Once this patch is included, any produce requests that include
>> > request.required.acks > 1 will result in an exception. This will be
>> > InvalidRequiredAcks in new versions (0.8.3 and up, I assume) and
>> > UnknownException in existing versions (sorry, but I can't add error
>> > codes retroactively).
>> >
>> > This behavior is already enforced by 0.8.2 producers (sync and new),
>> > but we expect impact on users with older producers that relied on acks
>> > > 1 and external clients (i.e python, go, etc).
>> >
>> > Users who relied on acks > 1 are expected to switch to using acks = -1
>> > and a min.isr parameter than matches their user case.
>> >
>> > This change was discussed in the past in the context of KAFKA-1555
>> > (min.isr), but let us know if you have any questions or concerns
>> > regarding this change.
>> >
>> > Gwen
>> >
>> 
>> 
>> 
>> --
>> Thanks,
>> Ewen
> 
> -- 
> You received this message because you are subscribed to the Google Groups "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kafka-clients+unsubscribe@googlegroups.com.
> To post to this group, send email to kafka-clients@googlegroups.com.
> Visit this group at http://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/CAA7ooCBtH2JjyQsArdx_%3DV25B4O1QJk0YvOu9U6kYt9sB4aqng%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.