You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Ying Zheng <yi...@uber.com.INVALID> on 2019/04/11 21:53:24 UTC

[VOTE] KIP-433: Block old clients on brokers

Hi here,

Please vote for
https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers

Thank you!

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Ismael Juma <is...@juma.me.uk>.

Hi Harsha,

Comments inline.

On Fri, Apr 12, 2019 at 8:22 PM Harsha <ka...@harsha.io> wrote:

> Hi Ismael,
>             I meant to say blocking clients based on their API version
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48
> But If I understand what you are saying, since each client release can
> support different versions for each of fetch, produce, offset commit etc..
> and it's harder to block just based on single min.api.version setting
> across different clients.
>

That's what I mean, yes.

> The idea I had in my mind was to do this via ApiVersionRequest, when a
> client makes api request to broker in response we return min and max
> version supported for each Api. When min.api.version enabled on broker, it
> returns the maxVersion it supports for each of the requests in that release
> as min versions to the clients.
>

Yes, my concern is not the implementation, that's straightforward, it's how
it's meant to be used.

Irrespective of the above approach I understand your point still stands
> which is sarama might not choose to implement all the higher version
> protocols for Kafka 1.1 release and they might introduce higher version of
> produce request in a subsequent minor release and it will be harder for
> users to figure out which release of sarama client they can use.
>

Yes, that's exactly right. The current KIP is not treating the Kafka
protocol as a specification that is implemented by multiple clients in
multiple languages. During KIP-35, it was agreed that the protocol would
not be tied to a particular Kafka version. The current KIP proposal doesn't
take that into account.

I looked at the latest updates and I don't think the core issue is
addressed:

https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=103091409&selectedPageVersions=15&selectedPageVersions=13

It would be useful to hear feedback from other people.

Ismael

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Harsha <ka...@harsha.io>.

Hi Gwen & Ismael,
                   Do you have any feedback on with the proposed approach, min.api.version allowing users to specify versions for every request.

Thanks,
Harsha

On Fri, Apr 19, 2019, at 10:24 AM, Harsha wrote:
> Thanks Ying for updating the KIP. 
> Hi Ismael,
>              Given min.api.version allows admin/users to specifiy 
> min.version for each request this should address your concerns right?
> 
> Thanks,
> Harsha
> 
> On Wed, Apr 17, 2019, at 2:29 PM, Ying Zheng wrote:
> > I have updated the config description in the KIP, made the example more
> > clear
> > 
> > The proposed change allows setting different min versions for different
> > APIs, and the ApiVersionRequest change is already in the KIP.
> > 
> > On Fri, Apr 12, 2019 at 8:22 PM Harsha <ka...@harsha.io> wrote:
> > 
> > > Hi Ismael,
> > >             I meant to say blocking clients based on their API version
> > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48
> > > But If I understand what you are saying, since each client release can
> > > support different versions for each of fetch, produce, offset commit etc..
> > > and it's harder to block just based on single min.api.version setting
> > > across different clients.
> > > The idea I had in my mind was to do this via ApiVersionRequest, when a
> > > client makes api request to broker in response we return min and max
> > > version supported for each Api. When min.api.version enabled on broker, it
> > > returns the maxVersion it supports for each of the requests in that release
> > > as min versions to the clients.
> > >
> > > Example:
> > > Kafka 1.1.1 broker and min.api.verson set to
> > > https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79
> > > (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for
> > > example produce request
> > >
> > > https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112
> > > Instead of returning all of the supported versions it will return
> > > PRODUCE_REQUEST_V5 as the only supported version.
> > >
> > > Irrespective of the above approach I understand your point still stands
> > > which is sarama might not choose to implement all the higher version
> > > protocols for Kafka 1.1 release and they might introduce higher version of
> > > produce request in a subsequent minor release and it will be harder for
> > > users to figure out which release of sarama client they can use.
> > >
> > >
> > > Ying, if you have a different apporach which might address this issue
> > > please add.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote:
> > > > Hi Harsha,
> > > >
> > > > There is no such thing as 1.1 protocol. I encourage you to describe an
> > > > example config that achieves what you are suggesting here. It's pretty
> > > > complicated because the versions are per API and each client evolves
> > > > independently.
> > > >
> > > > Ismael
> > > >
> > > > On Sat, Apr 13, 2019 at 4:09 AM Harsha <ka...@harsha.io> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > "Relying on min.version seems like a pretty clunky way to achieve the
> > > above
> > > > > > list. The challenge is that it's pretty difficult to do it in a way
> > > that
> > > > > > works for clients across languages. They each add support for new
> > > > > protocol
> > > > > > versions independently (it could even happen in a bug fix release).
> > > So,
> > > > > if
> > > > > > you tried to block Sarama in #2, you may block Java clients too."
> > > > >
> > > > > That's the intended effect, right?  if you as the admin/operator
> > > > > configures the broker to have min.api.version to be 1.1
> > > > > it should block java , sarama clients etc.. which are below the 1.1
> > > > > protocol.  As mentioned this is not just related to log.format upgrade
> > > > > problem but in general a forcing cause to get the users to upgrade
> > > their
> > > > > client version in a multi-tenant environment.
> > > > >
> > > > > "> For #3, it seems simplest to have a config that requires clients to
> > > > > support
> > > > > > a given message format version (or higher). For #2, it seems like
> > > you'd
> > > > > > want clients to advertise their versions. That would be useful for
> > > > > multiple
> > > > > > reasons."
> > > > > This kip offers the ability to block clients based on the protocol they
> > > > > support. This should be independent of the message format upgrade. Not
> > > all
> > > > > of the features or bugs are dependent on a message format and having a
> > > > > message format dependency to block clients means we have to upgrade to
> > > > > message.format and we cannot just say we've 1.1 brokers with 0.8.2
> > > message
> > > > > format and now we want to block all 0.8.x clients.
> > > > >
> > > > > min.api.version helps at the cluster level to say that all users
> > > required
> > > > > to upgrade clients to the at minimum need to speak the min.api.version
> > > and
> > > > > not tie to message.format because not all cases one wants to upgrade
> > > the
> > > > > message format and block the old clients.
> > > > >
> > > > >
> > > > > To Gwen's point, I think we should also return in the error message
> > > that
> > > > > the broker only supports min.api.version and above. So that users can
> > > see a
> > > > > clear message and upgrade to a newer version.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > >
> > > > > On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote:
> > > > > > Hi Ying,
> > > > > >
> > > > > > The actual reasons are important so that people can evaluate the KIP
> > > (and
> > > > > > vote). :) Thanks for providing a few more:
> > > > > >
> > > > > > (1) force users to check pointing in Kafka instead of zookeeper
> > > > > > (2) forbid an old go (sarama) client library which is known to have
> > > some
> > > > > > serious bugs
> > > > > > (3) force kafka 1.x clients with the ability to roll back if there's
> > > an
> > > > > > issue (unlike a message format upgrade)
> > > > > >
> > > > > > Relying on min.version seems like a pretty clunky way to achieve the
> > > > > above
> > > > > > list. The challenge is that it's pretty difficult to do it in a way
> > > that
> > > > > > works for clients across languages. They each add support for new
> > > > > protocol
> > > > > > versions independently (it could even happen in a bug fix release).
> > > So,
> > > > > if
> > > > > > you tried to block Sarama in #2, you may block Java clients too.
> > > > > >
> > > > > > For #3, it seems simplest to have a config that requires clients to
> > > > > support
> > > > > > a given message format version (or higher). For #2, it seems like
> > > you'd
> > > > > > want clients to advertise their versions. That would be useful for
> > > > > multiple
> > > > > > reasons.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Ismael,
> > > > > > >
> > > > > > > Those are just examples. I think the administrators should be able
> > > to
> > > > > block
> > > > > > > certain client libraries for whatever reason. Some other possible
> > > > > reasons
> > > > > > > include, force users to check pointing in Kafka instead of
> > > zookeeper,
> > > > > > > forbid an old go (sarama) client library which is known to have
> > > some
> > > > > > > serious bugs.
> > > > > > >
> > > > > > > message.downconversion.enable does not solve our problems. We are
> > > now
> > > > > > > planning to upgrade to message format V3, and force users to
> > > upgrade to
> > > > > > > Kafka 1.x clients. With the proposed min.api.version setting, in
> > > case
> > > > > of
> > > > > > > there is anything wrong, we can roll back the setting. If we
> > > upgrade
> > > > > the
> > > > > > > file format, there is no way to rollback (Kafka doesn't support
> > > > > downgrading
> > > > > > > message format).
> > > > > > >
> > > > > > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Ying,
> > > > > > > >
> > > > > > > > It looks to me that all the examples given in the KIP can be
> > > handled
> > > > > with
> > > > > > > > the existing "message.downconversion.enable" config and by
> > > > > configuring
> > > > > > > the
> > > > > > > > message format to be the latest:
> > > > > > > >
> > > > > > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains
> > > message
> > > > > > > header
> > > > > > > > > ( KAFKA-6739 - Down-conversion fails for records with headers
> > > > > > > RESOLVED  )
> > > > > > > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 (
> > > > > KAFKA-3160 -
> > > > > > > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED
> > > )
> > > > > > > > > 3. Performance penalty of converting message format from V3 to
> > > V1
> > > > > or V2
> > > > > > > > > for the old consumers (KIP-31 - Move to relative offsets in
> > > > > compressed
> > > > > > > > > message sets)
> > > > > > > >
> > > > > > > >
> > > > > > > > Am I missing something? Are there other examples that are not
> > > > > related to
> > > > > > > > message conversion?
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng
> > > <yi...@uber.com.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi here,
> > > > > > > > >
> > > > > > > > > Please vote for
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > > > > > > > >
> > > > > > > > > Thank you!
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Harsha <ka...@harsha.io>.

Thanks Ying for updating the KIP. 
Hi Ismael,
             Given min.api.version allows admin/users to specifiy min.version for each request this should address your concerns right?

Thanks,
Harsha

On Wed, Apr 17, 2019, at 2:29 PM, Ying Zheng wrote:
> I have updated the config description in the KIP, made the example more
> clear
> 
> The proposed change allows setting different min versions for different
> APIs, and the ApiVersionRequest change is already in the KIP.
> 
> On Fri, Apr 12, 2019 at 8:22 PM Harsha <ka...@harsha.io> wrote:
> 
> > Hi Ismael,
> >             I meant to say blocking clients based on their API version
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48
> > But If I understand what you are saying, since each client release can
> > support different versions for each of fetch, produce, offset commit etc..
> > and it's harder to block just based on single min.api.version setting
> > across different clients.
> > The idea I had in my mind was to do this via ApiVersionRequest, when a
> > client makes api request to broker in response we return min and max
> > version supported for each Api. When min.api.version enabled on broker, it
> > returns the maxVersion it supports for each of the requests in that release
> > as min versions to the clients.
> >
> > Example:
> > Kafka 1.1.1 broker and min.api.verson set to
> > https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79
> > (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for
> > example produce request
> >
> > https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112
> > Instead of returning all of the supported versions it will return
> > PRODUCE_REQUEST_V5 as the only supported version.
> >
> > Irrespective of the above approach I understand your point still stands
> > which is sarama might not choose to implement all the higher version
> > protocols for Kafka 1.1 release and they might introduce higher version of
> > produce request in a subsequent minor release and it will be harder for
> > users to figure out which release of sarama client they can use.
> >
> >
> > Ying, if you have a different apporach which might address this issue
> > please add.
> >
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote:
> > > Hi Harsha,
> > >
> > > There is no such thing as 1.1 protocol. I encourage you to describe an
> > > example config that achieves what you are suggesting here. It's pretty
> > > complicated because the versions are per API and each client evolves
> > > independently.
> > >
> > > Ismael
> > >
> > > On Sat, Apr 13, 2019 at 4:09 AM Harsha <ka...@harsha.io> wrote:
> > >
> > > > Hi,
> > > >
> > > > "Relying on min.version seems like a pretty clunky way to achieve the
> > above
> > > > > list. The challenge is that it's pretty difficult to do it in a way
> > that
> > > > > works for clients across languages. They each add support for new
> > > > protocol
> > > > > versions independently (it could even happen in a bug fix release).
> > So,
> > > > if
> > > > > you tried to block Sarama in #2, you may block Java clients too."
> > > >
> > > > That's the intended effect, right?  if you as the admin/operator
> > > > configures the broker to have min.api.version to be 1.1
> > > > it should block java , sarama clients etc.. which are below the 1.1
> > > > protocol.  As mentioned this is not just related to log.format upgrade
> > > > problem but in general a forcing cause to get the users to upgrade
> > their
> > > > client version in a multi-tenant environment.
> > > >
> > > > "> For #3, it seems simplest to have a config that requires clients to
> > > > support
> > > > > a given message format version (or higher). For #2, it seems like
> > you'd
> > > > > want clients to advertise their versions. That would be useful for
> > > > multiple
> > > > > reasons."
> > > > This kip offers the ability to block clients based on the protocol they
> > > > support. This should be independent of the message format upgrade. Not
> > all
> > > > of the features or bugs are dependent on a message format and having a
> > > > message format dependency to block clients means we have to upgrade to
> > > > message.format and we cannot just say we've 1.1 brokers with 0.8.2
> > message
> > > > format and now we want to block all 0.8.x clients.
> > > >
> > > > min.api.version helps at the cluster level to say that all users
> > required
> > > > to upgrade clients to the at minimum need to speak the min.api.version
> > and
> > > > not tie to message.format because not all cases one wants to upgrade
> > the
> > > > message format and block the old clients.
> > > >
> > > >
> > > > To Gwen's point, I think we should also return in the error message
> > that
> > > > the broker only supports min.api.version and above. So that users can
> > see a
> > > > clear message and upgrade to a newer version.
> > > >
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote:
> > > > > Hi Ying,
> > > > >
> > > > > The actual reasons are important so that people can evaluate the KIP
> > (and
> > > > > vote). :) Thanks for providing a few more:
> > > > >
> > > > > (1) force users to check pointing in Kafka instead of zookeeper
> > > > > (2) forbid an old go (sarama) client library which is known to have
> > some
> > > > > serious bugs
> > > > > (3) force kafka 1.x clients with the ability to roll back if there's
> > an
> > > > > issue (unlike a message format upgrade)
> > > > >
> > > > > Relying on min.version seems like a pretty clunky way to achieve the
> > > > above
> > > > > list. The challenge is that it's pretty difficult to do it in a way
> > that
> > > > > works for clients across languages. They each add support for new
> > > > protocol
> > > > > versions independently (it could even happen in a bug fix release).
> > So,
> > > > if
> > > > > you tried to block Sarama in #2, you may block Java clients too.
> > > > >
> > > > > For #3, it seems simplest to have a config that requires clients to
> > > > support
> > > > > a given message format version (or higher). For #2, it seems like
> > you'd
> > > > > want clients to advertise their versions. That would be useful for
> > > > multiple
> > > > > reasons.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi Ismael,
> > > > > >
> > > > > > Those are just examples. I think the administrators should be able
> > to
> > > > block
> > > > > > certain client libraries for whatever reason. Some other possible
> > > > reasons
> > > > > > include, force users to check pointing in Kafka instead of
> > zookeeper,
> > > > > > forbid an old go (sarama) client library which is known to have
> > some
> > > > > > serious bugs.
> > > > > >
> > > > > > message.downconversion.enable does not solve our problems. We are
> > now
> > > > > > planning to upgrade to message format V3, and force users to
> > upgrade to
> > > > > > Kafka 1.x clients. With the proposed min.api.version setting, in
> > case
> > > > of
> > > > > > there is anything wrong, we can roll back the setting. If we
> > upgrade
> > > > the
> > > > > > file format, there is no way to rollback (Kafka doesn't support
> > > > downgrading
> > > > > > message format).
> > > > > >
> > > > > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > > > >
> > > > > > > Hi Ying,
> > > > > > >
> > > > > > > It looks to me that all the examples given in the KIP can be
> > handled
> > > > with
> > > > > > > the existing "message.downconversion.enable" config and by
> > > > configuring
> > > > > > the
> > > > > > > message format to be the latest:
> > > > > > >
> > > > > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains
> > message
> > > > > > header
> > > > > > > > ( KAFKA-6739 - Down-conversion fails for records with headers
> > > > > > RESOLVED  )
> > > > > > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 (
> > > > KAFKA-3160 -
> > > > > > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED
> > )
> > > > > > > > 3. Performance penalty of converting message format from V3 to
> > V1
> > > > or V2
> > > > > > > > for the old consumers (KIP-31 - Move to relative offsets in
> > > > compressed
> > > > > > > > message sets)
> > > > > > >
> > > > > > >
> > > > > > > Am I missing something? Are there other examples that are not
> > > > related to
> > > > > > > message conversion?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng
> > <yi...@uber.com.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi here,
> > > > > > > >
> > > > > > > > Please vote for
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > > > > > > >
> > > > > > > > Thank you!
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Ying Zheng <yi...@uber.com.INVALID>.

I have updated the config description in the KIP, made the example more
clear

The proposed change allows setting different min versions for different
APIs, and the ApiVersionRequest change is already in the KIP.

On Fri, Apr 12, 2019 at 8:22 PM Harsha <ka...@harsha.io> wrote:

> Hi Ismael,
>             I meant to say blocking clients based on their API version
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48
> But If I understand what you are saying, since each client release can
> support different versions for each of fetch, produce, offset commit etc..
> and it's harder to block just based on single min.api.version setting
> across different clients.
> The idea I had in my mind was to do this via ApiVersionRequest, when a
> client makes api request to broker in response we return min and max
> version supported for each Api. When min.api.version enabled on broker, it
> returns the maxVersion it supports for each of the requests in that release
> as min versions to the clients.
>
> Example:
> Kafka 1.1.1 broker and min.api.verson set to
> https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79
> (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for
> example produce request
>
> https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112
> Instead of returning all of the supported versions it will return
> PRODUCE_REQUEST_V5 as the only supported version.
>
> Irrespective of the above approach I understand your point still stands
> which is sarama might not choose to implement all the higher version
> protocols for Kafka 1.1 release and they might introduce higher version of
> produce request in a subsequent minor release and it will be harder for
> users to figure out which release of sarama client they can use.
>
>
> Ying, if you have a different apporach which might address this issue
> please add.
>
>
> Thanks,
> Harsha
>
> On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote:
> > Hi Harsha,
> >
> > There is no such thing as 1.1 protocol. I encourage you to describe an
> > example config that achieves what you are suggesting here. It's pretty
> > complicated because the versions are per API and each client evolves
> > independently.
> >
> > Ismael
> >
> > On Sat, Apr 13, 2019 at 4:09 AM Harsha <ka...@harsha.io> wrote:
> >
> > > Hi,
> > >
> > > "Relying on min.version seems like a pretty clunky way to achieve the
> above
> > > > list. The challenge is that it's pretty difficult to do it in a way
> that
> > > > works for clients across languages. They each add support for new
> > > protocol
> > > > versions independently (it could even happen in a bug fix release).
> So,
> > > if
> > > > you tried to block Sarama in #2, you may block Java clients too."
> > >
> > > That's the intended effect, right?  if you as the admin/operator
> > > configures the broker to have min.api.version to be 1.1
> > > it should block java , sarama clients etc.. which are below the 1.1
> > > protocol.  As mentioned this is not just related to log.format upgrade
> > > problem but in general a forcing cause to get the users to upgrade
> their
> > > client version in a multi-tenant environment.
> > >
> > > "> For #3, it seems simplest to have a config that requires clients to
> > > support
> > > > a given message format version (or higher). For #2, it seems like
> you'd
> > > > want clients to advertise their versions. That would be useful for
> > > multiple
> > > > reasons."
> > > This kip offers the ability to block clients based on the protocol they
> > > support. This should be independent of the message format upgrade. Not
> all
> > > of the features or bugs are dependent on a message format and having a
> > > message format dependency to block clients means we have to upgrade to
> > > message.format and we cannot just say we've 1.1 brokers with 0.8.2
> message
> > > format and now we want to block all 0.8.x clients.
> > >
> > > min.api.version helps at the cluster level to say that all users
> required
> > > to upgrade clients to the at minimum need to speak the min.api.version
> and
> > > not tie to message.format because not all cases one wants to upgrade
> the
> > > message format and block the old clients.
> > >
> > >
> > > To Gwen's point, I think we should also return in the error message
> that
> > > the broker only supports min.api.version and above. So that users can
> see a
> > > clear message and upgrade to a newer version.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote:
> > > > Hi Ying,
> > > >
> > > > The actual reasons are important so that people can evaluate the KIP
> (and
> > > > vote). :) Thanks for providing a few more:
> > > >
> > > > (1) force users to check pointing in Kafka instead of zookeeper
> > > > (2) forbid an old go (sarama) client library which is known to have
> some
> > > > serious bugs
> > > > (3) force kafka 1.x clients with the ability to roll back if there's
> an
> > > > issue (unlike a message format upgrade)
> > > >
> > > > Relying on min.version seems like a pretty clunky way to achieve the
> > > above
> > > > list. The challenge is that it's pretty difficult to do it in a way
> that
> > > > works for clients across languages. They each add support for new
> > > protocol
> > > > versions independently (it could even happen in a bug fix release).
> So,
> > > if
> > > > you tried to block Sarama in #2, you may block Java clients too.
> > > >
> > > > For #3, it seems simplest to have a config that requires clients to
> > > support
> > > > a given message format version (or higher). For #2, it seems like
> you'd
> > > > want clients to advertise their versions. That would be useful for
> > > multiple
> > > > reasons.
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid>
> > > wrote:
> > > >
> > > > > Hi Ismael,
> > > > >
> > > > > Those are just examples. I think the administrators should be able
> to
> > > block
> > > > > certain client libraries for whatever reason. Some other possible
> > > reasons
> > > > > include, force users to check pointing in Kafka instead of
> zookeeper,
> > > > > forbid an old go (sarama) client library which is known to have
> some
> > > > > serious bugs.
> > > > >
> > > > > message.downconversion.enable does not solve our problems. We are
> now
> > > > > planning to upgrade to message format V3, and force users to
> upgrade to
> > > > > Kafka 1.x clients. With the proposed min.api.version setting, in
> case
> > > of
> > > > > there is anything wrong, we can roll back the setting. If we
> upgrade
> > > the
> > > > > file format, there is no way to rollback (Kafka doesn't support
> > > downgrading
> > > > > message format).
> > > > >
> > > > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <is...@juma.me.uk>
> wrote:
> > > > >
> > > > > > Hi Ying,
> > > > > >
> > > > > > It looks to me that all the examples given in the KIP can be
> handled
> > > with
> > > > > > the existing "message.downconversion.enable" config and by
> > > configuring
> > > > > the
> > > > > > message format to be the latest:
> > > > > >
> > > > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains
> message
> > > > > header
> > > > > > > ( KAFKA-6739 - Down-conversion fails for records with headers
> > > > > RESOLVED  )
> > > > > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 (
> > > KAFKA-3160 -
> > > > > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED
> )
> > > > > > > 3. Performance penalty of converting message format from V3 to
> V1
> > > or V2
> > > > > > > for the old consumers (KIP-31 - Move to relative offsets in
> > > compressed
> > > > > > > message sets)
> > > > > >
> > > > > >
> > > > > > Am I missing something? Are there other examples that are not
> > > related to
> > > > > > message conversion?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng
> <yi...@uber.com.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi here,
> > > > > > >
> > > > > > > Please vote for
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > > > > > >
> > > > > > > Thank you!
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Harsha <ka...@harsha.io>.

Hi Ismael,
            I meant to say blocking clients based on their API version https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48 
But If I understand what you are saying, since each client release can support different versions for each of fetch, produce, offset commit etc.. and it's harder to block just based on single min.api.version setting across different clients. 
The idea I had in my mind was to do this via ApiVersionRequest, when a client makes api request to broker in response we return min and max version supported for each Api. When min.api.version enabled on broker, it returns the maxVersion it supports for each of the requests in that release as min versions to the clients.

Example:
Kafka 1.1.1 broker and min.api.verson set to https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79 (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for example produce request 
https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112
Instead of returning all of the supported versions it will return PRODUCE_REQUEST_V5 as the only supported version.

Irrespective of the above approach I understand your point still stands which is sarama might not choose to implement all the higher version protocols for Kafka 1.1 release and they might introduce higher version of produce request in a subsequent minor release and it will be harder for users to figure out which release of sarama client they can use.


Ying, if you have a different apporach which might address this issue please add.


Thanks,
Harsha

On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote:
> Hi Harsha,
> 
> There is no such thing as 1.1 protocol. I encourage you to describe an
> example config that achieves what you are suggesting here. It's pretty
> complicated because the versions are per API and each client evolves
> independently.
> 
> Ismael
> 
> On Sat, Apr 13, 2019 at 4:09 AM Harsha <ka...@harsha.io> wrote:
> 
> > Hi,
> >
> > "Relying on min.version seems like a pretty clunky way to achieve the above
> > > list. The challenge is that it's pretty difficult to do it in a way that
> > > works for clients across languages. They each add support for new
> > protocol
> > > versions independently (it could even happen in a bug fix release). So,
> > if
> > > you tried to block Sarama in #2, you may block Java clients too."
> >
> > That's the intended effect, right?  if you as the admin/operator
> > configures the broker to have min.api.version to be 1.1
> > it should block java , sarama clients etc.. which are below the 1.1
> > protocol.  As mentioned this is not just related to log.format upgrade
> > problem but in general a forcing cause to get the users to upgrade their
> > client version in a multi-tenant environment.
> >
> > "> For #3, it seems simplest to have a config that requires clients to
> > support
> > > a given message format version (or higher). For #2, it seems like you'd
> > > want clients to advertise their versions. That would be useful for
> > multiple
> > > reasons."
> > This kip offers the ability to block clients based on the protocol they
> > support. This should be independent of the message format upgrade. Not all
> > of the features or bugs are dependent on a message format and having a
> > message format dependency to block clients means we have to upgrade to
> > message.format and we cannot just say we've 1.1 brokers with 0.8.2 message
> > format and now we want to block all 0.8.x clients.
> >
> > min.api.version helps at the cluster level to say that all users required
> > to upgrade clients to the at minimum need to speak the min.api.version and
> > not tie to message.format because not all cases one wants to upgrade the
> > message format and block the old clients.
> >
> >
> > To Gwen's point, I think we should also return in the error message that
> > the broker only supports min.api.version and above. So that users can see a
> > clear message and upgrade to a newer version.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote:
> > > Hi Ying,
> > >
> > > The actual reasons are important so that people can evaluate the KIP (and
> > > vote). :) Thanks for providing a few more:
> > >
> > > (1) force users to check pointing in Kafka instead of zookeeper
> > > (2) forbid an old go (sarama) client library which is known to have some
> > > serious bugs
> > > (3) force kafka 1.x clients with the ability to roll back if there's an
> > > issue (unlike a message format upgrade)
> > >
> > > Relying on min.version seems like a pretty clunky way to achieve the
> > above
> > > list. The challenge is that it's pretty difficult to do it in a way that
> > > works for clients across languages. They each add support for new
> > protocol
> > > versions independently (it could even happen in a bug fix release). So,
> > if
> > > you tried to block Sarama in #2, you may block Java clients too.
> > >
> > > For #3, it seems simplest to have a config that requires clients to
> > support
> > > a given message format version (or higher). For #2, it seems like you'd
> > > want clients to advertise their versions. That would be useful for
> > multiple
> > > reasons.
> > >
> > > Ismael
> > >
> > > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid>
> > wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > Those are just examples. I think the administrators should be able to
> > block
> > > > certain client libraries for whatever reason. Some other possible
> > reasons
> > > > include, force users to check pointing in Kafka instead of zookeeper,
> > > > forbid an old go (sarama) client library which is known to have some
> > > > serious bugs.
> > > >
> > > > message.downconversion.enable does not solve our problems. We are now
> > > > planning to upgrade to message format V3, and force users to upgrade to
> > > > Kafka 1.x clients. With the proposed min.api.version setting, in case
> > of
> > > > there is anything wrong, we can roll back the setting. If we upgrade
> > the
> > > > file format, there is no way to rollback (Kafka doesn't support
> > downgrading
> > > > message format).
> > > >
> > > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
> > > >
> > > > > Hi Ying,
> > > > >
> > > > > It looks to me that all the examples given in the KIP can be handled
> > with
> > > > > the existing "message.downconversion.enable" config and by
> > configuring
> > > > the
> > > > > message format to be the latest:
> > > > >
> > > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains message
> > > > header
> > > > > > ( KAFKA-6739 - Down-conversion fails for records with headers
> > > > RESOLVED  )
> > > > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 (
> > KAFKA-3160 -
> > > > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED  )
> > > > > > 3. Performance penalty of converting message format from V3 to V1
> > or V2
> > > > > > for the old consumers (KIP-31 - Move to relative offsets in
> > compressed
> > > > > > message sets)
> > > > >
> > > > >
> > > > > Am I missing something? Are there other examples that are not
> > related to
> > > > > message conversion?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng <yi...@uber.com.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi here,
> > > > > >
> > > > > > Please vote for
> > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > > > > >
> > > > > > Thank you!
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Ismael Juma <is...@juma.me.uk>.

Hi Harsha,

There is no such thing as 1.1 protocol. I encourage you to describe an
example config that achieves what you are suggesting here. It's pretty
complicated because the versions are per API and each client evolves
independently.

Ismael

On Sat, Apr 13, 2019 at 4:09 AM Harsha <ka...@harsha.io> wrote:

> Hi,
>
> "Relying on min.version seems like a pretty clunky way to achieve the above
> > list. The challenge is that it's pretty difficult to do it in a way that
> > works for clients across languages. They each add support for new
> protocol
> > versions independently (it could even happen in a bug fix release). So,
> if
> > you tried to block Sarama in #2, you may block Java clients too."
>
> That's the intended effect, right?  if you as the admin/operator
> configures the broker to have min.api.version to be 1.1
> it should block java , sarama clients etc.. which are below the 1.1
> protocol.  As mentioned this is not just related to log.format upgrade
> problem but in general a forcing cause to get the users to upgrade their
> client version in a multi-tenant environment.
>
> "> For #3, it seems simplest to have a config that requires clients to
> support
> > a given message format version (or higher). For #2, it seems like you'd
> > want clients to advertise their versions. That would be useful for
> multiple
> > reasons."
> This kip offers the ability to block clients based on the protocol they
> support. This should be independent of the message format upgrade. Not all
> of the features or bugs are dependent on a message format and having a
> message format dependency to block clients means we have to upgrade to
> message.format and we cannot just say we've 1.1 brokers with 0.8.2 message
> format and now we want to block all 0.8.x clients.
>
> min.api.version helps at the cluster level to say that all users required
> to upgrade clients to the at minimum need to speak the min.api.version and
> not tie to message.format because not all cases one wants to upgrade the
> message format and block the old clients.
>
>
> To Gwen's point, I think we should also return in the error message that
> the broker only supports min.api.version and above. So that users can see a
> clear message and upgrade to a newer version.
>
>
> Thanks,
> Harsha
>
>
> On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote:
> > Hi Ying,
> >
> > The actual reasons are important so that people can evaluate the KIP (and
> > vote). :) Thanks for providing a few more:
> >
> > (1) force users to check pointing in Kafka instead of zookeeper
> > (2) forbid an old go (sarama) client library which is known to have some
> > serious bugs
> > (3) force kafka 1.x clients with the ability to roll back if there's an
> > issue (unlike a message format upgrade)
> >
> > Relying on min.version seems like a pretty clunky way to achieve the
> above
> > list. The challenge is that it's pretty difficult to do it in a way that
> > works for clients across languages. They each add support for new
> protocol
> > versions independently (it could even happen in a bug fix release). So,
> if
> > you tried to block Sarama in #2, you may block Java clients too.
> >
> > For #3, it seems simplest to have a config that requires clients to
> support
> > a given message format version (or higher). For #2, it seems like you'd
> > want clients to advertise their versions. That would be useful for
> multiple
> > reasons.
> >
> > Ismael
> >
> > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid>
> wrote:
> >
> > > Hi Ismael,
> > >
> > > Those are just examples. I think the administrators should be able to
> block
> > > certain client libraries for whatever reason. Some other possible
> reasons
> > > include, force users to check pointing in Kafka instead of zookeeper,
> > > forbid an old go (sarama) client library which is known to have some
> > > serious bugs.
> > >
> > > message.downconversion.enable does not solve our problems. We are now
> > > planning to upgrade to message format V3, and force users to upgrade to
> > > Kafka 1.x clients. With the proposed min.api.version setting, in case
> of
> > > there is anything wrong, we can roll back the setting. If we upgrade
> the
> > > file format, there is no way to rollback (Kafka doesn't support
> downgrading
> > > message format).
> > >
> > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi Ying,
> > > >
> > > > It looks to me that all the examples given in the KIP can be handled
> with
> > > > the existing "message.downconversion.enable" config and by
> configuring
> > > the
> > > > message format to be the latest:
> > > >
> > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains message
> > > header
> > > > > ( KAFKA-6739 - Down-conversion fails for records with headers
> > > RESOLVED  )
> > > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 (
> KAFKA-3160 -
> > > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED  )
> > > > > 3. Performance penalty of converting message format from V3 to V1
> or V2
> > > > > for the old consumers (KIP-31 - Move to relative offsets in
> compressed
> > > > > message sets)
> > > >
> > > >
> > > > Am I missing something? Are there other examples that are not
> related to
> > > > message conversion?
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng <yi...@uber.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi here,
> > > > >
> > > > > Please vote for
> > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > > > >
> > > > > Thank you!
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Harsha <ka...@harsha.io>.

Hi,
      
"Relying on min.version seems like a pretty clunky way to achieve the above
> list. The challenge is that it's pretty difficult to do it in a way that
> works for clients across languages. They each add support for new protocol
> versions independently (it could even happen in a bug fix release). So, if
> you tried to block Sarama in #2, you may block Java clients too."

That's the intended effect, right?  if you as the admin/operator configures the broker to have min.api.version to be 1.1 
it should block java , sarama clients etc.. which are below the 1.1 protocol.  As mentioned this is not just related to log.format upgrade problem but in general a forcing cause to get the users to upgrade their client version in a multi-tenant environment.

"> For #3, it seems simplest to have a config that requires clients to support
> a given message format version (or higher). For #2, it seems like you'd
> want clients to advertise their versions. That would be useful for multiple
> reasons."
This kip offers the ability to block clients based on the protocol they support. This should be independent of the message format upgrade. Not all of the features or bugs are dependent on a message format and having a message format dependency to block clients means we have to upgrade to message.format and we cannot just say we've 1.1 brokers with 0.8.2 message format and now we want to block all 0.8.x clients.

min.api.version helps at the cluster level to say that all users required to upgrade clients to the at minimum need to speak the min.api.version and not tie to message.format because not all cases one wants to upgrade the message format and block the old clients.


To Gwen's point, I think we should also return in the error message that the broker only supports min.api.version and above. So that users can see a clear message and upgrade to a newer version.


Thanks,
Harsha


On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote:
> Hi Ying,
> 
> The actual reasons are important so that people can evaluate the KIP (and
> vote). :) Thanks for providing a few more:
> 
> (1) force users to check pointing in Kafka instead of zookeeper
> (2) forbid an old go (sarama) client library which is known to have some
> serious bugs
> (3) force kafka 1.x clients with the ability to roll back if there's an
> issue (unlike a message format upgrade)
> 
> Relying on min.version seems like a pretty clunky way to achieve the above
> list. The challenge is that it's pretty difficult to do it in a way that
> works for clients across languages. They each add support for new protocol
> versions independently (it could even happen in a bug fix release). So, if
> you tried to block Sarama in #2, you may block Java clients too.
> 
> For #3, it seems simplest to have a config that requires clients to support
> a given message format version (or higher). For #2, it seems like you'd
> want clients to advertise their versions. That would be useful for multiple
> reasons.
> 
> Ismael
> 
> On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid> wrote:
> 
> > Hi Ismael,
> >
> > Those are just examples. I think the administrators should be able to block
> > certain client libraries for whatever reason. Some other possible reasons
> > include, force users to check pointing in Kafka instead of zookeeper,
> > forbid an old go (sarama) client library which is known to have some
> > serious bugs.
> >
> > message.downconversion.enable does not solve our problems. We are now
> > planning to upgrade to message format V3, and force users to upgrade to
> > Kafka 1.x clients. With the proposed min.api.version setting, in case of
> > there is anything wrong, we can roll back the setting. If we upgrade the
> > file format, there is no way to rollback (Kafka doesn't support downgrading
> > message format).
> >
> > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Ying,
> > >
> > > It looks to me that all the examples given in the KIP can be handled with
> > > the existing "message.downconversion.enable" config and by configuring
> > the
> > > message format to be the latest:
> > >
> > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains message
> > header
> > > > ( KAFKA-6739 - Down-conversion fails for records with headers
> > RESOLVED  )
> > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 ( KAFKA-3160 -
> > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED  )
> > > > 3. Performance penalty of converting message format from V3 to V1 or V2
> > > > for the old consumers (KIP-31 - Move to relative offsets in compressed
> > > > message sets)
> > >
> > >
> > > Am I missing something? Are there other examples that are not related to
> > > message conversion?
> > >
> > > Ismael
> > >
> > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng <yi...@uber.com.invalid>
> > > wrote:
> > >
> > > > Hi here,
> > > >
> > > > Please vote for
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > > >
> > > > Thank you!
> > > >
> > >
> >
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Ismael Juma <is...@juma.me.uk>.

Hi Ying,

The actual reasons are important so that people can evaluate the KIP (and
vote). :) Thanks for providing a few more:

(1) force users to check pointing in Kafka instead of zookeeper
(2) forbid an old go (sarama) client library which is known to have some
serious bugs
(3) force kafka 1.x clients with the ability to roll back if there's an
issue (unlike a message format upgrade)

Relying on min.version seems like a pretty clunky way to achieve the above
list. The challenge is that it's pretty difficult to do it in a way that
works for clients across languages. They each add support for new protocol
versions independently (it could even happen in a bug fix release). So, if
you tried to block Sarama in #2, you may block Java clients too.

For #3, it seems simplest to have a config that requires clients to support
a given message format version (or higher). For #2, it seems like you'd
want clients to advertise their versions. That would be useful for multiple
reasons.

Ismael

On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid> wrote:

> Hi Ismael,
>
> Those are just examples. I think the administrators should be able to block
> certain client libraries for whatever reason. Some other possible reasons
> include, force users to check pointing in Kafka instead of zookeeper,
> forbid an old go (sarama) client library which is known to have some
> serious bugs.
>
> message.downconversion.enable does not solve our problems. We are now
> planning to upgrade to message format V3, and force users to upgrade to
> Kafka 1.x clients. With the proposed min.api.version setting, in case of
> there is anything wrong, we can roll back the setting. If we upgrade the
> file format, there is no way to rollback (Kafka doesn't support downgrading
> message format).
>
> On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Ying,
> >
> > It looks to me that all the examples given in the KIP can be handled with
> > the existing "message.downconversion.enable" config and by configuring
> the
> > message format to be the latest:
> >
> > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains message
> header
> > > ( KAFKA-6739 - Down-conversion fails for records with headers
> RESOLVED  )
> > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 ( KAFKA-3160 -
> > > Kafka LZ4 framing code miscalculates header checksum RESOLVED  )
> > > 3. Performance penalty of converting message format from V3 to V1 or V2
> > > for the old consumers (KIP-31 - Move to relative offsets in compressed
> > > message sets)
> >
> >
> > Am I missing something? Are there other examples that are not related to
> > message conversion?
> >
> > Ismael
> >
> > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng <yi...@uber.com.invalid>
> > wrote:
> >
> > > Hi here,
> > >
> > > Please vote for
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > >
> > > Thank you!
> > >
> >
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Ying Zheng <yi...@uber.com.INVALID>.

Hi Ismael,

Those are just examples. I think the administrators should be able to block
certain client libraries for whatever reason. Some other possible reasons
include, force users to check pointing in Kafka instead of zookeeper,
forbid an old go (sarama) client library which is known to have some
serious bugs.

message.downconversion.enable does not solve our problems. We are now
planning to upgrade to message format V3, and force users to upgrade to
Kafka 1.x clients. With the proposed min.api.version setting, in case of
there is anything wrong, we can roll back the setting. If we upgrade the
file format, there is no way to rollback (Kafka doesn't support downgrading
message format).

On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <is...@juma.me.uk> wrote:

> Hi Ying,
>
> It looks to me that all the examples given in the KIP can be handled with
> the existing "message.downconversion.enable" config and by configuring the
> message format to be the latest:
>
> 1. Kafka 8 / 9 / 10 consumer hangs when the message contains message header
> > ( KAFKA-6739 - Down-conversion fails for records with headers RESOLVED  )
> > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 ( KAFKA-3160 -
> > Kafka LZ4 framing code miscalculates header checksum RESOLVED  )
> > 3. Performance penalty of converting message format from V3 to V1 or V2
> > for the old consumers (KIP-31 - Move to relative offsets in compressed
> > message sets)
>
>
> Am I missing something? Are there other examples that are not related to
> message conversion?
>
> Ismael
>
> On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng <yi...@uber.com.invalid>
> wrote:
>
> > Hi here,
> >
> > Please vote for
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> >
> > Thank you!
> >
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Ismael Juma <is...@juma.me.uk>.

Hi Ying,

It looks to me that all the examples given in the KIP can be handled with
the existing "message.downconversion.enable" config and by configuring the
message format to be the latest:

1. Kafka 8 / 9 / 10 consumer hangs when the message contains message header
> ( KAFKA-6739 - Down-conversion fails for records with headers RESOLVED  )
> 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 ( KAFKA-3160 -
> Kafka LZ4 framing code miscalculates header checksum RESOLVED  )
> 3. Performance penalty of converting message format from V3 to V1 or V2
> for the old consumers (KIP-31 - Move to relative offsets in compressed
> message sets)


Am I missing something? Are there other examples that are not related to
message conversion?

Ismael

On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng <yi...@uber.com.invalid> wrote:

> Hi here,
>
> Please vote for
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
>
> Thank you!
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Ying Zheng <yi...@uber.com.INVALID>.

Hi Gwen,

Thank you very much for the feedback! I have updated the KIP.

"reject request" means return an UNSUPPORTED_VERSION (35) error. Client
libraries after 0.10.2 will show the message "The version of API is not
supported." Client libraries before 0.10.2 will treat this error as an
unknown server error.

On Thu, Apr 11, 2019 at 3:01 PM Gwen Shapira <gw...@confluent.io> wrote:

> In general, I support this proposal, but I'd like some more details on what
> "rejecting API requests" mean? Close the connections? Return some kind of
> error? Is there a way for the client to know what happened? Is there a way
> for the admin to know how many clients are rejected?
>
> As a nit, the "migration plan" part of the KIP still mentions the
> authorizer.
>
> Gwen
>
> On Thu, Apr 11, 2019 at 2:53 PM Ying Zheng <yi...@uber.com.invalid> wrote:
>
> > Hi here,
> >
> > Please vote for
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> >
> > Thank you!
> >
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>

Re: [VOTE] KIP-433: Block old clients on brokers

Posted by Gwen Shapira <gw...@confluent.io>.

In general, I support this proposal, but I'd like some more details on what
"rejecting API requests" mean? Close the connections? Return some kind of
error? Is there a way for the client to know what happened? Is there a way
for the admin to know how many clients are rejected?

As a nit, the "migration plan" part of the KIP still mentions the
authorizer.

Gwen

On Thu, Apr 11, 2019 at 2:53 PM Ying Zheng <yi...@uber.com.invalid> wrote:

> Hi here,
>
> Please vote for
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
>
> Thank you!
>

-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>