You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Colin McCabe <cm...@apache.org> on 2016/11/29 19:04:33 UTC
[DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Hi all,
I've been thinking about a KIP to improve the Kafka client's
compatibility policy. If you're interested, please check out:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy
cheers,
Colin
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Colin McCabe <cm...@apache.org>.
On Fri, Dec 2, 2016, at 13:41, Ashish Singh wrote:
> Colin,
>
> I just rebased KAFKA-3600's PR on trunk.
>
Thanks, Ashish!
> KAFKA-4457 is a good idea, however it is probably doing some redundant
> work. AdminClient has a network client that will already have api
> versions
> info (after KAFKA-3600 goes in), so I do not think we need to send out
> another ApiVersions request over wire.
>
That's a good point. As long as we can talk to all brokers. Let's
continue the discussion on Jira.
I'll start a vote thread for KIP-97.
Best,
Colin
> On Wed, Nov 30, 2016 at 5:33 PM, Colin McCabe <cm...@apache.org> wrote:
>
> > Hi all,
> >
> > I updated the KIP to include a command to print out the version
> > information of brokers (KAFKA-4457). This will be a useful command for
> > administrators.
> >
> > best,
> > Colin
> >
> >
> > On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> > > Thanks, Ashish. I think the idea of having the client make an
> > > ApiVersionRequest call when it starts up is a good one. This idea is
> > > described in both KIP-97, and the KAFKA-3600 patches. I also think we
> > > ought to maintain per-node version information. It would be good to get
> > > that in so that we can use it as a building block for the stuff
> > > described in this KIP. I'd be happy to review it.
> > >
> > > Colin
> > >
> > >
> > > On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > > > Hello Ismael,
> > > >
> > > > It is good to know that you are willing to review KAFKA-3600 again. As
> > > > before, we at Cloudera are highly in support of client compatibility,
> > and
> > > > KAFKA-3600 has always been a building block for that. Now that client
> > > > compatibility is at forefront again, thanks to Colin, I will be happy
> > to
> > > > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > > > KAFKA-3600's
> > > > approach on maintaining api version info for each connected broker is
> > > > highly welcome.
> > > >
> > > > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Thanks Colin, I think this is a good improvement.
> > > > >
> > > > > Ashish, some of the concerns with regards to KAFKA-3600 were related
> > to the
> > > > > cost versus benefit. Once one adds client compatibility, the benefit
> > is
> > > > > much higher. I would be happy to review and merge KAFKA-3600 if we
> > think it
> > > > > serves as a good first step towards client compatibility (if the
> > vote for
> > > > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and
> > see if
> > > > > you can build on that? Ashish, it may be worth merging trunk into
> > your
> > > > > branch and fixing the conflicts.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com>
> > wrote:
> > > > >
> > > > > > Hello Colin,
> > > > > >
> > > > > > In the KIP you mentioned that currently the client uses supported
> > api
> > > > > > versions information to check if the server supports its desired
> > > > > versions.
> > > > > > Not sure, if that is true. I had put together a PR for KAFKA-3600,
> > to do
> > > > > > that, but it never went in. Also, I could not find how you plan to
> > > > > perform
> > > > > > version check on client side. In KAFKA-3600, I am maintaining api
> > version
> > > > > > for each live connection, and that made a few folks think it is
> > too big
> > > > > of
> > > > > > a change.
> > > > > >
> > > > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cmccabe@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Sorry, that link should be:
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > > > compatibility policy. If you're interested, please check out:
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > > > >
> > > > > > > > cheers,
> > > > > > > > Colin
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Regards,
> > > > > > Ashish
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> >
>
>
>
> --
>
> Regards,
> Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Ashish Singh <as...@cloudera.com>.
Colin,
I just rebased KAFKA-3600's PR on trunk.
KAFKA-4457 is a good idea, however it is probably doing some redundant
work. AdminClient has a network client that will already have api versions
info (after KAFKA-3600 goes in), so I do not think we need to send out
another ApiVersions request over wire.
On Wed, Nov 30, 2016 at 5:33 PM, Colin McCabe <cm...@apache.org> wrote:
> Hi all,
>
> I updated the KIP to include a command to print out the version
> information of brokers (KAFKA-4457). This will be a useful command for
> administrators.
>
> best,
> Colin
>
>
> On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> > Thanks, Ashish. I think the idea of having the client make an
> > ApiVersionRequest call when it starts up is a good one. This idea is
> > described in both KIP-97, and the KAFKA-3600 patches. I also think we
> > ought to maintain per-node version information. It would be good to get
> > that in so that we can use it as a building block for the stuff
> > described in this KIP. I'd be happy to review it.
> >
> > Colin
> >
> >
> > On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > > Hello Ismael,
> > >
> > > It is good to know that you are willing to review KAFKA-3600 again. As
> > > before, we at Cloudera are highly in support of client compatibility,
> and
> > > KAFKA-3600 has always been a building block for that. Now that client
> > > compatibility is at forefront again, thanks to Colin, I will be happy
> to
> > > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > > KAFKA-3600's
> > > approach on maintaining api version info for each connected broker is
> > > highly welcome.
> > >
> > > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Thanks Colin, I think this is a good improvement.
> > > >
> > > > Ashish, some of the concerns with regards to KAFKA-3600 were related
> to the
> > > > cost versus benefit. Once one adds client compatibility, the benefit
> is
> > > > much higher. I would be happy to review and merge KAFKA-3600 if we
> think it
> > > > serves as a good first step towards client compatibility (if the
> vote for
> > > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and
> see if
> > > > you can build on that? Ashish, it may be worth merging trunk into
> your
> > > > branch and fixing the conflicts.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com>
> wrote:
> > > >
> > > > > Hello Colin,
> > > > >
> > > > > In the KIP you mentioned that currently the client uses supported
> api
> > > > > versions information to check if the server supports its desired
> > > > versions.
> > > > > Not sure, if that is true. I had put together a PR for KAFKA-3600,
> to do
> > > > > that, but it never went in. Also, I could not find how you plan to
> > > > perform
> > > > > version check on client side. In KAFKA-3600, I am maintaining api
> version
> > > > > for each live connection, and that made a few folks think it is
> too big
> > > > of
> > > > > a change.
> > > > >
> > > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cmccabe@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > Sorry, that link should be:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > > compatibility policy. If you're interested, please check out:
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > > >
> > > > > > > cheers,
> > > > > > > Colin
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Regards,
> > > > > Ashish
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
>
--
Regards,
Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Colin McCabe <cm...@apache.org>.
Hi all,
I updated the KIP to include a command to print out the version
information of brokers (KAFKA-4457). This will be a useful command for
administrators.
best,
Colin
On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> Thanks, Ashish. I think the idea of having the client make an
> ApiVersionRequest call when it starts up is a good one. This idea is
> described in both KIP-97, and the KAFKA-3600 patches. I also think we
> ought to maintain per-node version information. It would be good to get
> that in so that we can use it as a building block for the stuff
> described in this KIP. I'd be happy to review it.
>
> Colin
>
>
> On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > Hello Ismael,
> >
> > It is good to know that you are willing to review KAFKA-3600 again. As
> > before, we at Cloudera are highly in support of client compatibility, and
> > KAFKA-3600 has always been a building block for that. Now that client
> > compatibility is at forefront again, thanks to Colin, I will be happy to
> > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > KAFKA-3600's
> > approach on maintaining api version info for each connected broker is
> > highly welcome.
> >
> > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Thanks Colin, I think this is a good improvement.
> > >
> > > Ashish, some of the concerns with regards to KAFKA-3600 were related to the
> > > cost versus benefit. Once one adds client compatibility, the benefit is
> > > much higher. I would be happy to review and merge KAFKA-3600 if we think it
> > > serves as a good first step towards client compatibility (if the vote for
> > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
> > > you can build on that? Ashish, it may be worth merging trunk into your
> > > branch and fixing the conflicts.
> > >
> > > Ismael
> > >
> > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com> wrote:
> > >
> > > > Hello Colin,
> > > >
> > > > In the KIP you mentioned that currently the client uses supported api
> > > > versions information to check if the server supports its desired
> > > versions.
> > > > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > > > that, but it never went in. Also, I could not find how you plan to
> > > perform
> > > > version check on client side. In KAFKA-3600, I am maintaining api version
> > > > for each live connection, and that made a few folks think it is too big
> > > of
> > > > a change.
> > > >
> > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cm...@apache.org>
> > > wrote:
> > > >
> > > > > Sorry, that link should be:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > compatibility policy. If you're interested, please check out:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > >
> > > > > > cheers,
> > > > > > Colin
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Colin McCabe <cm...@apache.org>.
Thanks, Ashish. I think the idea of having the client make an
ApiVersionRequest call when it starts up is a good one. This idea is
described in both KIP-97, and the KAFKA-3600 patches. I also think we
ought to maintain per-node version information. It would be good to get
that in so that we can use it as a building block for the stuff
described in this KIP. I'd be happy to review it.
Colin
On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> Hello Ismael,
>
> It is good to know that you are willing to review KAFKA-3600 again. As
> before, we at Cloudera are highly in support of client compatibility, and
> KAFKA-3600 has always been a building block for that. Now that client
> compatibility is at forefront again, thanks to Colin, I will be happy to
> rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> KAFKA-3600's
> approach on maintaining api version info for each connected broker is
> highly welcome.
>
> On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Thanks Colin, I think this is a good improvement.
> >
> > Ashish, some of the concerns with regards to KAFKA-3600 were related to the
> > cost versus benefit. Once one adds client compatibility, the benefit is
> > much higher. I would be happy to review and merge KAFKA-3600 if we think it
> > serves as a good first step towards client compatibility (if the vote for
> > this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
> > you can build on that? Ashish, it may be worth merging trunk into your
> > branch and fixing the conflicts.
> >
> > Ismael
> >
> > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com> wrote:
> >
> > > Hello Colin,
> > >
> > > In the KIP you mentioned that currently the client uses supported api
> > > versions information to check if the server supports its desired
> > versions.
> > > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > > that, but it never went in. Also, I could not find how you plan to
> > perform
> > > version check on client side. In KAFKA-3600, I am maintaining api version
> > > for each live connection, and that made a few folks think it is too big
> > of
> > > a change.
> > >
> > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cm...@apache.org>
> > wrote:
> > >
> > > > Sorry, that link should be:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > >
> > > >
> > > >
> > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > Hi all,
> > > > >
> > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > compatibility policy. If you're interested, please check out:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > >
> > > > > cheers,
> > > > > Colin
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Ashish Singh <as...@cloudera.com>.
Hello Ismael,
It is good to know that you are willing to review KAFKA-3600 again. As
before, we at Cloudera are highly in support of client compatibility, and
KAFKA-3600 has always been a building block for that. Now that client
compatibility is at forefront again, thanks to Colin, I will be happy to
rebase KAFKA-3600 on trunk. However, any comments/ feedback on KAFKA-3600's
approach on maintaining api version info for each connected broker is
highly welcome.
On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Thanks Colin, I think this is a good improvement.
>
> Ashish, some of the concerns with regards to KAFKA-3600 were related to the
> cost versus benefit. Once one adds client compatibility, the benefit is
> much higher. I would be happy to review and merge KAFKA-3600 if we think it
> serves as a good first step towards client compatibility (if the vote for
> this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
> you can build on that? Ashish, it may be worth merging trunk into your
> branch and fixing the conflicts.
>
> Ismael
>
> On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com> wrote:
>
> > Hello Colin,
> >
> > In the KIP you mentioned that currently the client uses supported api
> > versions information to check if the server supports its desired
> versions.
> > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > that, but it never went in. Also, I could not find how you plan to
> perform
> > version check on client side. In KAFKA-3600, I am maintaining api version
> > for each live connection, and that made a few folks think it is too big
> of
> > a change.
> >
> > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cm...@apache.org>
> wrote:
> >
> > > Sorry, that link should be:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > >
> > >
> > >
> > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > Hi all,
> > > >
> > > > I've been thinking about a KIP to improve the Kafka client's
> > > > compatibility policy. If you're interested, please check out:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > >
> > > > cheers,
> > > > Colin
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>
--
Regards,
Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks Colin, I think this is a good improvement.
Ashish, some of the concerns with regards to KAFKA-3600 were related to the
cost versus benefit. Once one adds client compatibility, the benefit is
much higher. I would be happy to review and merge KAFKA-3600 if we think it
serves as a good first step towards client compatibility (if the vote for
this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
you can build on that? Ashish, it may be worth merging trunk into your
branch and fixing the conflicts.
Ismael
On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com> wrote:
> Hello Colin,
>
> In the KIP you mentioned that currently the client uses supported api
> versions information to check if the server supports its desired versions.
> Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> that, but it never went in. Also, I could not find how you plan to perform
> version check on client side. In KAFKA-3600, I am maintaining api version
> for each live connection, and that made a few folks think it is too big of
> a change.
>
> On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cm...@apache.org> wrote:
>
> > Sorry, that link should be:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> >
> >
> >
> > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > Hi all,
> > >
> > > I've been thinking about a KIP to improve the Kafka client's
> > > compatibility policy. If you're interested, please check out:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Compatibility+Policy
> > >
> > > cheers,
> > > Colin
> >
>
>
>
> --
>
> Regards,
> Ashish
>
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Ashish Singh <as...@cloudera.com>.
On Tue, Nov 29, 2016 at 4:11 PM, Colin McCabe <co...@cmccabe.xyz> wrote:
> On Tue, Nov 29, 2016, at 11:39, Ashish Singh wrote:
> > Hello Colin,
> >
> > In the KIP you mentioned that currently the client uses supported api
> > versions information to check if the server supports its desired
> > versions.
> > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > that, but it never went in.
>
> I was taking a look at the RPC implementation today myself and I think
> you're right. I don't think the client actually invokes
> ApiVersionRequest-- or if it does, I can't find the invocation site.
> Still, the API versions are sent over the wire, and the server does use
> them to support older clients. I suppose this means that currently,
> clients which are too new fail at the time they attempt to send an RPC
> which is not in the server's supported version range for that RPC. I
> will edit the KIP to reflect this-- thanks.
>
> > Also, I could not find how you plan to perform
> > version check on client side. In KAFKA-3600, I am maintaining api version
> > for each live connection, and that made a few folks think it is too big
> > of a change.
>
> The version check would be performed by calling ApiVersionRequest, just
> like in your patch.
>
> I think that people are coming around to the idea that we need both
> forward and backwards compatibility for the client. As the project
> grows, and there are bigger and more deployments, it is probably worth
> spending a little more time on compatibility...
>
> P.S. If you have any spare cycles, it would be great to collaborate on
> this!
>
Kafka-3600 was part of voted-in KIP-35, and a pre-requisite for client
compat. If people are coming around to the idea, maybe we can revisit
KAFKA-3600 and try to find why is it not meaningful to have it in? If we
get KAFKA-3600 in, follow up changes to enable feature selection in clients
should be pretty straight forward.
>
> cheers,
> Colin
>
> >
> > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cm...@apache.org>
> > wrote:
> >
> > > Sorry, that link should be:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > >
> > >
> > >
> > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > Hi all,
> > > >
> > > > I've been thinking about a KIP to improve the Kafka client's
> > > > compatibility policy. If you're interested, please check out:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > >
> > > > cheers,
> > > > Colin
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
>
--
Regards,
Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Colin McCabe <co...@cmccabe.xyz>.
On Tue, Nov 29, 2016, at 11:39, Ashish Singh wrote:
> Hello Colin,
>
> In the KIP you mentioned that currently the client uses supported api
> versions information to check if the server supports its desired
> versions.
> Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> that, but it never went in.
I was taking a look at the RPC implementation today myself and I think
you're right. I don't think the client actually invokes
ApiVersionRequest-- or if it does, I can't find the invocation site.
Still, the API versions are sent over the wire, and the server does use
them to support older clients. I suppose this means that currently,
clients which are too new fail at the time they attempt to send an RPC
which is not in the server's supported version range for that RPC. I
will edit the KIP to reflect this-- thanks.
> Also, I could not find how you plan to perform
> version check on client side. In KAFKA-3600, I am maintaining api version
> for each live connection, and that made a few folks think it is too big
> of a change.
The version check would be performed by calling ApiVersionRequest, just
like in your patch.
I think that people are coming around to the idea that we need both
forward and backwards compatibility for the client. As the project
grows, and there are bigger and more deployments, it is probably worth
spending a little more time on compatibility...
P.S. If you have any spare cycles, it would be great to collaborate on
this!
cheers,
Colin
>
> On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cm...@apache.org>
> wrote:
>
> > Sorry, that link should be:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> >
> >
> >
> > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > Hi all,
> > >
> > > I've been thinking about a KIP to improve the Kafka client's
> > > compatibility policy. If you're interested, please check out:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Compatibility+Policy
> > >
> > > cheers,
> > > Colin
> >
>
>
>
> --
>
> Regards,
> Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Ashish Singh <as...@cloudera.com>.
Hello Colin,
In the KIP you mentioned that currently the client uses supported api
versions information to check if the server supports its desired versions.
Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
that, but it never went in. Also, I could not find how you plan to perform
version check on client side. In KAFKA-3600, I am maintaining api version
for each live connection, and that made a few folks think it is too big of
a change.
On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cm...@apache.org> wrote:
> Sorry, that link should be:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
>
>
>
> On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > Hi all,
> >
> > I've been thinking about a KIP to improve the Kafka client's
> > compatibility policy. If you're interested, please check out:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 97%3A+Improved+Kafka+Compatibility+Policy
> >
> > cheers,
> > Colin
>
--
Regards,
Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Colin McCabe <cm...@apache.org>.
Sorry, that link should be:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> Hi all,
>
> I've been thinking about a KIP to improve the Kafka client's
> compatibility policy. If you're interested, please check out:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy
>
> cheers,
> Colin