You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Colin McCabe <cm...@apache.org> on 2016/12/01 01:33:28 UTC
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Hi all,
I updated the KIP to include a command to print out the version
information of brokers (KAFKA-4457). This will be a useful command for
administrators.
best,
Colin
On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> Thanks, Ashish. I think the idea of having the client make an
> ApiVersionRequest call when it starts up is a good one. This idea is
> described in both KIP-97, and the KAFKA-3600 patches. I also think we
> ought to maintain per-node version information. It would be good to get
> that in so that we can use it as a building block for the stuff
> described in this KIP. I'd be happy to review it.
>
> Colin
>
>
> On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > Hello Ismael,
> >
> > It is good to know that you are willing to review KAFKA-3600 again. As
> > before, we at Cloudera are highly in support of client compatibility, and
> > KAFKA-3600 has always been a building block for that. Now that client
> > compatibility is at forefront again, thanks to Colin, I will be happy to
> > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > KAFKA-3600's
> > approach on maintaining api version info for each connected broker is
> > highly welcome.
> >
> > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Thanks Colin, I think this is a good improvement.
> > >
> > > Ashish, some of the concerns with regards to KAFKA-3600 were related to the
> > > cost versus benefit. Once one adds client compatibility, the benefit is
> > > much higher. I would be happy to review and merge KAFKA-3600 if we think it
> > > serves as a good first step towards client compatibility (if the vote for
> > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
> > > you can build on that? Ashish, it may be worth merging trunk into your
> > > branch and fixing the conflicts.
> > >
> > > Ismael
> > >
> > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com> wrote:
> > >
> > > > Hello Colin,
> > > >
> > > > In the KIP you mentioned that currently the client uses supported api
> > > > versions information to check if the server supports its desired
> > > versions.
> > > > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > > > that, but it never went in. Also, I could not find how you plan to
> > > perform
> > > > version check on client side. In KAFKA-3600, I am maintaining api version
> > > > for each live connection, and that made a few folks think it is too big
> > > of
> > > > a change.
> > > >
> > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cm...@apache.org>
> > > wrote:
> > > >
> > > > > Sorry, that link should be:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > compatibility policy. If you're interested, please check out:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > >
> > > > > > cheers,
> > > > > > Colin
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Colin McCabe <cm...@apache.org>.
On Fri, Dec 2, 2016, at 13:41, Ashish Singh wrote:
> Colin,
>
> I just rebased KAFKA-3600's PR on trunk.
>
Thanks, Ashish!
> KAFKA-4457 is a good idea, however it is probably doing some redundant
> work. AdminClient has a network client that will already have api
> versions
> info (after KAFKA-3600 goes in), so I do not think we need to send out
> another ApiVersions request over wire.
>
That's a good point. As long as we can talk to all brokers. Let's
continue the discussion on Jira.
I'll start a vote thread for KIP-97.
Best,
Colin
> On Wed, Nov 30, 2016 at 5:33 PM, Colin McCabe <cm...@apache.org> wrote:
>
> > Hi all,
> >
> > I updated the KIP to include a command to print out the version
> > information of brokers (KAFKA-4457). This will be a useful command for
> > administrators.
> >
> > best,
> > Colin
> >
> >
> > On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> > > Thanks, Ashish. I think the idea of having the client make an
> > > ApiVersionRequest call when it starts up is a good one. This idea is
> > > described in both KIP-97, and the KAFKA-3600 patches. I also think we
> > > ought to maintain per-node version information. It would be good to get
> > > that in so that we can use it as a building block for the stuff
> > > described in this KIP. I'd be happy to review it.
> > >
> > > Colin
> > >
> > >
> > > On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > > > Hello Ismael,
> > > >
> > > > It is good to know that you are willing to review KAFKA-3600 again. As
> > > > before, we at Cloudera are highly in support of client compatibility,
> > and
> > > > KAFKA-3600 has always been a building block for that. Now that client
> > > > compatibility is at forefront again, thanks to Colin, I will be happy
> > to
> > > > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > > > KAFKA-3600's
> > > > approach on maintaining api version info for each connected broker is
> > > > highly welcome.
> > > >
> > > > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Thanks Colin, I think this is a good improvement.
> > > > >
> > > > > Ashish, some of the concerns with regards to KAFKA-3600 were related
> > to the
> > > > > cost versus benefit. Once one adds client compatibility, the benefit
> > is
> > > > > much higher. I would be happy to review and merge KAFKA-3600 if we
> > think it
> > > > > serves as a good first step towards client compatibility (if the
> > vote for
> > > > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and
> > see if
> > > > > you can build on that? Ashish, it may be worth merging trunk into
> > your
> > > > > branch and fixing the conflicts.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com>
> > wrote:
> > > > >
> > > > > > Hello Colin,
> > > > > >
> > > > > > In the KIP you mentioned that currently the client uses supported
> > api
> > > > > > versions information to check if the server supports its desired
> > > > > versions.
> > > > > > Not sure, if that is true. I had put together a PR for KAFKA-3600,
> > to do
> > > > > > that, but it never went in. Also, I could not find how you plan to
> > > > > perform
> > > > > > version check on client side. In KAFKA-3600, I am maintaining api
> > version
> > > > > > for each live connection, and that made a few folks think it is
> > too big
> > > > > of
> > > > > > a change.
> > > > > >
> > > > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cmccabe@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Sorry, that link should be:
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > > > compatibility policy. If you're interested, please check out:
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > > > >
> > > > > > > > cheers,
> > > > > > > > Colin
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Regards,
> > > > > > Ashish
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> >
>
>
>
> --
>
> Regards,
> Ashish
Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy
Posted by Ashish Singh <as...@cloudera.com>.
Colin,
I just rebased KAFKA-3600's PR on trunk.
KAFKA-4457 is a good idea, however it is probably doing some redundant
work. AdminClient has a network client that will already have api versions
info (after KAFKA-3600 goes in), so I do not think we need to send out
another ApiVersions request over wire.
On Wed, Nov 30, 2016 at 5:33 PM, Colin McCabe <cm...@apache.org> wrote:
> Hi all,
>
> I updated the KIP to include a command to print out the version
> information of brokers (KAFKA-4457). This will be a useful command for
> administrators.
>
> best,
> Colin
>
>
> On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> > Thanks, Ashish. I think the idea of having the client make an
> > ApiVersionRequest call when it starts up is a good one. This idea is
> > described in both KIP-97, and the KAFKA-3600 patches. I also think we
> > ought to maintain per-node version information. It would be good to get
> > that in so that we can use it as a building block for the stuff
> > described in this KIP. I'd be happy to review it.
> >
> > Colin
> >
> >
> > On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > > Hello Ismael,
> > >
> > > It is good to know that you are willing to review KAFKA-3600 again. As
> > > before, we at Cloudera are highly in support of client compatibility,
> and
> > > KAFKA-3600 has always been a building block for that. Now that client
> > > compatibility is at forefront again, thanks to Colin, I will be happy
> to
> > > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > > KAFKA-3600's
> > > approach on maintaining api version info for each connected broker is
> > > highly welcome.
> > >
> > > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Thanks Colin, I think this is a good improvement.
> > > >
> > > > Ashish, some of the concerns with regards to KAFKA-3600 were related
> to the
> > > > cost versus benefit. Once one adds client compatibility, the benefit
> is
> > > > much higher. I would be happy to review and merge KAFKA-3600 if we
> think it
> > > > serves as a good first step towards client compatibility (if the
> vote for
> > > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and
> see if
> > > > you can build on that? Ashish, it may be worth merging trunk into
> your
> > > > branch and fixing the conflicts.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh <as...@cloudera.com>
> wrote:
> > > >
> > > > > Hello Colin,
> > > > >
> > > > > In the KIP you mentioned that currently the client uses supported
> api
> > > > > versions information to check if the server supports its desired
> > > > versions.
> > > > > Not sure, if that is true. I had put together a PR for KAFKA-3600,
> to do
> > > > > that, but it never went in. Also, I could not find how you plan to
> > > > perform
> > > > > version check on client side. In KAFKA-3600, I am maintaining api
> version
> > > > > for each live connection, and that made a few folks think it is
> too big
> > > > of
> > > > > a change.
> > > > >
> > > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe <cmccabe@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > Sorry, that link should be:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > > compatibility policy. If you're interested, please check out:
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > > >
> > > > > > > cheers,
> > > > > > > Colin
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Regards,
> > > > > Ashish
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
>
--
Regards,
Ashish