You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by ch...@post.ch on 2020/06/12 13:46:30 UTC
RE: kafka-clients version
Hi
Please see the discussion below.
Would it make sense to always pack the latest minor version of kafka-clients-jar in nifi-kafka-bundle?
The processors could be named “2.x” (or simply “2” really) and the specific minor version can change in NiFi releases.
Regards
Chris
From: Bryan Bende <bb...@gmail.com>
Sent: Friday, 12. June 2020 15:33
To: users@nifi.apache.org
Subject: Re: kafka-clients version
Gotcha. Yes, we can obviously change the version of the client lib, but it's misleading because they were named "2_0" because they use the "2_0" client, and now that won't be true anymore, but if others agree with this then maybe it's fine.
Maybe more of a question for the dev list.
On Fri, Jun 12, 2020 at 9:28 AM <ch...@post.ch>> wrote:
Yes: https://issues.apache.org/jira/browse/KAFKA-9815
> we'd have to be sure the latest 2.x client also works well with all previous 2.x versions back to 2.0
I think kafka-clients should make sure of that.
The processors could still be named 2_0, it’s only the client jar that should be the latest 2.x version.
Specifically https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/pom.xml#L31
Regards
Chris
From: Bryan Bende <bb...@gmail.com>>
Sent: Friday, 12. June 2020 15:18
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: kafka-clients version
Usually we make a new version of the processors when something changes that causes a previous version of the client to not work well with the newer broker (i.e. 0.9 client with 0.10 broker, in the older versions).
The latest processors are named with "2_0" so not sure if it makes sense to upgrade the client lib and then the name is no longer accurate, plus we'd have to be sure the latest 2.x client also works well with all previous 2.x versions back to 2.0.
The other option is to make a new copy of the processor like "2_5" or whatever the latest version is, but I think we only want to do this if it's really necessary. AFAIK the 2.0 client works with the latest 2.x brokers.
Is there any specific issue you are looking to solve by using the latest client?
On Fri, Jun 12, 2020 at 8:20 AM <ch...@post.ch>> wrote:
Hi
Any reason why nifi-kafka-bundle is still bundled with kafka-clients-2.0.0.jar and not a newer version?
Regards
Chris
Re: kafka-clients version
Posted by Bryan Bende <bb...@gmail.com>.
There was a discussion on users list, the relevant issue is this -
https://issues.apache.org/jira/browse/KAFKA-9815
On Fri, Jun 12, 2020 at 9:53 AM Jon Logan <jm...@buffalo.edu> wrote:
> Is there a reason why this question is being posed? If the version used is
> compatible with the latest server version, and there's no missing
> functionality or bugs being reported, I would strongly discourage updating,
> as you're likely mandating server upgrades for at least some users. Even if
> you make new processors, just creating new processors for no benefit is not
> beneficial.
>
> On Fri, Jun 12, 2020 at 9:50 AM Joe Witt <jo...@gmail.com> wrote:
>
> > Chris
> >
> > In my experience with the clients the answer is a definite no. An older
> > client will generally work (with important caveats at times) with a newer
> > broker. A newer client will generally not work with an older broker. At
> > the very least both major and minor version of the client matters so
> users
> > can appropriately map to their broker.
> >
> > We just need to put out a 2.5 one now
> >
> > thanks
> >
> > On Fri, Jun 12, 2020 at 6:46 AM <ch...@post.ch> wrote:
> >
> > > Hi
> > >
> > > Please see the discussion below.
> > > Would it make sense to always pack the latest minor version of
> > > kafka-clients-jar in nifi-kafka-bundle?
> > >
> > > The processors could be named “2.x” (or simply “2” really) and the
> > > specific minor version can change in NiFi releases.
> > >
> > > Regards
> > > Chris
> > >
> > > From: Bryan Bende <bb...@gmail.com>
> > > Sent: Friday, 12. June 2020 15:33
> > > To: users@nifi.apache.org
> > > Subject: Re: kafka-clients version
> > >
> > > Gotcha. Yes, we can obviously change the version of the client lib, but
> > > it's misleading because they were named "2_0" because they use the
> "2_0"
> > > client, and now that won't be true anymore, but if others agree with
> this
> > > then maybe it's fine.
> > >
> > > Maybe more of a question for the dev list.
> > >
> > > On Fri, Jun 12, 2020 at 9:28 AM <christophe.monnet@post.ch<mailto:
> > > christophe.monnet@post.ch>> wrote:
> > > Yes: https://issues.apache.org/jira/browse/KAFKA-9815
> > >
> > > > we'd have to be sure the latest 2.x client also works well with all
> > > previous 2.x versions back to 2.0
> > >
> > > I think kafka-clients should make sure of that.
> > >
> > > The processors could still be named 2_0, it’s only the client jar that
> > > should be the latest 2.x version.
> > > Specifically
> > >
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/pom.xml#L31
> > >
> > > Regards
> > > Chris
> > >
> > > From: Bryan Bende <bb...@gmail.com>>
> > > Sent: Friday, 12. June 2020 15:18
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> > > Subject: Re: kafka-clients version
> > >
> > > Usually we make a new version of the processors when something changes
> > > that causes a previous version of the client to not work well with the
> > > newer broker (i.e. 0.9 client with 0.10 broker, in the older versions).
> > >
> > > The latest processors are named with "2_0" so not sure if it makes
> sense
> > > to upgrade the client lib and then the name is no longer accurate, plus
> > > we'd have to be sure the latest 2.x client also works well with all
> > > previous 2.x versions back to 2.0.
> > >
> > > The other option is to make a new copy of the processor like "2_5" or
> > > whatever the latest version is, but I think we only want to do this if
> > it's
> > > really necessary. AFAIK the 2.0 client works with the latest 2.x
> brokers.
> > >
> > > Is there any specific issue you are looking to solve by using the
> latest
> > > client?
> > >
> > >
> > > On Fri, Jun 12, 2020 at 8:20 AM <christophe.monnet@post.ch<mailto:
> > > christophe.monnet@post.ch>> wrote:
> > > Hi
> > >
> > > Any reason why nifi-kafka-bundle is still bundled with
> > > kafka-clients-2.0.0.jar and not a newer version?
> > >
> > > Regards
> > > Chris
> > >
> >
>
Re: kafka-clients version
Posted by Jon Logan <jm...@buffalo.edu>.
Is there a reason why this question is being posed? If the version used is
compatible with the latest server version, and there's no missing
functionality or bugs being reported, I would strongly discourage updating,
as you're likely mandating server upgrades for at least some users. Even if
you make new processors, just creating new processors for no benefit is not
beneficial.
On Fri, Jun 12, 2020 at 9:50 AM Joe Witt <jo...@gmail.com> wrote:
> Chris
>
> In my experience with the clients the answer is a definite no. An older
> client will generally work (with important caveats at times) with a newer
> broker. A newer client will generally not work with an older broker. At
> the very least both major and minor version of the client matters so users
> can appropriately map to their broker.
>
> We just need to put out a 2.5 one now
>
> thanks
>
> On Fri, Jun 12, 2020 at 6:46 AM <ch...@post.ch> wrote:
>
> > Hi
> >
> > Please see the discussion below.
> > Would it make sense to always pack the latest minor version of
> > kafka-clients-jar in nifi-kafka-bundle?
> >
> > The processors could be named “2.x” (or simply “2” really) and the
> > specific minor version can change in NiFi releases.
> >
> > Regards
> > Chris
> >
> > From: Bryan Bende <bb...@gmail.com>
> > Sent: Friday, 12. June 2020 15:33
> > To: users@nifi.apache.org
> > Subject: Re: kafka-clients version
> >
> > Gotcha. Yes, we can obviously change the version of the client lib, but
> > it's misleading because they were named "2_0" because they use the "2_0"
> > client, and now that won't be true anymore, but if others agree with this
> > then maybe it's fine.
> >
> > Maybe more of a question for the dev list.
> >
> > On Fri, Jun 12, 2020 at 9:28 AM <christophe.monnet@post.ch<mailto:
> > christophe.monnet@post.ch>> wrote:
> > Yes: https://issues.apache.org/jira/browse/KAFKA-9815
> >
> > > we'd have to be sure the latest 2.x client also works well with all
> > previous 2.x versions back to 2.0
> >
> > I think kafka-clients should make sure of that.
> >
> > The processors could still be named 2_0, it’s only the client jar that
> > should be the latest 2.x version.
> > Specifically
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/pom.xml#L31
> >
> > Regards
> > Chris
> >
> > From: Bryan Bende <bb...@gmail.com>>
> > Sent: Friday, 12. June 2020 15:18
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
> > Subject: Re: kafka-clients version
> >
> > Usually we make a new version of the processors when something changes
> > that causes a previous version of the client to not work well with the
> > newer broker (i.e. 0.9 client with 0.10 broker, in the older versions).
> >
> > The latest processors are named with "2_0" so not sure if it makes sense
> > to upgrade the client lib and then the name is no longer accurate, plus
> > we'd have to be sure the latest 2.x client also works well with all
> > previous 2.x versions back to 2.0.
> >
> > The other option is to make a new copy of the processor like "2_5" or
> > whatever the latest version is, but I think we only want to do this if
> it's
> > really necessary. AFAIK the 2.0 client works with the latest 2.x brokers.
> >
> > Is there any specific issue you are looking to solve by using the latest
> > client?
> >
> >
> > On Fri, Jun 12, 2020 at 8:20 AM <christophe.monnet@post.ch<mailto:
> > christophe.monnet@post.ch>> wrote:
> > Hi
> >
> > Any reason why nifi-kafka-bundle is still bundled with
> > kafka-clients-2.0.0.jar and not a newer version?
> >
> > Regards
> > Chris
> >
>
RE: kafka-clients version
Posted by ch...@post.ch.
Hi Joe
According to https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix "newer versions of the Java clients and other clients that support KIP-35 such as librdkafka are able to fall back to older request types or throw appropriate errors if functionality is not available. This flexibility is a big win for users, who now have much more flexibility in adopting new client versions without requiring broker upgrades first."
That's why I suggested to always pack the latest client version.
If in your experience both major and minor version matters, then NiFi should allow to precisely select them.
At the moment we're stuck with client 2.0 while our brokers are being migrated from 2.4 to 2.5, so we're affected by issues like https://issues.apache.org/jira/browse/KAFKA-9815 and the blocker is the NiFi distribution.
As a workaround we could replace kafka-clients.jar in nifi-kafka-bundle.nar, what do you think?
Chris
-----Original Message-----
From: Joe Witt <jo...@gmail.com>
Sent: Friday, 12. June 2020 15:50
To: dev@nifi.apache.org
Subject: Re: kafka-clients version
Chris
In my experience with the clients the answer is a definite no. An older client will generally work (with important caveats at times) with a newer broker. A newer client will generally not work with an older broker. At the very least both major and minor version of the client matters so users can appropriately map to their broker.
We just need to put out a 2.5 one now
thanks
On Fri, Jun 12, 2020 at 6:46 AM <ch...@post.ch> wrote:
> Hi
>
> Please see the discussion below.
> Would it make sense to always pack the latest minor version of
> kafka-clients-jar in nifi-kafka-bundle?
>
> The processors could be named “2.x” (or simply “2” really) and the
> specific minor version can change in NiFi releases.
>
> Regards
> Chris
>
> From: Bryan Bende <bb...@gmail.com>
> Sent: Friday, 12. June 2020 15:33
> To: users@nifi.apache.org
> Subject: Re: kafka-clients version
>
> Gotcha. Yes, we can obviously change the version of the client lib,
> but it's misleading because they were named "2_0" because they use the "2_0"
> client, and now that won't be true anymore, but if others agree with
> this then maybe it's fine.
>
> Maybe more of a question for the dev list.
>
> On Fri, Jun 12, 2020 at 9:28 AM <christophe.monnet@post.ch<mailto:
> christophe.monnet@post.ch>> wrote:
> Yes: https://issues.apache.org/jira/browse/KAFKA-9815
>
> > we'd have to be sure the latest 2.x client also works well with all
> previous 2.x versions back to 2.0
>
> I think kafka-clients should make sure of that.
>
> The processors could still be named 2_0, it’s only the client jar that
> should be the latest 2.x version.
> Specifically
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka
> -bundle/pom.xml#L31
>
> Regards
> Chris
>
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Friday, 12. June 2020 15:18
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: kafka-clients version
>
> Usually we make a new version of the processors when something changes
> that causes a previous version of the client to not work well with the
> newer broker (i.e. 0.9 client with 0.10 broker, in the older versions).
>
> The latest processors are named with "2_0" so not sure if it makes
> sense to upgrade the client lib and then the name is no longer
> accurate, plus we'd have to be sure the latest 2.x client also works
> well with all previous 2.x versions back to 2.0.
>
> The other option is to make a new copy of the processor like "2_5" or
> whatever the latest version is, but I think we only want to do this if
> it's really necessary. AFAIK the 2.0 client works with the latest 2.x brokers.
>
> Is there any specific issue you are looking to solve by using the
> latest client?
>
>
> On Fri, Jun 12, 2020 at 8:20 AM <christophe.monnet@post.ch<mailto:
> christophe.monnet@post.ch>> wrote:
> Hi
>
> Any reason why nifi-kafka-bundle is still bundled with
> kafka-clients-2.0.0.jar and not a newer version?
>
> Regards
> Chris
>
Re: kafka-clients version
Posted by Joe Witt <jo...@gmail.com>.
Chris
In my experience with the clients the answer is a definite no. An older
client will generally work (with important caveats at times) with a newer
broker. A newer client will generally not work with an older broker. At
the very least both major and minor version of the client matters so users
can appropriately map to their broker.
We just need to put out a 2.5 one now
thanks
On Fri, Jun 12, 2020 at 6:46 AM <ch...@post.ch> wrote:
> Hi
>
> Please see the discussion below.
> Would it make sense to always pack the latest minor version of
> kafka-clients-jar in nifi-kafka-bundle?
>
> The processors could be named “2.x” (or simply “2” really) and the
> specific minor version can change in NiFi releases.
>
> Regards
> Chris
>
> From: Bryan Bende <bb...@gmail.com>
> Sent: Friday, 12. June 2020 15:33
> To: users@nifi.apache.org
> Subject: Re: kafka-clients version
>
> Gotcha. Yes, we can obviously change the version of the client lib, but
> it's misleading because they were named "2_0" because they use the "2_0"
> client, and now that won't be true anymore, but if others agree with this
> then maybe it's fine.
>
> Maybe more of a question for the dev list.
>
> On Fri, Jun 12, 2020 at 9:28 AM <christophe.monnet@post.ch<mailto:
> christophe.monnet@post.ch>> wrote:
> Yes: https://issues.apache.org/jira/browse/KAFKA-9815
>
> > we'd have to be sure the latest 2.x client also works well with all
> previous 2.x versions back to 2.0
>
> I think kafka-clients should make sure of that.
>
> The processors could still be named 2_0, it’s only the client jar that
> should be the latest 2.x version.
> Specifically
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/pom.xml#L31
>
> Regards
> Chris
>
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Friday, 12. June 2020 15:18
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: kafka-clients version
>
> Usually we make a new version of the processors when something changes
> that causes a previous version of the client to not work well with the
> newer broker (i.e. 0.9 client with 0.10 broker, in the older versions).
>
> The latest processors are named with "2_0" so not sure if it makes sense
> to upgrade the client lib and then the name is no longer accurate, plus
> we'd have to be sure the latest 2.x client also works well with all
> previous 2.x versions back to 2.0.
>
> The other option is to make a new copy of the processor like "2_5" or
> whatever the latest version is, but I think we only want to do this if it's
> really necessary. AFAIK the 2.0 client works with the latest 2.x brokers.
>
> Is there any specific issue you are looking to solve by using the latest
> client?
>
>
> On Fri, Jun 12, 2020 at 8:20 AM <christophe.monnet@post.ch<mailto:
> christophe.monnet@post.ch>> wrote:
> Hi
>
> Any reason why nifi-kafka-bundle is still bundled with
> kafka-clients-2.0.0.jar and not a newer version?
>
> Regards
> Chris
>