You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Gary Russell <gr...@pivotal.io> on 2020/04/02 20:36:52 UTC
2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Is there any way to determine the broker version in the kafka-clients?
I need to determine whether I can use the new sendOffsetsToTransaction
with ConsumerGroupMetadata or use the old one.
If I use the new API with a 2.4.1 broker, I get
UpsupportedVersionException: Attempted to write a non-default generationId
at version 2
Alternatively, couldn't the client simply extract the groupId from the
ConsumerGroupMetadata and use the old struct if the broker is too old?
I'd rather not have a user property in my framework to tell us which API to
use.
Thanks in advance.
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Gary Russell <gr...@pivotal.io>.
Thanks; Matthias,
Very useful.
On Mon, Apr 6, 2020 at 8:05 PM Matthias J. Sax <mj...@apache.org> wrote:
> I guess one important point to mention is why Kafka Streams needs the
> internal config though: it's about a save upgrade path.
>
> Even if the user tells us that they are on old brokers, we call the new
> `sendOffsetsToTransaction()` API blindly and let the producer downgrade
> the request. If the user upgrades their brokers, the producer can
> automatically switch to the new request type. During this phase, the old
> `transaction.id` fencing _and_ the new fetch-offset-request fencing are
> both active at the same time.
>
> Because both fencing mechanism are active, a user can now switch the
> config to enable eos-beta and do a single round of rolling bounces.
>
> It is not save to have only `transaction.id` fencing (but not
> fetch-offset-request fencing) and switch to fetch-offset-request fencing
> in a single rolling bounce. If an error occurs during the upgrade, EOS
> might be violated.
>
> Thus, in the end the internal config is a way to simplify the upgrade
> mechanism. We could also have chosen to add a new public config to Kafka
> Streams instead, but we wanted to improve the user experience.
>
> Thus, for your use case Gary, you would need to have one more config
> value for the upgrade mode in which you still use the old model but
> enable fetch-offset-request fencing in parallel. An upgrade would
> require two rolling bounce, first to enable upgrade mode, second to
> enable the new feature.
>
>
> -Matthias
>
> On 4/6/20 12:10 PM, Gary Russell wrote:
> > Thanks, all,
> >
> >> Just to clarify, even for Streams client it cannot detect automatically
> the
> > broker's version and hence as KIP-447 proposed, the customer needs to
> set a
> > config value indicating that she is assured the broker version is newer
> and
> > hence the new API can be used.
> >
> > Yes, I noticed that; Ok, I'll go the same way, then and make it a
> > configuration option.
> >
> > Yes, KIP-584 looks interesting to me.
> >
> > Thanks again.
> >
> >
> > On Sun, Apr 5, 2020 at 5:30 PM Guozhang Wang <wa...@gmail.com> wrote:
> >
> >> Hello Gary,
> >>
> >> Just to clarify, even for Streams client it cannot detect automatically
> the
> >> broker's version and hence as KIP-447 proposed, the customer needs to
> set a
> >> config value indicating that she is assured the broker version is newer
> and
> >> hence the new API can be used. On the other hand, if the config is not
> set,
> >> even if broker is on newer version we would still use the old behavior
> (one
> >> producer per task) and the although overloaded sendOffsetsToTransaction
> is
> >> used, it would not be much helpful in the scope of KIP-447.
> >>
> >> As for your case, currently there's no good ways to determine if the new
> >> overloaded function can be safely used except the users need to indicate
> >> (via configs, for example) that she's assured the broker version is
> newer.
> >> I think the ultimate solution for you would be KIP-584, in which client
> can
> >> dynamically ask the broker cluster which version(s) they would support
> and
> >> decides which function to trigger accordingly.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Sat, Apr 4, 2020 at 2:46 PM Boyang Chen <re...@gmail.com>
> >> wrote:
> >>
> >>> That’s a fair point Ismael. After a second thought, I feel that if Gary
> >> is
> >>> building frameworks for general purpose usage, relying on private flag
> >>> seems not a good idea.
> >>>
> >>> On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma <is...@juma.me.uk> wrote:
> >>>
> >>>> The internal config was meant to be internal, right? That is, no
> >>>> compatibility guarantees are offered? The current discussion implies
> we
> >>> are
> >>>> considering it a public config.
> >>>>
> >>>> Ismael
> >>>>
> >>>> On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <bo...@confluent.io>
> >> wrote:
> >>>>
> >>>>> For Gary's case, I think the internal config should be a sort of
> >> help,
> >>>> and
> >>>>> not violating our public agreement.
> >>>>>
> >>>>> On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org>
> >>> wrote:
> >>>>>
> >>>>>> I guess you would need to catch the exception and retry?
> >>>>>>
> >>>>>> It's a little unfortunate. Not sure if we could back-port the
> >>> internal
> >>>>>> producer config that we add in 2.6 for auto-downgrade to a 2.5 bug
> >>> fix
> >>>>>> release?
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>>
> >>>>>> On 4/2/20 7:25 PM, Gary Russell wrote:
> >>>>>>> Thanks Mattias
> >>>>>>>
> >>>>>>>> Hence, why do you want/need to switch to the newer overload that
> >>>> only
> >>>>>>> works for 2.5+ brokers?
> >>>>>>>
> >>>>>>> So I can choose to use the producer per consumer thread Vs. the
> >>>>> producer
> >>>>>>> per group/topic/partition threading model for zombie fencing,
> >> based
> >>>> on
> >>>>>> the
> >>>>>>> broker version.
> >>>>>>>
> >>>>>>> I don't have the same luxury as kafka streams (i.e. don't use
> >>> streams
> >>>>> 2.6
> >>>>>>> unless you have 2.5+ brokers).
> >>>>>>>
> >>>>>>> I add new features with each minor release (and try to use the
> >>> latest
> >>>>>>> kafka-clients as they become available).
> >>>>>>>
> >>>>>>> Users may want other new features, not related to EOS, and they
> >>> might
> >>>>>> stay
> >>>>>>> on old brokers.
> >>>>>>>
> >>>>>>> Other users might want to take advantage of the improved
> >>> performance
> >>>> of
> >>>>>> the
> >>>>>>> new EOS so I need to support both APIs.
> >>>>>>>
> >>>>>>> Many enterprises take forever to upgrade their brokers. I
> >> recently
> >>>> had
> >>>>> a
> >>>>>>> question of why my latest version won't work with an 0.9.x.x
> >> broker
> >>>>>> (sigh).
> >>>>>>>
> >>>>>>> Spring versioning rules don't allow me to bump kafka-clients
> >>> versions
> >>>>> in
> >>>>>> a
> >>>>>>> patch release so I am already supporting 4 active branches and I
> >> am
> >>>>>> trying
> >>>>>>> to avoid supporting a fifth.
> >>>>>>>
> >>>>>>> Thanks again.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mjsax@apache.org
> >>>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Gary,
> >>>>>>>>
> >>>>>>>> thanks for the question. We recently had a discussion about the
> >>>> exact
> >>>>>>>> some topic:
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
> >>>>>>>>
> >>>>>>>> Note that the "old" `sendOffsetsToTransaction(..., String
> >>> groupId)`
> >>>> is
> >>>>>>>> not deprecated. Hence, why do you want/need to switch to the
> >> newer
> >>>>>>>> overload that only works for 2.5+ brokers? For many/most cases,
> >>> the
> >>>>>>>> "old" API that is compatible with older broker still does what
> >> you
> >>>>> need
> >>>>>>>> and there in not need to switch to the newer API.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 4/2/20 1:54 PM, Gary Russell wrote:
> >>>>>>>>> Thanks, Boyang,
> >>>>>>>>>
> >>>>>>>>> I maintain a framework (Spring for Apache Kafka) that sits on
> >> top
> >>>> of
> >>>>>> the
> >>>>>>>>> clients, and I would like to be able to support all broker
> >>>> versions.
> >>>>> I
> >>>>>>>>> don't have control over what brokers my users are using.
> >>>>>>>>>
> >>>>>>>>> You guys have done a great job since 0.10.2.0 (I think) of
> >>>> supporting
> >>>>>>>> older
> >>>>>>>>> brokers from newer clients but this one's a blocker for me.
> >>>>>>>>>
> >>>>>>>>> My framework will enforce the proper semantics for EOS,
> >> depending
> >>>> on
> >>>>>> the
> >>>>>>>>> broker version, but I need to know which model to use at
> >> runtime.
> >>>>>>>>>
> >>>>>>>>> As I said, I can have a property that the user can set to tell
> >>> the
> >>>>>>>>> framework that the broker is >= 2.5 but it would be cleaner if
> >> I
> >>>>> could
> >>>>>>>> stay
> >>>>>>>>> away from that.
> >>>>>>>>>
> >>>>>>>>> Something like KafkaAdminClient.brokerApi() (or add the lowest
> >>>>>> API/broker
> >>>>>>>>> version to describeCluster()), would be helpful.
> >>>>>>>>>
> >>>>>>>>> Worst case, I'll add a configuration option.
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>>
> >>>>>>>>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <
> >>>>> reluctanthero104@gmail.com
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks for the question Gary. The reasoning for crash the new
> >>>>>>>>>> sendTxnOffsets API is because we don't want users to
> >>> unconsciously
> >>>>>>>> violate
> >>>>>>>>>> the EOS guarantee. In your case, using this API with 2.4.1 is
> >>> not
> >>>>>>>> supported
> >>>>>>>>>> anyway, so the upgrade path has to start from broker first to
> >>> 2.5,
> >>>>> and
> >>>>>>>> then
> >>>>>>>>>> client binaries. Is there any further concern that blocks you
> >>> from
> >>>>>>>> getting
> >>>>>>>>>> the broker side upgrade first before using the new API?
> >>>>>>>>>>
> >>>>>>>>>> Boyang
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <
> >>> grussell@pivotal.io>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Is there any way to determine the broker version in the
> >>>>>> kafka-clients?
> >>>>>>>>>>>
> >>>>>>>>>>> I need to determine whether I can use the new
> >>>>>> sendOffsetsToTransaction
> >>>>>>>>>>> with ConsumerGroupMetadata or use the old one.
> >>>>>>>>>>>
> >>>>>>>>>>> If I use the new API with a 2.4.1 broker, I get
> >>>>>>>>>>>
> >>>>>>>>>>> UpsupportedVersionException: Attempted to write a non-default
> >>>>>>>>>> generationId
> >>>>>>>>>>> at version 2
> >>>>>>>>>>>
> >>>>>>>>>>> Alternatively, couldn't the client simply extract the groupId
> >>>> from
> >>>>>> the
> >>>>>>>>>>> ConsumerGroupMetadata and use the old struct if the broker is
> >>> too
> >>>>>> old?
> >>>>>>>>>>>
> >>>>>>>>>>> I'd rather not have a user property in my framework to tell
> >> us
> >>>>> which
> >>>>>>>> API
> >>>>>>>>>> to
> >>>>>>>>>>> use.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks in advance.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards
Compatibility
Posted by "Matthias J. Sax" <mj...@apache.org>.
I guess one important point to mention is why Kafka Streams needs the
internal config though: it's about a save upgrade path.
Even if the user tells us that they are on old brokers, we call the new
`sendOffsetsToTransaction()` API blindly and let the producer downgrade
the request. If the user upgrades their brokers, the producer can
automatically switch to the new request type. During this phase, the old
`transaction.id` fencing _and_ the new fetch-offset-request fencing are
both active at the same time.
Because both fencing mechanism are active, a user can now switch the
config to enable eos-beta and do a single round of rolling bounces.
It is not save to have only `transaction.id` fencing (but not
fetch-offset-request fencing) and switch to fetch-offset-request fencing
in a single rolling bounce. If an error occurs during the upgrade, EOS
might be violated.
Thus, in the end the internal config is a way to simplify the upgrade
mechanism. We could also have chosen to add a new public config to Kafka
Streams instead, but we wanted to improve the user experience.
Thus, for your use case Gary, you would need to have one more config
value for the upgrade mode in which you still use the old model but
enable fetch-offset-request fencing in parallel. An upgrade would
require two rolling bounce, first to enable upgrade mode, second to
enable the new feature.
-Matthias
On 4/6/20 12:10 PM, Gary Russell wrote:
> Thanks, all,
>
>> Just to clarify, even for Streams client it cannot detect automatically the
> broker's version and hence as KIP-447 proposed, the customer needs to set a
> config value indicating that she is assured the broker version is newer and
> hence the new API can be used.
>
> Yes, I noticed that; Ok, I'll go the same way, then and make it a
> configuration option.
>
> Yes, KIP-584 looks interesting to me.
>
> Thanks again.
>
>
> On Sun, Apr 5, 2020 at 5:30 PM Guozhang Wang <wa...@gmail.com> wrote:
>
>> Hello Gary,
>>
>> Just to clarify, even for Streams client it cannot detect automatically the
>> broker's version and hence as KIP-447 proposed, the customer needs to set a
>> config value indicating that she is assured the broker version is newer and
>> hence the new API can be used. On the other hand, if the config is not set,
>> even if broker is on newer version we would still use the old behavior (one
>> producer per task) and the although overloaded sendOffsetsToTransaction is
>> used, it would not be much helpful in the scope of KIP-447.
>>
>> As for your case, currently there's no good ways to determine if the new
>> overloaded function can be safely used except the users need to indicate
>> (via configs, for example) that she's assured the broker version is newer.
>> I think the ultimate solution for you would be KIP-584, in which client can
>> dynamically ask the broker cluster which version(s) they would support and
>> decides which function to trigger accordingly.
>>
>>
>> Guozhang
>>
>>
>> On Sat, Apr 4, 2020 at 2:46 PM Boyang Chen <re...@gmail.com>
>> wrote:
>>
>>> That’s a fair point Ismael. After a second thought, I feel that if Gary
>> is
>>> building frameworks for general purpose usage, relying on private flag
>>> seems not a good idea.
>>>
>>> On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma <is...@juma.me.uk> wrote:
>>>
>>>> The internal config was meant to be internal, right? That is, no
>>>> compatibility guarantees are offered? The current discussion implies we
>>> are
>>>> considering it a public config.
>>>>
>>>> Ismael
>>>>
>>>> On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <bo...@confluent.io>
>> wrote:
>>>>
>>>>> For Gary's case, I think the internal config should be a sort of
>> help,
>>>> and
>>>>> not violating our public agreement.
>>>>>
>>>>> On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org>
>>> wrote:
>>>>>
>>>>>> I guess you would need to catch the exception and retry?
>>>>>>
>>>>>> It's a little unfortunate. Not sure if we could back-port the
>>> internal
>>>>>> producer config that we add in 2.6 for auto-downgrade to a 2.5 bug
>>> fix
>>>>>> release?
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>> On 4/2/20 7:25 PM, Gary Russell wrote:
>>>>>>> Thanks Mattias
>>>>>>>
>>>>>>>> Hence, why do you want/need to switch to the newer overload that
>>>> only
>>>>>>> works for 2.5+ brokers?
>>>>>>>
>>>>>>> So I can choose to use the producer per consumer thread Vs. the
>>>>> producer
>>>>>>> per group/topic/partition threading model for zombie fencing,
>> based
>>>> on
>>>>>> the
>>>>>>> broker version.
>>>>>>>
>>>>>>> I don't have the same luxury as kafka streams (i.e. don't use
>>> streams
>>>>> 2.6
>>>>>>> unless you have 2.5+ brokers).
>>>>>>>
>>>>>>> I add new features with each minor release (and try to use the
>>> latest
>>>>>>> kafka-clients as they become available).
>>>>>>>
>>>>>>> Users may want other new features, not related to EOS, and they
>>> might
>>>>>> stay
>>>>>>> on old brokers.
>>>>>>>
>>>>>>> Other users might want to take advantage of the improved
>>> performance
>>>> of
>>>>>> the
>>>>>>> new EOS so I need to support both APIs.
>>>>>>>
>>>>>>> Many enterprises take forever to upgrade their brokers. I
>> recently
>>>> had
>>>>> a
>>>>>>> question of why my latest version won't work with an 0.9.x.x
>> broker
>>>>>> (sigh).
>>>>>>>
>>>>>>> Spring versioning rules don't allow me to bump kafka-clients
>>> versions
>>>>> in
>>>>>> a
>>>>>>> patch release so I am already supporting 4 active branches and I
>> am
>>>>>> trying
>>>>>>> to avoid supporting a fifth.
>>>>>>>
>>>>>>> Thanks again.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mjsax@apache.org
>>>
>>>>> wrote:
>>>>>>>
>>>>>>>> Gary,
>>>>>>>>
>>>>>>>> thanks for the question. We recently had a discussion about the
>>>> exact
>>>>>>>> some topic:
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
>>>>>>>>
>>>>>>>> Note that the "old" `sendOffsetsToTransaction(..., String
>>> groupId)`
>>>> is
>>>>>>>> not deprecated. Hence, why do you want/need to switch to the
>> newer
>>>>>>>> overload that only works for 2.5+ brokers? For many/most cases,
>>> the
>>>>>>>> "old" API that is compatible with older broker still does what
>> you
>>>>> need
>>>>>>>> and there in not need to switch to the newer API.
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/2/20 1:54 PM, Gary Russell wrote:
>>>>>>>>> Thanks, Boyang,
>>>>>>>>>
>>>>>>>>> I maintain a framework (Spring for Apache Kafka) that sits on
>> top
>>>> of
>>>>>> the
>>>>>>>>> clients, and I would like to be able to support all broker
>>>> versions.
>>>>> I
>>>>>>>>> don't have control over what brokers my users are using.
>>>>>>>>>
>>>>>>>>> You guys have done a great job since 0.10.2.0 (I think) of
>>>> supporting
>>>>>>>> older
>>>>>>>>> brokers from newer clients but this one's a blocker for me.
>>>>>>>>>
>>>>>>>>> My framework will enforce the proper semantics for EOS,
>> depending
>>>> on
>>>>>> the
>>>>>>>>> broker version, but I need to know which model to use at
>> runtime.
>>>>>>>>>
>>>>>>>>> As I said, I can have a property that the user can set to tell
>>> the
>>>>>>>>> framework that the broker is >= 2.5 but it would be cleaner if
>> I
>>>>> could
>>>>>>>> stay
>>>>>>>>> away from that.
>>>>>>>>>
>>>>>>>>> Something like KafkaAdminClient.brokerApi() (or add the lowest
>>>>>> API/broker
>>>>>>>>> version to describeCluster()), would be helpful.
>>>>>>>>>
>>>>>>>>> Worst case, I'll add a configuration option.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <
>>>>> reluctanthero104@gmail.com
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the question Gary. The reasoning for crash the new
>>>>>>>>>> sendTxnOffsets API is because we don't want users to
>>> unconsciously
>>>>>>>> violate
>>>>>>>>>> the EOS guarantee. In your case, using this API with 2.4.1 is
>>> not
>>>>>>>> supported
>>>>>>>>>> anyway, so the upgrade path has to start from broker first to
>>> 2.5,
>>>>> and
>>>>>>>> then
>>>>>>>>>> client binaries. Is there any further concern that blocks you
>>> from
>>>>>>>> getting
>>>>>>>>>> the broker side upgrade first before using the new API?
>>>>>>>>>>
>>>>>>>>>> Boyang
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <
>>> grussell@pivotal.io>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Is there any way to determine the broker version in the
>>>>>> kafka-clients?
>>>>>>>>>>>
>>>>>>>>>>> I need to determine whether I can use the new
>>>>>> sendOffsetsToTransaction
>>>>>>>>>>> with ConsumerGroupMetadata or use the old one.
>>>>>>>>>>>
>>>>>>>>>>> If I use the new API with a 2.4.1 broker, I get
>>>>>>>>>>>
>>>>>>>>>>> UpsupportedVersionException: Attempted to write a non-default
>>>>>>>>>> generationId
>>>>>>>>>>> at version 2
>>>>>>>>>>>
>>>>>>>>>>> Alternatively, couldn't the client simply extract the groupId
>>>> from
>>>>>> the
>>>>>>>>>>> ConsumerGroupMetadata and use the old struct if the broker is
>>> too
>>>>>> old?
>>>>>>>>>>>
>>>>>>>>>>> I'd rather not have a user property in my framework to tell
>> us
>>>>> which
>>>>>>>> API
>>>>>>>>>> to
>>>>>>>>>>> use.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> -- Guozhang
>>
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Gary Russell <gr...@pivotal.io>.
Thanks, all,
>Just to clarify, even for Streams client it cannot detect automatically the
broker's version and hence as KIP-447 proposed, the customer needs to set a
config value indicating that she is assured the broker version is newer and
hence the new API can be used.
Yes, I noticed that; Ok, I'll go the same way, then and make it a
configuration option.
Yes, KIP-584 looks interesting to me.
Thanks again.
On Sun, Apr 5, 2020 at 5:30 PM Guozhang Wang <wa...@gmail.com> wrote:
> Hello Gary,
>
> Just to clarify, even for Streams client it cannot detect automatically the
> broker's version and hence as KIP-447 proposed, the customer needs to set a
> config value indicating that she is assured the broker version is newer and
> hence the new API can be used. On the other hand, if the config is not set,
> even if broker is on newer version we would still use the old behavior (one
> producer per task) and the although overloaded sendOffsetsToTransaction is
> used, it would not be much helpful in the scope of KIP-447.
>
> As for your case, currently there's no good ways to determine if the new
> overloaded function can be safely used except the users need to indicate
> (via configs, for example) that she's assured the broker version is newer.
> I think the ultimate solution for you would be KIP-584, in which client can
> dynamically ask the broker cluster which version(s) they would support and
> decides which function to trigger accordingly.
>
>
> Guozhang
>
>
> On Sat, Apr 4, 2020 at 2:46 PM Boyang Chen <re...@gmail.com>
> wrote:
>
> > That’s a fair point Ismael. After a second thought, I feel that if Gary
> is
> > building frameworks for general purpose usage, relying on private flag
> > seems not a good idea.
> >
> > On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > The internal config was meant to be internal, right? That is, no
> > > compatibility guarantees are offered? The current discussion implies we
> > are
> > > considering it a public config.
> > >
> > > Ismael
> > >
> > > On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <bo...@confluent.io>
> wrote:
> > >
> > > > For Gary's case, I think the internal config should be a sort of
> help,
> > > and
> > > > not violating our public agreement.
> > > >
> > > > On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > > >
> > > > > I guess you would need to catch the exception and retry?
> > > > >
> > > > > It's a little unfortunate. Not sure if we could back-port the
> > internal
> > > > > producer config that we add in 2.6 for auto-downgrade to a 2.5 bug
> > fix
> > > > > release?
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > >
> > > > > On 4/2/20 7:25 PM, Gary Russell wrote:
> > > > > > Thanks Mattias
> > > > > >
> > > > > >> Hence, why do you want/need to switch to the newer overload that
> > > only
> > > > > > works for 2.5+ brokers?
> > > > > >
> > > > > > So I can choose to use the producer per consumer thread Vs. the
> > > > producer
> > > > > > per group/topic/partition threading model for zombie fencing,
> based
> > > on
> > > > > the
> > > > > > broker version.
> > > > > >
> > > > > > I don't have the same luxury as kafka streams (i.e. don't use
> > streams
> > > > 2.6
> > > > > > unless you have 2.5+ brokers).
> > > > > >
> > > > > > I add new features with each minor release (and try to use the
> > latest
> > > > > > kafka-clients as they become available).
> > > > > >
> > > > > > Users may want other new features, not related to EOS, and they
> > might
> > > > > stay
> > > > > > on old brokers.
> > > > > >
> > > > > > Other users might want to take advantage of the improved
> > performance
> > > of
> > > > > the
> > > > > > new EOS so I need to support both APIs.
> > > > > >
> > > > > > Many enterprises take forever to upgrade their brokers. I
> recently
> > > had
> > > > a
> > > > > > question of why my latest version won't work with an 0.9.x.x
> broker
> > > > > (sigh).
> > > > > >
> > > > > > Spring versioning rules don't allow me to bump kafka-clients
> > versions
> > > > in
> > > > > a
> > > > > > patch release so I am already supporting 4 active branches and I
> am
> > > > > trying
> > > > > > to avoid supporting a fifth.
> > > > > >
> > > > > > Thanks again.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mjsax@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > >> Gary,
> > > > > >>
> > > > > >> thanks for the question. We recently had a discussion about the
> > > exact
> > > > > >> some topic:
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
> > > > > >>
> > > > > >> Note that the "old" `sendOffsetsToTransaction(..., String
> > groupId)`
> > > is
> > > > > >> not deprecated. Hence, why do you want/need to switch to the
> newer
> > > > > >> overload that only works for 2.5+ brokers? For many/most cases,
> > the
> > > > > >> "old" API that is compatible with older broker still does what
> you
> > > > need
> > > > > >> and there in not need to switch to the newer API.
> > > > > >>
> > > > > >>
> > > > > >> -Matthias
> > > > > >>
> > > > > >>
> > > > > >> On 4/2/20 1:54 PM, Gary Russell wrote:
> > > > > >>> Thanks, Boyang,
> > > > > >>>
> > > > > >>> I maintain a framework (Spring for Apache Kafka) that sits on
> top
> > > of
> > > > > the
> > > > > >>> clients, and I would like to be able to support all broker
> > > versions.
> > > > I
> > > > > >>> don't have control over what brokers my users are using.
> > > > > >>>
> > > > > >>> You guys have done a great job since 0.10.2.0 (I think) of
> > > supporting
> > > > > >> older
> > > > > >>> brokers from newer clients but this one's a blocker for me.
> > > > > >>>
> > > > > >>> My framework will enforce the proper semantics for EOS,
> depending
> > > on
> > > > > the
> > > > > >>> broker version, but I need to know which model to use at
> runtime.
> > > > > >>>
> > > > > >>> As I said, I can have a property that the user can set to tell
> > the
> > > > > >>> framework that the broker is >= 2.5 but it would be cleaner if
> I
> > > > could
> > > > > >> stay
> > > > > >>> away from that.
> > > > > >>>
> > > > > >>> Something like KafkaAdminClient.brokerApi() (or add the lowest
> > > > > API/broker
> > > > > >>> version to describeCluster()), would be helpful.
> > > > > >>>
> > > > > >>> Worst case, I'll add a configuration option.
> > > > > >>>
> > > > > >>> Thanks.
> > > > > >>>
> > > > > >>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <
> > > > reluctanthero104@gmail.com
> > > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Thanks for the question Gary. The reasoning for crash the new
> > > > > >>>> sendTxnOffsets API is because we don't want users to
> > unconsciously
> > > > > >> violate
> > > > > >>>> the EOS guarantee. In your case, using this API with 2.4.1 is
> > not
> > > > > >> supported
> > > > > >>>> anyway, so the upgrade path has to start from broker first to
> > 2.5,
> > > > and
> > > > > >> then
> > > > > >>>> client binaries. Is there any further concern that blocks you
> > from
> > > > > >> getting
> > > > > >>>> the broker side upgrade first before using the new API?
> > > > > >>>>
> > > > > >>>> Boyang
> > > > > >>>>
> > > > > >>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <
> > grussell@pivotal.io>
> > > > > >> wrote:
> > > > > >>>>
> > > > > >>>>> Is there any way to determine the broker version in the
> > > > > kafka-clients?
> > > > > >>>>>
> > > > > >>>>> I need to determine whether I can use the new
> > > > > sendOffsetsToTransaction
> > > > > >>>>> with ConsumerGroupMetadata or use the old one.
> > > > > >>>>>
> > > > > >>>>> If I use the new API with a 2.4.1 broker, I get
> > > > > >>>>>
> > > > > >>>>> UpsupportedVersionException: Attempted to write a non-default
> > > > > >>>> generationId
> > > > > >>>>> at version 2
> > > > > >>>>>
> > > > > >>>>> Alternatively, couldn't the client simply extract the groupId
> > > from
> > > > > the
> > > > > >>>>> ConsumerGroupMetadata and use the old struct if the broker is
> > too
> > > > > old?
> > > > > >>>>>
> > > > > >>>>> I'd rather not have a user property in my framework to tell
> us
> > > > which
> > > > > >> API
> > > > > >>>> to
> > > > > >>>>> use.
> > > > > >>>>>
> > > > > >>>>> Thanks in advance.
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Gary,
Just to clarify, even for Streams client it cannot detect automatically the
broker's version and hence as KIP-447 proposed, the customer needs to set a
config value indicating that she is assured the broker version is newer and
hence the new API can be used. On the other hand, if the config is not set,
even if broker is on newer version we would still use the old behavior (one
producer per task) and the although overloaded sendOffsetsToTransaction is
used, it would not be much helpful in the scope of KIP-447.
As for your case, currently there's no good ways to determine if the new
overloaded function can be safely used except the users need to indicate
(via configs, for example) that she's assured the broker version is newer.
I think the ultimate solution for you would be KIP-584, in which client can
dynamically ask the broker cluster which version(s) they would support and
decides which function to trigger accordingly.
Guozhang
On Sat, Apr 4, 2020 at 2:46 PM Boyang Chen <re...@gmail.com>
wrote:
> That’s a fair point Ismael. After a second thought, I feel that if Gary is
> building frameworks for general purpose usage, relying on private flag
> seems not a good idea.
>
> On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma <is...@juma.me.uk> wrote:
>
> > The internal config was meant to be internal, right? That is, no
> > compatibility guarantees are offered? The current discussion implies we
> are
> > considering it a public config.
> >
> > Ismael
> >
> > On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <bo...@confluent.io> wrote:
> >
> > > For Gary's case, I think the internal config should be a sort of help,
> > and
> > > not violating our public agreement.
> > >
> > > On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> > >
> > > > I guess you would need to catch the exception and retry?
> > > >
> > > > It's a little unfortunate. Not sure if we could back-port the
> internal
> > > > producer config that we add in 2.6 for auto-downgrade to a 2.5 bug
> fix
> > > > release?
> > > >
> > > >
> > > > -Matthias
> > > >
> > > >
> > > > On 4/2/20 7:25 PM, Gary Russell wrote:
> > > > > Thanks Mattias
> > > > >
> > > > >> Hence, why do you want/need to switch to the newer overload that
> > only
> > > > > works for 2.5+ brokers?
> > > > >
> > > > > So I can choose to use the producer per consumer thread Vs. the
> > > producer
> > > > > per group/topic/partition threading model for zombie fencing, based
> > on
> > > > the
> > > > > broker version.
> > > > >
> > > > > I don't have the same luxury as kafka streams (i.e. don't use
> streams
> > > 2.6
> > > > > unless you have 2.5+ brokers).
> > > > >
> > > > > I add new features with each minor release (and try to use the
> latest
> > > > > kafka-clients as they become available).
> > > > >
> > > > > Users may want other new features, not related to EOS, and they
> might
> > > > stay
> > > > > on old brokers.
> > > > >
> > > > > Other users might want to take advantage of the improved
> performance
> > of
> > > > the
> > > > > new EOS so I need to support both APIs.
> > > > >
> > > > > Many enterprises take forever to upgrade their brokers. I recently
> > had
> > > a
> > > > > question of why my latest version won't work with an 0.9.x.x broker
> > > > (sigh).
> > > > >
> > > > > Spring versioning rules don't allow me to bump kafka-clients
> versions
> > > in
> > > > a
> > > > > patch release so I am already supporting 4 active branches and I am
> > > > trying
> > > > > to avoid supporting a fifth.
> > > > >
> > > > > Thanks again.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org>
> > > wrote:
> > > > >
> > > > >> Gary,
> > > > >>
> > > > >> thanks for the question. We recently had a discussion about the
> > exact
> > > > >> some topic:
> > > > >>
> > > > >>
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
> > > > >>
> > > > >> Note that the "old" `sendOffsetsToTransaction(..., String
> groupId)`
> > is
> > > > >> not deprecated. Hence, why do you want/need to switch to the newer
> > > > >> overload that only works for 2.5+ brokers? For many/most cases,
> the
> > > > >> "old" API that is compatible with older broker still does what you
> > > need
> > > > >> and there in not need to switch to the newer API.
> > > > >>
> > > > >>
> > > > >> -Matthias
> > > > >>
> > > > >>
> > > > >> On 4/2/20 1:54 PM, Gary Russell wrote:
> > > > >>> Thanks, Boyang,
> > > > >>>
> > > > >>> I maintain a framework (Spring for Apache Kafka) that sits on top
> > of
> > > > the
> > > > >>> clients, and I would like to be able to support all broker
> > versions.
> > > I
> > > > >>> don't have control over what brokers my users are using.
> > > > >>>
> > > > >>> You guys have done a great job since 0.10.2.0 (I think) of
> > supporting
> > > > >> older
> > > > >>> brokers from newer clients but this one's a blocker for me.
> > > > >>>
> > > > >>> My framework will enforce the proper semantics for EOS, depending
> > on
> > > > the
> > > > >>> broker version, but I need to know which model to use at runtime.
> > > > >>>
> > > > >>> As I said, I can have a property that the user can set to tell
> the
> > > > >>> framework that the broker is >= 2.5 but it would be cleaner if I
> > > could
> > > > >> stay
> > > > >>> away from that.
> > > > >>>
> > > > >>> Something like KafkaAdminClient.brokerApi() (or add the lowest
> > > > API/broker
> > > > >>> version to describeCluster()), would be helpful.
> > > > >>>
> > > > >>> Worst case, I'll add a configuration option.
> > > > >>>
> > > > >>> Thanks.
> > > > >>>
> > > > >>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <
> > > reluctanthero104@gmail.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Thanks for the question Gary. The reasoning for crash the new
> > > > >>>> sendTxnOffsets API is because we don't want users to
> unconsciously
> > > > >> violate
> > > > >>>> the EOS guarantee. In your case, using this API with 2.4.1 is
> not
> > > > >> supported
> > > > >>>> anyway, so the upgrade path has to start from broker first to
> 2.5,
> > > and
> > > > >> then
> > > > >>>> client binaries. Is there any further concern that blocks you
> from
> > > > >> getting
> > > > >>>> the broker side upgrade first before using the new API?
> > > > >>>>
> > > > >>>> Boyang
> > > > >>>>
> > > > >>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <
> grussell@pivotal.io>
> > > > >> wrote:
> > > > >>>>
> > > > >>>>> Is there any way to determine the broker version in the
> > > > kafka-clients?
> > > > >>>>>
> > > > >>>>> I need to determine whether I can use the new
> > > > sendOffsetsToTransaction
> > > > >>>>> with ConsumerGroupMetadata or use the old one.
> > > > >>>>>
> > > > >>>>> If I use the new API with a 2.4.1 broker, I get
> > > > >>>>>
> > > > >>>>> UpsupportedVersionException: Attempted to write a non-default
> > > > >>>> generationId
> > > > >>>>> at version 2
> > > > >>>>>
> > > > >>>>> Alternatively, couldn't the client simply extract the groupId
> > from
> > > > the
> > > > >>>>> ConsumerGroupMetadata and use the old struct if the broker is
> too
> > > > old?
> > > > >>>>>
> > > > >>>>> I'd rather not have a user property in my framework to tell us
> > > which
> > > > >> API
> > > > >>>> to
> > > > >>>>> use.
> > > > >>>>>
> > > > >>>>> Thanks in advance.
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>
--
-- Guozhang
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Boyang Chen <re...@gmail.com>.
That’s a fair point Ismael. After a second thought, I feel that if Gary is
building frameworks for general purpose usage, relying on private flag
seems not a good idea.
On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma <is...@juma.me.uk> wrote:
> The internal config was meant to be internal, right? That is, no
> compatibility guarantees are offered? The current discussion implies we are
> considering it a public config.
>
> Ismael
>
> On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <bo...@confluent.io> wrote:
>
> > For Gary's case, I think the internal config should be a sort of help,
> and
> > not violating our public agreement.
> >
> > On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org> wrote:
> >
> > > I guess you would need to catch the exception and retry?
> > >
> > > It's a little unfortunate. Not sure if we could back-port the internal
> > > producer config that we add in 2.6 for auto-downgrade to a 2.5 bug fix
> > > release?
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 4/2/20 7:25 PM, Gary Russell wrote:
> > > > Thanks Mattias
> > > >
> > > >> Hence, why do you want/need to switch to the newer overload that
> only
> > > > works for 2.5+ brokers?
> > > >
> > > > So I can choose to use the producer per consumer thread Vs. the
> > producer
> > > > per group/topic/partition threading model for zombie fencing, based
> on
> > > the
> > > > broker version.
> > > >
> > > > I don't have the same luxury as kafka streams (i.e. don't use streams
> > 2.6
> > > > unless you have 2.5+ brokers).
> > > >
> > > > I add new features with each minor release (and try to use the latest
> > > > kafka-clients as they become available).
> > > >
> > > > Users may want other new features, not related to EOS, and they might
> > > stay
> > > > on old brokers.
> > > >
> > > > Other users might want to take advantage of the improved performance
> of
> > > the
> > > > new EOS so I need to support both APIs.
> > > >
> > > > Many enterprises take forever to upgrade their brokers. I recently
> had
> > a
> > > > question of why my latest version won't work with an 0.9.x.x broker
> > > (sigh).
> > > >
> > > > Spring versioning rules don't allow me to bump kafka-clients versions
> > in
> > > a
> > > > patch release so I am already supporting 4 active branches and I am
> > > trying
> > > > to avoid supporting a fifth.
> > > >
> > > > Thanks again.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > > >
> > > >> Gary,
> > > >>
> > > >> thanks for the question. We recently had a discussion about the
> exact
> > > >> some topic:
> > > >>
> > > >>
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
> > > >>
> > > >> Note that the "old" `sendOffsetsToTransaction(..., String groupId)`
> is
> > > >> not deprecated. Hence, why do you want/need to switch to the newer
> > > >> overload that only works for 2.5+ brokers? For many/most cases, the
> > > >> "old" API that is compatible with older broker still does what you
> > need
> > > >> and there in not need to switch to the newer API.
> > > >>
> > > >>
> > > >> -Matthias
> > > >>
> > > >>
> > > >> On 4/2/20 1:54 PM, Gary Russell wrote:
> > > >>> Thanks, Boyang,
> > > >>>
> > > >>> I maintain a framework (Spring for Apache Kafka) that sits on top
> of
> > > the
> > > >>> clients, and I would like to be able to support all broker
> versions.
> > I
> > > >>> don't have control over what brokers my users are using.
> > > >>>
> > > >>> You guys have done a great job since 0.10.2.0 (I think) of
> supporting
> > > >> older
> > > >>> brokers from newer clients but this one's a blocker for me.
> > > >>>
> > > >>> My framework will enforce the proper semantics for EOS, depending
> on
> > > the
> > > >>> broker version, but I need to know which model to use at runtime.
> > > >>>
> > > >>> As I said, I can have a property that the user can set to tell the
> > > >>> framework that the broker is >= 2.5 but it would be cleaner if I
> > could
> > > >> stay
> > > >>> away from that.
> > > >>>
> > > >>> Something like KafkaAdminClient.brokerApi() (or add the lowest
> > > API/broker
> > > >>> version to describeCluster()), would be helpful.
> > > >>>
> > > >>> Worst case, I'll add a configuration option.
> > > >>>
> > > >>> Thanks.
> > > >>>
> > > >>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <
> > reluctanthero104@gmail.com
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> Thanks for the question Gary. The reasoning for crash the new
> > > >>>> sendTxnOffsets API is because we don't want users to unconsciously
> > > >> violate
> > > >>>> the EOS guarantee. In your case, using this API with 2.4.1 is not
> > > >> supported
> > > >>>> anyway, so the upgrade path has to start from broker first to 2.5,
> > and
> > > >> then
> > > >>>> client binaries. Is there any further concern that blocks you from
> > > >> getting
> > > >>>> the broker side upgrade first before using the new API?
> > > >>>>
> > > >>>> Boyang
> > > >>>>
> > > >>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gr...@pivotal.io>
> > > >> wrote:
> > > >>>>
> > > >>>>> Is there any way to determine the broker version in the
> > > kafka-clients?
> > > >>>>>
> > > >>>>> I need to determine whether I can use the new
> > > sendOffsetsToTransaction
> > > >>>>> with ConsumerGroupMetadata or use the old one.
> > > >>>>>
> > > >>>>> If I use the new API with a 2.4.1 broker, I get
> > > >>>>>
> > > >>>>> UpsupportedVersionException: Attempted to write a non-default
> > > >>>> generationId
> > > >>>>> at version 2
> > > >>>>>
> > > >>>>> Alternatively, couldn't the client simply extract the groupId
> from
> > > the
> > > >>>>> ConsumerGroupMetadata and use the old struct if the broker is too
> > > old?
> > > >>>>>
> > > >>>>> I'd rather not have a user property in my framework to tell us
> > which
> > > >> API
> > > >>>> to
> > > >>>>> use.
> > > >>>>>
> > > >>>>> Thanks in advance.
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> > >
> >
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Ismael Juma <is...@juma.me.uk>.
The internal config was meant to be internal, right? That is, no
compatibility guarantees are offered? The current discussion implies we are
considering it a public config.
Ismael
On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen <bo...@confluent.io> wrote:
> For Gary's case, I think the internal config should be a sort of help, and
> not violating our public agreement.
>
> On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org> wrote:
>
> > I guess you would need to catch the exception and retry?
> >
> > It's a little unfortunate. Not sure if we could back-port the internal
> > producer config that we add in 2.6 for auto-downgrade to a 2.5 bug fix
> > release?
> >
> >
> > -Matthias
> >
> >
> > On 4/2/20 7:25 PM, Gary Russell wrote:
> > > Thanks Mattias
> > >
> > >> Hence, why do you want/need to switch to the newer overload that only
> > > works for 2.5+ brokers?
> > >
> > > So I can choose to use the producer per consumer thread Vs. the
> producer
> > > per group/topic/partition threading model for zombie fencing, based on
> > the
> > > broker version.
> > >
> > > I don't have the same luxury as kafka streams (i.e. don't use streams
> 2.6
> > > unless you have 2.5+ brokers).
> > >
> > > I add new features with each minor release (and try to use the latest
> > > kafka-clients as they become available).
> > >
> > > Users may want other new features, not related to EOS, and they might
> > stay
> > > on old brokers.
> > >
> > > Other users might want to take advantage of the improved performance of
> > the
> > > new EOS so I need to support both APIs.
> > >
> > > Many enterprises take forever to upgrade their brokers. I recently had
> a
> > > question of why my latest version won't work with an 0.9.x.x broker
> > (sigh).
> > >
> > > Spring versioning rules don't allow me to bump kafka-clients versions
> in
> > a
> > > patch release so I am already supporting 4 active branches and I am
> > trying
> > > to avoid supporting a fifth.
> > >
> > > Thanks again.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> > >
> > >> Gary,
> > >>
> > >> thanks for the question. We recently had a discussion about the exact
> > >> some topic:
> > >>
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
> > >>
> > >> Note that the "old" `sendOffsetsToTransaction(..., String groupId)` is
> > >> not deprecated. Hence, why do you want/need to switch to the newer
> > >> overload that only works for 2.5+ brokers? For many/most cases, the
> > >> "old" API that is compatible with older broker still does what you
> need
> > >> and there in not need to switch to the newer API.
> > >>
> > >>
> > >> -Matthias
> > >>
> > >>
> > >> On 4/2/20 1:54 PM, Gary Russell wrote:
> > >>> Thanks, Boyang,
> > >>>
> > >>> I maintain a framework (Spring for Apache Kafka) that sits on top of
> > the
> > >>> clients, and I would like to be able to support all broker versions.
> I
> > >>> don't have control over what brokers my users are using.
> > >>>
> > >>> You guys have done a great job since 0.10.2.0 (I think) of supporting
> > >> older
> > >>> brokers from newer clients but this one's a blocker for me.
> > >>>
> > >>> My framework will enforce the proper semantics for EOS, depending on
> > the
> > >>> broker version, but I need to know which model to use at runtime.
> > >>>
> > >>> As I said, I can have a property that the user can set to tell the
> > >>> framework that the broker is >= 2.5 but it would be cleaner if I
> could
> > >> stay
> > >>> away from that.
> > >>>
> > >>> Something like KafkaAdminClient.brokerApi() (or add the lowest
> > API/broker
> > >>> version to describeCluster()), would be helpful.
> > >>>
> > >>> Worst case, I'll add a configuration option.
> > >>>
> > >>> Thanks.
> > >>>
> > >>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <
> reluctanthero104@gmail.com
> > >
> > >>> wrote:
> > >>>
> > >>>> Thanks for the question Gary. The reasoning for crash the new
> > >>>> sendTxnOffsets API is because we don't want users to unconsciously
> > >> violate
> > >>>> the EOS guarantee. In your case, using this API with 2.4.1 is not
> > >> supported
> > >>>> anyway, so the upgrade path has to start from broker first to 2.5,
> and
> > >> then
> > >>>> client binaries. Is there any further concern that blocks you from
> > >> getting
> > >>>> the broker side upgrade first before using the new API?
> > >>>>
> > >>>> Boyang
> > >>>>
> > >>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gr...@pivotal.io>
> > >> wrote:
> > >>>>
> > >>>>> Is there any way to determine the broker version in the
> > kafka-clients?
> > >>>>>
> > >>>>> I need to determine whether I can use the new
> > sendOffsetsToTransaction
> > >>>>> with ConsumerGroupMetadata or use the old one.
> > >>>>>
> > >>>>> If I use the new API with a 2.4.1 broker, I get
> > >>>>>
> > >>>>> UpsupportedVersionException: Attempted to write a non-default
> > >>>> generationId
> > >>>>> at version 2
> > >>>>>
> > >>>>> Alternatively, couldn't the client simply extract the groupId from
> > the
> > >>>>> ConsumerGroupMetadata and use the old struct if the broker is too
> > old?
> > >>>>>
> > >>>>> I'd rather not have a user property in my framework to tell us
> which
> > >> API
> > >>>> to
> > >>>>> use.
> > >>>>>
> > >>>>> Thanks in advance.
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >
> >
> >
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Boyang Chen <bo...@confluent.io>.
For Gary's case, I think the internal config should be a sort of help, and
not violating our public agreement.
On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax <mj...@apache.org> wrote:
> I guess you would need to catch the exception and retry?
>
> It's a little unfortunate. Not sure if we could back-port the internal
> producer config that we add in 2.6 for auto-downgrade to a 2.5 bug fix
> release?
>
>
> -Matthias
>
>
> On 4/2/20 7:25 PM, Gary Russell wrote:
> > Thanks Mattias
> >
> >> Hence, why do you want/need to switch to the newer overload that only
> > works for 2.5+ brokers?
> >
> > So I can choose to use the producer per consumer thread Vs. the producer
> > per group/topic/partition threading model for zombie fencing, based on
> the
> > broker version.
> >
> > I don't have the same luxury as kafka streams (i.e. don't use streams 2.6
> > unless you have 2.5+ brokers).
> >
> > I add new features with each minor release (and try to use the latest
> > kafka-clients as they become available).
> >
> > Users may want other new features, not related to EOS, and they might
> stay
> > on old brokers.
> >
> > Other users might want to take advantage of the improved performance of
> the
> > new EOS so I need to support both APIs.
> >
> > Many enterprises take forever to upgrade their brokers. I recently had a
> > question of why my latest version won't work with an 0.9.x.x broker
> (sigh).
> >
> > Spring versioning rules don't allow me to bump kafka-clients versions in
> a
> > patch release so I am already supporting 4 active branches and I am
> trying
> > to avoid supporting a fifth.
> >
> > Thanks again.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org> wrote:
> >
> >> Gary,
> >>
> >> thanks for the question. We recently had a discussion about the exact
> >> some topic:
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
> >>
> >> Note that the "old" `sendOffsetsToTransaction(..., String groupId)` is
> >> not deprecated. Hence, why do you want/need to switch to the newer
> >> overload that only works for 2.5+ brokers? For many/most cases, the
> >> "old" API that is compatible with older broker still does what you need
> >> and there in not need to switch to the newer API.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 4/2/20 1:54 PM, Gary Russell wrote:
> >>> Thanks, Boyang,
> >>>
> >>> I maintain a framework (Spring for Apache Kafka) that sits on top of
> the
> >>> clients, and I would like to be able to support all broker versions. I
> >>> don't have control over what brokers my users are using.
> >>>
> >>> You guys have done a great job since 0.10.2.0 (I think) of supporting
> >> older
> >>> brokers from newer clients but this one's a blocker for me.
> >>>
> >>> My framework will enforce the proper semantics for EOS, depending on
> the
> >>> broker version, but I need to know which model to use at runtime.
> >>>
> >>> As I said, I can have a property that the user can set to tell the
> >>> framework that the broker is >= 2.5 but it would be cleaner if I could
> >> stay
> >>> away from that.
> >>>
> >>> Something like KafkaAdminClient.brokerApi() (or add the lowest
> API/broker
> >>> version to describeCluster()), would be helpful.
> >>>
> >>> Worst case, I'll add a configuration option.
> >>>
> >>> Thanks.
> >>>
> >>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <reluctanthero104@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Thanks for the question Gary. The reasoning for crash the new
> >>>> sendTxnOffsets API is because we don't want users to unconsciously
> >> violate
> >>>> the EOS guarantee. In your case, using this API with 2.4.1 is not
> >> supported
> >>>> anyway, so the upgrade path has to start from broker first to 2.5, and
> >> then
> >>>> client binaries. Is there any further concern that blocks you from
> >> getting
> >>>> the broker side upgrade first before using the new API?
> >>>>
> >>>> Boyang
> >>>>
> >>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gr...@pivotal.io>
> >> wrote:
> >>>>
> >>>>> Is there any way to determine the broker version in the
> kafka-clients?
> >>>>>
> >>>>> I need to determine whether I can use the new
> sendOffsetsToTransaction
> >>>>> with ConsumerGroupMetadata or use the old one.
> >>>>>
> >>>>> If I use the new API with a 2.4.1 broker, I get
> >>>>>
> >>>>> UpsupportedVersionException: Attempted to write a non-default
> >>>> generationId
> >>>>> at version 2
> >>>>>
> >>>>> Alternatively, couldn't the client simply extract the groupId from
> the
> >>>>> ConsumerGroupMetadata and use the old struct if the broker is too
> old?
> >>>>>
> >>>>> I'd rather not have a user property in my framework to tell us which
> >> API
> >>>> to
> >>>>> use.
> >>>>>
> >>>>> Thanks in advance.
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards
Compatibility
Posted by "Matthias J. Sax" <mj...@apache.org>.
I guess you would need to catch the exception and retry?
It's a little unfortunate. Not sure if we could back-port the internal
producer config that we add in 2.6 for auto-downgrade to a 2.5 bug fix
release?
-Matthias
On 4/2/20 7:25 PM, Gary Russell wrote:
> Thanks Mattias
>
>> Hence, why do you want/need to switch to the newer overload that only
> works for 2.5+ brokers?
>
> So I can choose to use the producer per consumer thread Vs. the producer
> per group/topic/partition threading model for zombie fencing, based on the
> broker version.
>
> I don't have the same luxury as kafka streams (i.e. don't use streams 2.6
> unless you have 2.5+ brokers).
>
> I add new features with each minor release (and try to use the latest
> kafka-clients as they become available).
>
> Users may want other new features, not related to EOS, and they might stay
> on old brokers.
>
> Other users might want to take advantage of the improved performance of the
> new EOS so I need to support both APIs.
>
> Many enterprises take forever to upgrade their brokers. I recently had a
> question of why my latest version won't work with an 0.9.x.x broker (sigh).
>
> Spring versioning rules don't allow me to bump kafka-clients versions in a
> patch release so I am already supporting 4 active branches and I am trying
> to avoid supporting a fifth.
>
> Thanks again.
>
>
>
>
>
>
>
>
>
> On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org> wrote:
>
>> Gary,
>>
>> thanks for the question. We recently had a discussion about the exact
>> some topic:
>>
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
>>
>> Note that the "old" `sendOffsetsToTransaction(..., String groupId)` is
>> not deprecated. Hence, why do you want/need to switch to the newer
>> overload that only works for 2.5+ brokers? For many/most cases, the
>> "old" API that is compatible with older broker still does what you need
>> and there in not need to switch to the newer API.
>>
>>
>> -Matthias
>>
>>
>> On 4/2/20 1:54 PM, Gary Russell wrote:
>>> Thanks, Boyang,
>>>
>>> I maintain a framework (Spring for Apache Kafka) that sits on top of the
>>> clients, and I would like to be able to support all broker versions. I
>>> don't have control over what brokers my users are using.
>>>
>>> You guys have done a great job since 0.10.2.0 (I think) of supporting
>> older
>>> brokers from newer clients but this one's a blocker for me.
>>>
>>> My framework will enforce the proper semantics for EOS, depending on the
>>> broker version, but I need to know which model to use at runtime.
>>>
>>> As I said, I can have a property that the user can set to tell the
>>> framework that the broker is >= 2.5 but it would be cleaner if I could
>> stay
>>> away from that.
>>>
>>> Something like KafkaAdminClient.brokerApi() (or add the lowest API/broker
>>> version to describeCluster()), would be helpful.
>>>
>>> Worst case, I'll add a configuration option.
>>>
>>> Thanks.
>>>
>>> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <re...@gmail.com>
>>> wrote:
>>>
>>>> Thanks for the question Gary. The reasoning for crash the new
>>>> sendTxnOffsets API is because we don't want users to unconsciously
>> violate
>>>> the EOS guarantee. In your case, using this API with 2.4.1 is not
>> supported
>>>> anyway, so the upgrade path has to start from broker first to 2.5, and
>> then
>>>> client binaries. Is there any further concern that blocks you from
>> getting
>>>> the broker side upgrade first before using the new API?
>>>>
>>>> Boyang
>>>>
>>>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gr...@pivotal.io>
>> wrote:
>>>>
>>>>> Is there any way to determine the broker version in the kafka-clients?
>>>>>
>>>>> I need to determine whether I can use the new sendOffsetsToTransaction
>>>>> with ConsumerGroupMetadata or use the old one.
>>>>>
>>>>> If I use the new API with a 2.4.1 broker, I get
>>>>>
>>>>> UpsupportedVersionException: Attempted to write a non-default
>>>> generationId
>>>>> at version 2
>>>>>
>>>>> Alternatively, couldn't the client simply extract the groupId from the
>>>>> ConsumerGroupMetadata and use the old struct if the broker is too old?
>>>>>
>>>>> I'd rather not have a user property in my framework to tell us which
>> API
>>>> to
>>>>> use.
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>
>>>
>>
>>
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Gary Russell <gr...@pivotal.io>.
Thanks Mattias
>Hence, why do you want/need to switch to the newer overload that only
works for 2.5+ brokers?
So I can choose to use the producer per consumer thread Vs. the producer
per group/topic/partition threading model for zombie fencing, based on the
broker version.
I don't have the same luxury as kafka streams (i.e. don't use streams 2.6
unless you have 2.5+ brokers).
I add new features with each minor release (and try to use the latest
kafka-clients as they become available).
Users may want other new features, not related to EOS, and they might stay
on old brokers.
Other users might want to take advantage of the improved performance of the
new EOS so I need to support both APIs.
Many enterprises take forever to upgrade their brokers. I recently had a
question of why my latest version won't work with an 0.9.x.x broker (sigh).
Spring versioning rules don't allow me to bump kafka-clients versions in a
patch release so I am already supporting 4 active branches and I am trying
to avoid supporting a fifth.
Thanks again.
On Thu, Apr 2, 2020 at 8:23 PM Matthias J. Sax <mj...@apache.org> wrote:
> Gary,
>
> thanks for the question. We recently had a discussion about the exact
> some topic:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
>
> Note that the "old" `sendOffsetsToTransaction(..., String groupId)` is
> not deprecated. Hence, why do you want/need to switch to the newer
> overload that only works for 2.5+ brokers? For many/most cases, the
> "old" API that is compatible with older broker still does what you need
> and there in not need to switch to the newer API.
>
>
> -Matthias
>
>
> On 4/2/20 1:54 PM, Gary Russell wrote:
> > Thanks, Boyang,
> >
> > I maintain a framework (Spring for Apache Kafka) that sits on top of the
> > clients, and I would like to be able to support all broker versions. I
> > don't have control over what brokers my users are using.
> >
> > You guys have done a great job since 0.10.2.0 (I think) of supporting
> older
> > brokers from newer clients but this one's a blocker for me.
> >
> > My framework will enforce the proper semantics for EOS, depending on the
> > broker version, but I need to know which model to use at runtime.
> >
> > As I said, I can have a property that the user can set to tell the
> > framework that the broker is >= 2.5 but it would be cleaner if I could
> stay
> > away from that.
> >
> > Something like KafkaAdminClient.brokerApi() (or add the lowest API/broker
> > version to describeCluster()), would be helpful.
> >
> > Worst case, I'll add a configuration option.
> >
> > Thanks.
> >
> > On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <re...@gmail.com>
> > wrote:
> >
> >> Thanks for the question Gary. The reasoning for crash the new
> >> sendTxnOffsets API is because we don't want users to unconsciously
> violate
> >> the EOS guarantee. In your case, using this API with 2.4.1 is not
> supported
> >> anyway, so the upgrade path has to start from broker first to 2.5, and
> then
> >> client binaries. Is there any further concern that blocks you from
> getting
> >> the broker side upgrade first before using the new API?
> >>
> >> Boyang
> >>
> >> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gr...@pivotal.io>
> wrote:
> >>
> >>> Is there any way to determine the broker version in the kafka-clients?
> >>>
> >>> I need to determine whether I can use the new sendOffsetsToTransaction
> >>> with ConsumerGroupMetadata or use the old one.
> >>>
> >>> If I use the new API with a 2.4.1 broker, I get
> >>>
> >>> UpsupportedVersionException: Attempted to write a non-default
> >> generationId
> >>> at version 2
> >>>
> >>> Alternatively, couldn't the client simply extract the groupId from the
> >>> ConsumerGroupMetadata and use the old struct if the broker is too old?
> >>>
> >>> I'd rather not have a user property in my framework to tell us which
> API
> >> to
> >>> use.
> >>>
> >>> Thanks in advance.
> >>>
> >>
> >
>
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards
Compatibility
Posted by "Matthias J. Sax" <mj...@apache.org>.
Gary,
thanks for the question. We recently had a discussion about the exact
some topic:
http://mail-archives.apache.org/mod_mbox/kafka-dev/202003.mbox/%3CCAJKanumaUg7bcRr%3DoZqQq9aWuO%3DfA5U1uvxAciB6RbYsvsEbYQ%40mail.gmail.com%3E
Note that the "old" `sendOffsetsToTransaction(..., String groupId)` is
not deprecated. Hence, why do you want/need to switch to the newer
overload that only works for 2.5+ brokers? For many/most cases, the
"old" API that is compatible with older broker still does what you need
and there in not need to switch to the newer API.
-Matthias
On 4/2/20 1:54 PM, Gary Russell wrote:
> Thanks, Boyang,
>
> I maintain a framework (Spring for Apache Kafka) that sits on top of the
> clients, and I would like to be able to support all broker versions. I
> don't have control over what brokers my users are using.
>
> You guys have done a great job since 0.10.2.0 (I think) of supporting older
> brokers from newer clients but this one's a blocker for me.
>
> My framework will enforce the proper semantics for EOS, depending on the
> broker version, but I need to know which model to use at runtime.
>
> As I said, I can have a property that the user can set to tell the
> framework that the broker is >= 2.5 but it would be cleaner if I could stay
> away from that.
>
> Something like KafkaAdminClient.brokerApi() (or add the lowest API/broker
> version to describeCluster()), would be helpful.
>
> Worst case, I'll add a configuration option.
>
> Thanks.
>
> On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <re...@gmail.com>
> wrote:
>
>> Thanks for the question Gary. The reasoning for crash the new
>> sendTxnOffsets API is because we don't want users to unconsciously violate
>> the EOS guarantee. In your case, using this API with 2.4.1 is not supported
>> anyway, so the upgrade path has to start from broker first to 2.5, and then
>> client binaries. Is there any further concern that blocks you from getting
>> the broker side upgrade first before using the new API?
>>
>> Boyang
>>
>> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gr...@pivotal.io> wrote:
>>
>>> Is there any way to determine the broker version in the kafka-clients?
>>>
>>> I need to determine whether I can use the new sendOffsetsToTransaction
>>> with ConsumerGroupMetadata or use the old one.
>>>
>>> If I use the new API with a 2.4.1 broker, I get
>>>
>>> UpsupportedVersionException: Attempted to write a non-default
>> generationId
>>> at version 2
>>>
>>> Alternatively, couldn't the client simply extract the groupId from the
>>> ConsumerGroupMetadata and use the old struct if the broker is too old?
>>>
>>> I'd rather not have a user property in my framework to tell us which API
>> to
>>> use.
>>>
>>> Thanks in advance.
>>>
>>
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Gary Russell <gr...@pivotal.io>.
Thanks, Boyang,
I maintain a framework (Spring for Apache Kafka) that sits on top of the
clients, and I would like to be able to support all broker versions. I
don't have control over what brokers my users are using.
You guys have done a great job since 0.10.2.0 (I think) of supporting older
brokers from newer clients but this one's a blocker for me.
My framework will enforce the proper semantics for EOS, depending on the
broker version, but I need to know which model to use at runtime.
As I said, I can have a property that the user can set to tell the
framework that the broker is >= 2.5 but it would be cleaner if I could stay
away from that.
Something like KafkaAdminClient.brokerApi() (or add the lowest API/broker
version to describeCluster()), would be helpful.
Worst case, I'll add a configuration option.
Thanks.
On Thu, Apr 2, 2020 at 4:45 PM Boyang Chen <re...@gmail.com>
wrote:
> Thanks for the question Gary. The reasoning for crash the new
> sendTxnOffsets API is because we don't want users to unconsciously violate
> the EOS guarantee. In your case, using this API with 2.4.1 is not supported
> anyway, so the upgrade path has to start from broker first to 2.5, and then
> client binaries. Is there any further concern that blocks you from getting
> the broker side upgrade first before using the new API?
>
> Boyang
>
> On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gr...@pivotal.io> wrote:
>
> > Is there any way to determine the broker version in the kafka-clients?
> >
> > I need to determine whether I can use the new sendOffsetsToTransaction
> > with ConsumerGroupMetadata or use the old one.
> >
> > If I use the new API with a 2.4.1 broker, I get
> >
> > UpsupportedVersionException: Attempted to write a non-default
> generationId
> > at version 2
> >
> > Alternatively, couldn't the client simply extract the groupId from the
> > ConsumerGroupMetadata and use the old struct if the broker is too old?
> >
> > I'd rather not have a user property in my framework to tell us which API
> to
> > use.
> >
> > Thanks in advance.
> >
>
Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility
Posted by Boyang Chen <re...@gmail.com>.
Thanks for the question Gary. The reasoning for crash the new
sendTxnOffsets API is because we don't want users to unconsciously violate
the EOS guarantee. In your case, using this API with 2.4.1 is not supported
anyway, so the upgrade path has to start from broker first to 2.5, and then
client binaries. Is there any further concern that blocks you from getting
the broker side upgrade first before using the new API?
Boyang
On Thu, Apr 2, 2020 at 1:37 PM Gary Russell <gr...@pivotal.io> wrote:
> Is there any way to determine the broker version in the kafka-clients?
>
> I need to determine whether I can use the new sendOffsetsToTransaction
> with ConsumerGroupMetadata or use the old one.
>
> If I use the new API with a 2.4.1 broker, I get
>
> UpsupportedVersionException: Attempted to write a non-default generationId
> at version 2
>
> Alternatively, couldn't the client simply extract the groupId from the
> ConsumerGroupMetadata and use the old struct if the broker is too old?
>
> I'd rather not have a user property in my framework to tell us which API to
> use.
>
> Thanks in advance.
>