You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jonathan Albrecht <Jo...@ibm.com> on 2022/02/03 14:15:43 UTC
Kafka <= 3.1 upgrade RocksDB to v6.27.3?
The rocksdbjni dependency has been upgraded to v6.27.3 on trunk and I
wanted to ask if it would be ok to also upgrade it to v6.27.3 on the 3.1
branch (and possibly earlier branches). I thought I should ask in case
there are some policies around changing dependency versions in point
releases.
The motivation is that this is the first version of rocksdbjni that
supports s390x and it allows kafka to be built out of the box on this
platform. Having this support in earlier releases helps users on s390x that
may need a specific minor release.
If upgrading earlier releases is ok, how far back would be reasonable? I'm
happy to create the issues and PRs and do the local testing, of course.
Thanks,
Jonathan Albrecht
Advisory Software Developer
Linux on IBM Z Open Source Ecosystem
1 905 413 3577 Office
jonathan.albrecht@ibm.com
IBM
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Guozhang Wang <wa...@gmail.com>.
We have a 4-month major release cadence in Apache Kafka, and given that
3.1.0 is just released, a tentative estimate for 3.2.0 would be around
early June.
Guozhang
On Tue, Feb 8, 2022 at 8:19 AM Jonathan Albrecht <Jo...@ibm.com>
wrote:
>
>
>
> No problem, glad that was more clear and thanks for considering it. I
> totally
> understand if this is not the type of change to put into a point release.
>
> I'd like to understand the release schedule a bit more. Is 3.2 probably
> coming
> out in May?
>
> Thanks,
>
> Jon
>
> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-08 04:21:50 AM:
>
> > From: "Bruno Cadonna" <ca...@apache.org>
> > To: dev@kafka.apache.org
> > Date: 2022-02-08 04:22 AM
> > Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> >
> > Thank you Jonathan! Now I understand the situation better.
> >
> > I agree with Guozhang about the waiting times for 3.2.0, 3.0.1, and
> 3.1.1.
> >
> > Obviously that does not satisfy the organizational requirements you
> > mentioned.
> >
> > Best,
> > Bruno
> >
> > On 07.02.22 23:24, Guozhang Wang wrote:
> > > Thanks for the clarification Jonathan, I think timing wise, waiting to
> get
> > > 3.2.0 v.s. get 3.1.1 / 3.0.1 would not be much longer.
> > >
> > > On Mon, Feb 7, 2022 at 12:16 PM Jonathan Albrecht
> <Jo...@ibm.com>
> > > wrote:
> > >
> > >>
> > >>
> > >>
> > >> Hi Bruno and Guozhang,
> > >>
> > >> Thanks for your thoughtful replies and I hope I can clarify. I work on
> > >> a team that helps port open source software to s390x. The example I
> used
> > >> about users needing to be on a specific minor release is a general one
> > >> that I see and not specific to kafka and usually its not due to
> technical
> > >> reasons. For example, sometimes organizations want to support the same
> > >> version of a component across platforms and back porting support for
> > >> s390x mean they can get to a supported version sooner. Maybe in
> kafka's
> > >> case, there are not technical reasons for that to happen but sometimes
> > >> there are organizational reasons.
> > >>
> > >> Hope that clarifies where I'm coming from. I definitely don't have a
> > >> specific technical issue that would be solved by back porting. I
> > >> understand there are risks and it might not be appropriate for a
> bugfix
> > >> release which is why I wanted to ask first before going any further.
> > >>
> > >> Thanks,
> > >>
> > >> Jon
> > >>
> > >> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-07 05:30:05 AM:
> > >>
> > >>> From: "Bruno Cadonna" <ca...@apache.org>
> > >>> To: dev@kafka.apache.org
> > >>> Date: 2022-02-07 05:30 AM
> > >>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> > >>>
> > >>> Hi Jonathan and Guozhang,
> > >>>
> > >>> I am pretty sure that upgrading RocksDB in 3.0.x and 3.1.x should not
> be
> > >>> an issue, since we recently upgraded it to 6.27.3 on trunk and
> judging
> > >>> from the compatibility report in the ticket [1] and the code the API
> > >>> does not break backward-compatibility for the AK 3.x series.
> > >>>
> > >>> I am wondering, why users would not be willing to use the upcoming
> 3.2.0
> > >>> if they are migrating or creating a Kafka Streams application on the
> > >>> s390x platform. Even if we ported the upgrade to 3.1.1 and 3.0.1,
> they
> > >>> would need to wait until those versions are released. Additionally,
> > >>> their applications need to comply with 3.x APIs and APIs should be
> > >>> backward compatible in the 3.x releases. So, everything that works
> with
> > >>> 3.1.1 and 3.0.1 should also work with 3.2.0. Of course there could be
> an
> > >>> issue in 3.2.0 that will force them to use a 3.1.1 or 3.0.1, but that
> > >>> would be rather an exception that we can then fix when the exception
> > >>> happens.
> > >>>
> > >>> Jonathan, could you clarify what the motivation of using hypothetical
> > >>> 3.1.1 or 3.0.1 with the RocksDB upgrade instead of 3.2.0 on a
> platform
> > >>> that was not supported before (i.e., 3.1.0 and 3.0.0) might be?
> > >>>
> > >>> In the end, it is always a risk to upgrade a library in a bugfix
> release
> > >>> without a critical issue that the upgrade fixes.
> > >>>
> > >>>
> > >>> Best,
> > >>> Bruno
> > >>>
> > >>>
> > >>> [1]
> > >>
> https://issues.apache.org/jira/browse/KAFKA-13599
>
> > >>
>
--
-- Guozhang
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Jonathan Albrecht <Jo...@ibm.com>.
No problem, glad that was more clear and thanks for considering it. I
totally
understand if this is not the type of change to put into a point release.
I'd like to understand the release schedule a bit more. Is 3.2 probably
coming
out in May?
Thanks,
Jon
"Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-08 04:21:50 AM:
> From: "Bruno Cadonna" <ca...@apache.org>
> To: dev@kafka.apache.org
> Date: 2022-02-08 04:22 AM
> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>
> Thank you Jonathan! Now I understand the situation better.
>
> I agree with Guozhang about the waiting times for 3.2.0, 3.0.1, and
3.1.1.
>
> Obviously that does not satisfy the organizational requirements you
> mentioned.
>
> Best,
> Bruno
>
> On 07.02.22 23:24, Guozhang Wang wrote:
> > Thanks for the clarification Jonathan, I think timing wise, waiting to
get
> > 3.2.0 v.s. get 3.1.1 / 3.0.1 would not be much longer.
> >
> > On Mon, Feb 7, 2022 at 12:16 PM Jonathan Albrecht
<Jo...@ibm.com>
> > wrote:
> >
> >>
> >>
> >>
> >> Hi Bruno and Guozhang,
> >>
> >> Thanks for your thoughtful replies and I hope I can clarify. I work on
> >> a team that helps port open source software to s390x. The example I
used
> >> about users needing to be on a specific minor release is a general one
> >> that I see and not specific to kafka and usually its not due to
technical
> >> reasons. For example, sometimes organizations want to support the same
> >> version of a component across platforms and back porting support for
> >> s390x mean they can get to a supported version sooner. Maybe in
kafka's
> >> case, there are not technical reasons for that to happen but sometimes
> >> there are organizational reasons.
> >>
> >> Hope that clarifies where I'm coming from. I definitely don't have a
> >> specific technical issue that would be solved by back porting. I
> >> understand there are risks and it might not be appropriate for a
bugfix
> >> release which is why I wanted to ask first before going any further.
> >>
> >> Thanks,
> >>
> >> Jon
> >>
> >> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-07 05:30:05 AM:
> >>
> >>> From: "Bruno Cadonna" <ca...@apache.org>
> >>> To: dev@kafka.apache.org
> >>> Date: 2022-02-07 05:30 AM
> >>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> >>>
> >>> Hi Jonathan and Guozhang,
> >>>
> >>> I am pretty sure that upgrading RocksDB in 3.0.x and 3.1.x should not
be
> >>> an issue, since we recently upgraded it to 6.27.3 on trunk and
judging
> >>> from the compatibility report in the ticket [1] and the code the API
> >>> does not break backward-compatibility for the AK 3.x series.
> >>>
> >>> I am wondering, why users would not be willing to use the upcoming
3.2.0
> >>> if they are migrating or creating a Kafka Streams application on the
> >>> s390x platform. Even if we ported the upgrade to 3.1.1 and 3.0.1,
they
> >>> would need to wait until those versions are released. Additionally,
> >>> their applications need to comply with 3.x APIs and APIs should be
> >>> backward compatible in the 3.x releases. So, everything that works
with
> >>> 3.1.1 and 3.0.1 should also work with 3.2.0. Of course there could be
an
> >>> issue in 3.2.0 that will force them to use a 3.1.1 or 3.0.1, but that
> >>> would be rather an exception that we can then fix when the exception
> >>> happens.
> >>>
> >>> Jonathan, could you clarify what the motivation of using hypothetical
> >>> 3.1.1 or 3.0.1 with the RocksDB upgrade instead of 3.2.0 on a
platform
> >>> that was not supported before (i.e., 3.1.0 and 3.0.0) might be?
> >>>
> >>> In the end, it is always a risk to upgrade a library in a bugfix
release
> >>> without a critical issue that the upgrade fixes.
> >>>
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>>
> >>> [1]
> >>
https://issues.apache.org/jira/browse/KAFKA-13599
> >>
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Bruno Cadonna <ca...@apache.org>.
Thank you Jonathan! Now I understand the situation better.
I agree with Guozhang about the waiting times for 3.2.0, 3.0.1, and 3.1.1.
Obviously that does not satisfy the organizational requirements you
mentioned.
Best,
Bruno
On 07.02.22 23:24, Guozhang Wang wrote:
> Thanks for the clarification Jonathan, I think timing wise, waiting to get
> 3.2.0 v.s. get 3.1.1 / 3.0.1 would not be much longer.
>
> On Mon, Feb 7, 2022 at 12:16 PM Jonathan Albrecht <Jo...@ibm.com>
> wrote:
>
>>
>>
>>
>> Hi Bruno and Guozhang,
>>
>> Thanks for your thoughtful replies and I hope I can clarify. I work on
>> a team that helps port open source software to s390x. The example I used
>> about users needing to be on a specific minor release is a general one
>> that I see and not specific to kafka and usually its not due to technical
>> reasons. For example, sometimes organizations want to support the same
>> version of a component across platforms and back porting support for
>> s390x mean they can get to a supported version sooner. Maybe in kafka's
>> case, there are not technical reasons for that to happen but sometimes
>> there are organizational reasons.
>>
>> Hope that clarifies where I'm coming from. I definitely don't have a
>> specific technical issue that would be solved by back porting. I
>> understand there are risks and it might not be appropriate for a bugfix
>> release which is why I wanted to ask first before going any further.
>>
>> Thanks,
>>
>> Jon
>>
>> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-07 05:30:05 AM:
>>
>>> From: "Bruno Cadonna" <ca...@apache.org>
>>> To: dev@kafka.apache.org
>>> Date: 2022-02-07 05:30 AM
>>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>>>
>>> Hi Jonathan and Guozhang,
>>>
>>> I am pretty sure that upgrading RocksDB in 3.0.x and 3.1.x should not be
>>> an issue, since we recently upgraded it to 6.27.3 on trunk and judging
>>> from the compatibility report in the ticket [1] and the code the API
>>> does not break backward-compatibility for the AK 3.x series.
>>>
>>> I am wondering, why users would not be willing to use the upcoming 3.2.0
>>> if they are migrating or creating a Kafka Streams application on the
>>> s390x platform. Even if we ported the upgrade to 3.1.1 and 3.0.1, they
>>> would need to wait until those versions are released. Additionally,
>>> their applications need to comply with 3.x APIs and APIs should be
>>> backward compatible in the 3.x releases. So, everything that works with
>>> 3.1.1 and 3.0.1 should also work with 3.2.0. Of course there could be an
>>> issue in 3.2.0 that will force them to use a 3.1.1 or 3.0.1, but that
>>> would be rather an exception that we can then fix when the exception
>>> happens.
>>>
>>> Jonathan, could you clarify what the motivation of using hypothetical
>>> 3.1.1 or 3.0.1 with the RocksDB upgrade instead of 3.2.0 on a platform
>>> that was not supported before (i.e., 3.1.0 and 3.0.0) might be?
>>>
>>> In the end, it is always a risk to upgrade a library in a bugfix release
>>> without a critical issue that the upgrade fixes.
>>>
>>>
>>> Best,
>>> Bruno
>>>
>>>
>>> [1]
>> https://issues.apache.org/jira/browse/KAFKA-13599
>>
>>>
>>> On 07.02.22 04:19, Guozhang Wang wrote:
>>>> Hi Jonathan,
>>>>
>>>> I'm not against the idea of upgrading in 3.0.x and 3.1.x, assuming that
>> the
>>>> v6.27.3 version does not make any API or any semantic behavioral
>> changes.
>>>> But I can only speak for myself, not the whole community. For older
>>>> versions as Bruno mentioned since there's compatibility issues we
>> cannot
>>>> upgrade RocksDB any more.
>>>>
>>>>
>>>> Guozhang
>>>>
>>>> On Thu, Feb 3, 2022 at 1:56 PM Jonathan Albrecht
>> <Jo...@ibm.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> Thanks Guozhang, yes the motivation is to support the s390x platform.
>> It's
>>>>> not a critical bug for other platforms.
>>>>>
>>>>> Any chance that gaining platform support is also a valid reason? I was
>>>>> hoping it would be but I won't submit a PR if it isn't.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jon
>>>>>
>>>>> "Guozhang Wang" <wa...@gmail.com> wrote on 2022-02-03 02:14:34 PM:
>>>>>
>>>>>> From: "Guozhang Wang" <wa...@gmail.com>
>>>>>> To: "dev" <de...@kafka.apache.org>
>>>>>> Date: 2022-02-03 02:15 PM
>>>>>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>>>>>>
>>>>>> Hello Jonathan,
>>>>>>
>>>>>> I think Bruno's point is that we can only upgrade in the bugfix
>> releases if
>>>>>> the old version of rocksDB has a critical bug that the new version
>> would
>>>>>> fix. For 6.27.3 it seems not fixing a critical bug but for a new
>> feature,
>>>>>> and hence unless we change the policy we cannot upgrade in 3.0.1 /
>> 3.1.1
>>>>>> releases.
>>>>>>
>>>>>>
>>>>>> Guozhang
>>>>>>
>>>>>> On Thu, Feb 3, 2022 at 9:22 AM Jonathan Albrecht
>> <Jo...@ibm.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks for the info Bruno. In that case, if no other concerns, I'll
>> try
>>>>>>> updating RocksDB to v6.27.3 on the 3.0 and 3.1 branches and file
>> issues and
>>>>>>> PRs if everything looks good.
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>>
>>>>>>> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-03 10:40:00
>> AM:
>>>>>>>
>>>>>>>> From: "Bruno Cadonna" <ca...@apache.org>
>>>>>>>> To: dev@kafka.apache.org
>>>>>>>> Date: 2022-02-03 10:40 AM
>>>>>>>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>>>>>>>>
>>>>>>>> Hi Jonathan,
>>>>>>>>
>>>>>>>> We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to
>> source
>>>>>>>> compatibility issue. More specifically, we expose RocksDB APIs in
>> Kafka
>>>>>>>> Streams for configuring RocksDB and those RocksDB APIs changed. So
>>>>>>>> upgrading RocksDB was actually a compatibility breaking change. We
>> had
>>>>>>>> to wait for the major release 3.0.0 to make the upgrade. That
>> means, if
>>>>>>>> the policy allows to upgrade dependencies in bugfix releases we can
>> only
>>>>>>>> upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading
>> RocksDB in
>>>>>>>> earlier releases would break compatibility.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Bruno
>>>>>>>>
>>>>>>>> On 03.02.22 15:15, Jonathan Albrecht wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The rocksdbjni dependency has been upgraded to v6.27.3 on trunk
>> and
>>>>>>>>> wanted to ask if it would be ok to also upgrade it to v6.27.3 on
>> the 3.1
>>>>>>>>> branch (and possibly earlier branches). I thought I should ask in
>> case
>>>>>>>>> there are some policies around changing dependency versions in
>> point
>>>>>>>>> releases.
>>>>>>>>>
>>>>>>>>> The motivation is that this is the first version of rocksdbjni
>> that
>>>>>>>>> supports s390x and it allows kafka to be built out of the box on
>> this
>>>>>>>>> platform. Having this support in earlier releases helps users on
>> s390x
>>>>>>>>> that
>>>>>>>>> may need a specific minor release.
>>>>>>>>>
>>>>>>>>> If upgrading earlier releases is ok, how far back would be
>> reasonable?
>>>>>>>>> I'm
>>>>>>>>> happy to create the issues and PRs and do the local testing, of
>> course.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jonathan Albrecht
>>>>>>>>> Advisory Software Developer
>>>>>>>>> Linux on IBM Z Open Source Ecosystem
>>>>>>>>> 1 905 413 3577 Office
>>>>>>>>> jonathan.albrecht@ibm.com
>>>>>>>>>
>>>>>>>>> IBM
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -- Guozhang
>>>>>
>>>>
>>>>
>>
>
>
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Guozhang Wang <wa...@gmail.com>.
Thanks for the clarification Jonathan, I think timing wise, waiting to get
3.2.0 v.s. get 3.1.1 / 3.0.1 would not be much longer.
On Mon, Feb 7, 2022 at 12:16 PM Jonathan Albrecht <Jo...@ibm.com>
wrote:
>
>
>
> Hi Bruno and Guozhang,
>
> Thanks for your thoughtful replies and I hope I can clarify. I work on
> a team that helps port open source software to s390x. The example I used
> about users needing to be on a specific minor release is a general one
> that I see and not specific to kafka and usually its not due to technical
> reasons. For example, sometimes organizations want to support the same
> version of a component across platforms and back porting support for
> s390x mean they can get to a supported version sooner. Maybe in kafka's
> case, there are not technical reasons for that to happen but sometimes
> there are organizational reasons.
>
> Hope that clarifies where I'm coming from. I definitely don't have a
> specific technical issue that would be solved by back porting. I
> understand there are risks and it might not be appropriate for a bugfix
> release which is why I wanted to ask first before going any further.
>
> Thanks,
>
> Jon
>
> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-07 05:30:05 AM:
>
> > From: "Bruno Cadonna" <ca...@apache.org>
> > To: dev@kafka.apache.org
> > Date: 2022-02-07 05:30 AM
> > Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> >
> > Hi Jonathan and Guozhang,
> >
> > I am pretty sure that upgrading RocksDB in 3.0.x and 3.1.x should not be
> > an issue, since we recently upgraded it to 6.27.3 on trunk and judging
> > from the compatibility report in the ticket [1] and the code the API
> > does not break backward-compatibility for the AK 3.x series.
> >
> > I am wondering, why users would not be willing to use the upcoming 3.2.0
> > if they are migrating or creating a Kafka Streams application on the
> > s390x platform. Even if we ported the upgrade to 3.1.1 and 3.0.1, they
> > would need to wait until those versions are released. Additionally,
> > their applications need to comply with 3.x APIs and APIs should be
> > backward compatible in the 3.x releases. So, everything that works with
> > 3.1.1 and 3.0.1 should also work with 3.2.0. Of course there could be an
> > issue in 3.2.0 that will force them to use a 3.1.1 or 3.0.1, but that
> > would be rather an exception that we can then fix when the exception
> > happens.
> >
> > Jonathan, could you clarify what the motivation of using hypothetical
> > 3.1.1 or 3.0.1 with the RocksDB upgrade instead of 3.2.0 on a platform
> > that was not supported before (i.e., 3.1.0 and 3.0.0) might be?
> >
> > In the end, it is always a risk to upgrade a library in a bugfix release
> > without a critical issue that the upgrade fixes.
> >
> >
> > Best,
> > Bruno
> >
> >
> > [1]
> https://issues.apache.org/jira/browse/KAFKA-13599
>
> >
> > On 07.02.22 04:19, Guozhang Wang wrote:
> > > Hi Jonathan,
> > >
> > > I'm not against the idea of upgrading in 3.0.x and 3.1.x, assuming that
> the
> > > v6.27.3 version does not make any API or any semantic behavioral
> changes.
> > > But I can only speak for myself, not the whole community. For older
> > > versions as Bruno mentioned since there's compatibility issues we
> cannot
> > > upgrade RocksDB any more.
> > >
> > >
> > > Guozhang
> > >
> > > On Thu, Feb 3, 2022 at 1:56 PM Jonathan Albrecht
> <Jo...@ibm.com>
> > > wrote:
> > >
> > >>
> > >>
> > >> Thanks Guozhang, yes the motivation is to support the s390x platform.
> It's
> > >> not a critical bug for other platforms.
> > >>
> > >> Any chance that gaining platform support is also a valid reason? I was
> > >> hoping it would be but I won't submit a PR if it isn't.
> > >>
> > >> Thanks,
> > >>
> > >> Jon
> > >>
> > >> "Guozhang Wang" <wa...@gmail.com> wrote on 2022-02-03 02:14:34 PM:
> > >>
> > >>> From: "Guozhang Wang" <wa...@gmail.com>
> > >>> To: "dev" <de...@kafka.apache.org>
> > >>> Date: 2022-02-03 02:15 PM
> > >>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> > >>>
> > >>> Hello Jonathan,
> > >>>
> > >>> I think Bruno's point is that we can only upgrade in the bugfix
> releases if
> > >>> the old version of rocksDB has a critical bug that the new version
> would
> > >>> fix. For 6.27.3 it seems not fixing a critical bug but for a new
> feature,
> > >>> and hence unless we change the policy we cannot upgrade in 3.0.1 /
> 3.1.1
> > >>> releases.
> > >>>
> > >>>
> > >>> Guozhang
> > >>>
> > >>> On Thu, Feb 3, 2022 at 9:22 AM Jonathan Albrecht
> <Jo...@ibm.com> wrote:
> > >>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> Thanks for the info Bruno. In that case, if no other concerns, I'll
> try
> > >>>> updating RocksDB to v6.27.3 on the 3.0 and 3.1 branches and file
> issues and
> > >>>> PRs if everything looks good.
> > >>>>
> > >>>> Jon
> > >>>>
> > >>>>
> > >>>> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-03 10:40:00
> AM:
> > >>>>
> > >>>>> From: "Bruno Cadonna" <ca...@apache.org>
> > >>>>> To: dev@kafka.apache.org
> > >>>>> Date: 2022-02-03 10:40 AM
> > >>>>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> > >>>>>
> > >>>>> Hi Jonathan,
> > >>>>>
> > >>>>> We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to
> source
> > >>>>> compatibility issue. More specifically, we expose RocksDB APIs in
> Kafka
> > >>>>> Streams for configuring RocksDB and those RocksDB APIs changed. So
> > >>>>> upgrading RocksDB was actually a compatibility breaking change. We
> had
> > >>>>> to wait for the major release 3.0.0 to make the upgrade. That
> means, if
> > >>>>> the policy allows to upgrade dependencies in bugfix releases we can
> only
> > >>>>> upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading
> RocksDB in
> > >>>>> earlier releases would break compatibility.
> > >>>>>
> > >>>>> Best,
> > >>>>> Bruno
> > >>>>>
> > >>>>> On 03.02.22 15:15, Jonathan Albrecht wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>> The rocksdbjni dependency has been upgraded to v6.27.3 on trunk
> and
> > >>>>>> wanted to ask if it would be ok to also upgrade it to v6.27.3 on
> the 3.1
> > >>>>>> branch (and possibly earlier branches). I thought I should ask in
> case
> > >>>>>> there are some policies around changing dependency versions in
> point
> > >>>>>> releases.
> > >>>>>>
> > >>>>>> The motivation is that this is the first version of rocksdbjni
> that
> > >>>>>> supports s390x and it allows kafka to be built out of the box on
> this
> > >>>>>> platform. Having this support in earlier releases helps users on
> s390x
> > >>>>>> that
> > >>>>>> may need a specific minor release.
> > >>>>>>
> > >>>>>> If upgrading earlier releases is ok, how far back would be
> reasonable?
> > >>>>>> I'm
> > >>>>>> happy to create the issues and PRs and do the local testing, of
> course.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Jonathan Albrecht
> > >>>>>> Advisory Software Developer
> > >>>>>> Linux on IBM Z Open Source Ecosystem
> > >>>>>> 1 905 413 3577 Office
> > >>>>>> jonathan.albrecht@ibm.com
> > >>>>>>
> > >>>>>> IBM
> > >>>>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>
> > >
> > >
>
--
-- Guozhang
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Jonathan Albrecht <Jo...@ibm.com>.
Hi Bruno and Guozhang,
Thanks for your thoughtful replies and I hope I can clarify. I work on
a team that helps port open source software to s390x. The example I used
about users needing to be on a specific minor release is a general one
that I see and not specific to kafka and usually its not due to technical
reasons. For example, sometimes organizations want to support the same
version of a component across platforms and back porting support for
s390x mean they can get to a supported version sooner. Maybe in kafka's
case, there are not technical reasons for that to happen but sometimes
there are organizational reasons.
Hope that clarifies where I'm coming from. I definitely don't have a
specific technical issue that would be solved by back porting. I
understand there are risks and it might not be appropriate for a bugfix
release which is why I wanted to ask first before going any further.
Thanks,
Jon
"Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-07 05:30:05 AM:
> From: "Bruno Cadonna" <ca...@apache.org>
> To: dev@kafka.apache.org
> Date: 2022-02-07 05:30 AM
> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>
> Hi Jonathan and Guozhang,
>
> I am pretty sure that upgrading RocksDB in 3.0.x and 3.1.x should not be
> an issue, since we recently upgraded it to 6.27.3 on trunk and judging
> from the compatibility report in the ticket [1] and the code the API
> does not break backward-compatibility for the AK 3.x series.
>
> I am wondering, why users would not be willing to use the upcoming 3.2.0
> if they are migrating or creating a Kafka Streams application on the
> s390x platform. Even if we ported the upgrade to 3.1.1 and 3.0.1, they
> would need to wait until those versions are released. Additionally,
> their applications need to comply with 3.x APIs and APIs should be
> backward compatible in the 3.x releases. So, everything that works with
> 3.1.1 and 3.0.1 should also work with 3.2.0. Of course there could be an
> issue in 3.2.0 that will force them to use a 3.1.1 or 3.0.1, but that
> would be rather an exception that we can then fix when the exception
> happens.
>
> Jonathan, could you clarify what the motivation of using hypothetical
> 3.1.1 or 3.0.1 with the RocksDB upgrade instead of 3.2.0 on a platform
> that was not supported before (i.e., 3.1.0 and 3.0.0) might be?
>
> In the end, it is always a risk to upgrade a library in a bugfix release
> without a critical issue that the upgrade fixes.
>
>
> Best,
> Bruno
>
>
> [1]
https://issues.apache.org/jira/browse/KAFKA-13599
>
> On 07.02.22 04:19, Guozhang Wang wrote:
> > Hi Jonathan,
> >
> > I'm not against the idea of upgrading in 3.0.x and 3.1.x, assuming that
the
> > v6.27.3 version does not make any API or any semantic behavioral
changes.
> > But I can only speak for myself, not the whole community. For older
> > versions as Bruno mentioned since there's compatibility issues we
cannot
> > upgrade RocksDB any more.
> >
> >
> > Guozhang
> >
> > On Thu, Feb 3, 2022 at 1:56 PM Jonathan Albrecht
<Jo...@ibm.com>
> > wrote:
> >
> >>
> >>
> >> Thanks Guozhang, yes the motivation is to support the s390x platform.
It's
> >> not a critical bug for other platforms.
> >>
> >> Any chance that gaining platform support is also a valid reason? I was
> >> hoping it would be but I won't submit a PR if it isn't.
> >>
> >> Thanks,
> >>
> >> Jon
> >>
> >> "Guozhang Wang" <wa...@gmail.com> wrote on 2022-02-03 02:14:34 PM:
> >>
> >>> From: "Guozhang Wang" <wa...@gmail.com>
> >>> To: "dev" <de...@kafka.apache.org>
> >>> Date: 2022-02-03 02:15 PM
> >>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> >>>
> >>> Hello Jonathan,
> >>>
> >>> I think Bruno's point is that we can only upgrade in the bugfix
releases if
> >>> the old version of rocksDB has a critical bug that the new version
would
> >>> fix. For 6.27.3 it seems not fixing a critical bug but for a new
feature,
> >>> and hence unless we change the policy we cannot upgrade in 3.0.1 /
3.1.1
> >>> releases.
> >>>
> >>>
> >>> Guozhang
> >>>
> >>> On Thu, Feb 3, 2022 at 9:22 AM Jonathan Albrecht
<Jo...@ibm.com> wrote:
> >>>
> >>>>
> >>>>
> >>>>
> >>>> Thanks for the info Bruno. In that case, if no other concerns, I'll
try
> >>>> updating RocksDB to v6.27.3 on the 3.0 and 3.1 branches and file
issues and
> >>>> PRs if everything looks good.
> >>>>
> >>>> Jon
> >>>>
> >>>>
> >>>> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-03 10:40:00
AM:
> >>>>
> >>>>> From: "Bruno Cadonna" <ca...@apache.org>
> >>>>> To: dev@kafka.apache.org
> >>>>> Date: 2022-02-03 10:40 AM
> >>>>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> >>>>>
> >>>>> Hi Jonathan,
> >>>>>
> >>>>> We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to
source
> >>>>> compatibility issue. More specifically, we expose RocksDB APIs in
Kafka
> >>>>> Streams for configuring RocksDB and those RocksDB APIs changed. So
> >>>>> upgrading RocksDB was actually a compatibility breaking change. We
had
> >>>>> to wait for the major release 3.0.0 to make the upgrade. That
means, if
> >>>>> the policy allows to upgrade dependencies in bugfix releases we can
only
> >>>>> upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading
RocksDB in
> >>>>> earlier releases would break compatibility.
> >>>>>
> >>>>> Best,
> >>>>> Bruno
> >>>>>
> >>>>> On 03.02.22 15:15, Jonathan Albrecht wrote:
> >>>>>>
> >>>>>>
> >>>>>> The rocksdbjni dependency has been upgraded to v6.27.3 on trunk
and
> >>>>>> wanted to ask if it would be ok to also upgrade it to v6.27.3 on
the 3.1
> >>>>>> branch (and possibly earlier branches). I thought I should ask in
case
> >>>>>> there are some policies around changing dependency versions in
point
> >>>>>> releases.
> >>>>>>
> >>>>>> The motivation is that this is the first version of rocksdbjni
that
> >>>>>> supports s390x and it allows kafka to be built out of the box on
this
> >>>>>> platform. Having this support in earlier releases helps users on
s390x
> >>>>>> that
> >>>>>> may need a specific minor release.
> >>>>>>
> >>>>>> If upgrading earlier releases is ok, how far back would be
reasonable?
> >>>>>> I'm
> >>>>>> happy to create the issues and PRs and do the local testing, of
course.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Jonathan Albrecht
> >>>>>> Advisory Software Developer
> >>>>>> Linux on IBM Z Open Source Ecosystem
> >>>>>> 1 905 413 3577 Office
> >>>>>> jonathan.albrecht@ibm.com
> >>>>>>
> >>>>>> IBM
> >>>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>
> >
> >
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Bruno Cadonna <ca...@apache.org>.
Hi Jonathan and Guozhang,
I am pretty sure that upgrading RocksDB in 3.0.x and 3.1.x should not be
an issue, since we recently upgraded it to 6.27.3 on trunk and judging
from the compatibility report in the ticket [1] and the code the API
does not break backward-compatibility for the AK 3.x series.
I am wondering, why users would not be willing to use the upcoming 3.2.0
if they are migrating or creating a Kafka Streams application on the
s390x platform. Even if we ported the upgrade to 3.1.1 and 3.0.1, they
would need to wait until those versions are released. Additionally,
their applications need to comply with 3.x APIs and APIs should be
backward compatible in the 3.x releases. So, everything that works with
3.1.1 and 3.0.1 should also work with 3.2.0. Of course there could be an
issue in 3.2.0 that will force them to use a 3.1.1 or 3.0.1, but that
would be rather an exception that we can then fix when the exception
happens.
Jonathan, could you clarify what the motivation of using hypothetical
3.1.1 or 3.0.1 with the RocksDB upgrade instead of 3.2.0 on a platform
that was not supported before (i.e., 3.1.0 and 3.0.0) might be?
In the end, it is always a risk to upgrade a library in a bugfix release
without a critical issue that the upgrade fixes.
Best,
Bruno
[1] https://issues.apache.org/jira/browse/KAFKA-13599
On 07.02.22 04:19, Guozhang Wang wrote:
> Hi Jonathan,
>
> I'm not against the idea of upgrading in 3.0.x and 3.1.x, assuming that the
> v6.27.3 version does not make any API or any semantic behavioral changes.
> But I can only speak for myself, not the whole community. For older
> versions as Bruno mentioned since there's compatibility issues we cannot
> upgrade RocksDB any more.
>
>
> Guozhang
>
> On Thu, Feb 3, 2022 at 1:56 PM Jonathan Albrecht <Jo...@ibm.com>
> wrote:
>
>>
>>
>> Thanks Guozhang, yes the motivation is to support the s390x platform. It's
>> not a critical bug for other platforms.
>>
>> Any chance that gaining platform support is also a valid reason? I was
>> hoping it would be but I won't submit a PR if it isn't.
>>
>> Thanks,
>>
>> Jon
>>
>> "Guozhang Wang" <wa...@gmail.com> wrote on 2022-02-03 02:14:34 PM:
>>
>>> From: "Guozhang Wang" <wa...@gmail.com>
>>> To: "dev" <de...@kafka.apache.org>
>>> Date: 2022-02-03 02:15 PM
>>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>>>
>>> Hello Jonathan,
>>>
>>> I think Bruno's point is that we can only upgrade in the bugfix releases
>> if
>>> the old version of rocksDB has a critical bug that the new version would
>>> fix. For 6.27.3 it seems not fixing a critical bug but for a new feature,
>>> and hence unless we change the policy we cannot upgrade in 3.0.1 / 3.1.1
>>> releases.
>>>
>>>
>>> Guozhang
>>>
>>> On Thu, Feb 3, 2022 at 9:22 AM Jonathan Albrecht
>> <Jo...@ibm.com>
>>> wrote:
>>>
>>>>
>>>>
>>>>
>>>> Thanks for the info Bruno. In that case, if no other concerns, I'll try
>>>> updating RocksDB to v6.27.3 on the 3.0 and 3.1 branches and file issues
>> and
>>>> PRs if everything looks good.
>>>>
>>>> Jon
>>>>
>>>>
>>>> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-03 10:40:00 AM:
>>>>
>>>>> From: "Bruno Cadonna" <ca...@apache.org>
>>>>> To: dev@kafka.apache.org
>>>>> Date: 2022-02-03 10:40 AM
>>>>> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>>>>>
>>>>> Hi Jonathan,
>>>>>
>>>>> We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to
>> source
>>>>> compatibility issue. More specifically, we expose RocksDB APIs in
>> Kafka
>>>>> Streams for configuring RocksDB and those RocksDB APIs changed. So
>>>>> upgrading RocksDB was actually a compatibility breaking change. We
>> had
>>>>> to wait for the major release 3.0.0 to make the upgrade. That means,
>> if
>>>>> the policy allows to upgrade dependencies in bugfix releases we can
>> only
>>>>> upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading RocksDB
>> in
>>>>> earlier releases would break compatibility.
>>>>>
>>>>> Best,
>>>>> Bruno
>>>>>
>>>>> On 03.02.22 15:15, Jonathan Albrecht wrote:
>>>>>>
>>>>>>
>>>>>> The rocksdbjni dependency has been upgraded to v6.27.3 on trunk and
>> I
>>>>>> wanted to ask if it would be ok to also upgrade it to v6.27.3 on
>> the
>>>>>> 3.1
>>>>>> branch (and possibly earlier branches). I thought I should ask in
>> case
>>>>>> there are some policies around changing dependency versions in
>> point
>>>>>> releases.
>>>>>>
>>>>>> The motivation is that this is the first version of rocksdbjni that
>>>>>> supports s390x and it allows kafka to be built out of the box on
>> this
>>>>>> platform. Having this support in earlier releases helps users on
>> s390x
>>>>>> that
>>>>>> may need a specific minor release.
>>>>>>
>>>>>> If upgrading earlier releases is ok, how far back would be
>> reasonable?
>>>>>> I'm
>>>>>> happy to create the issues and PRs and do the local testing, of
>> course.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jonathan Albrecht
>>>>>> Advisory Software Developer
>>>>>> Linux on IBM Z Open Source Ecosystem
>>>>>> 1 905 413 3577 Office
>>>>>> jonathan.albrecht@ibm.com
>>>>>>
>>>>>> IBM
>>>>>>
>>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>
>
>
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Jonathan,
I'm not against the idea of upgrading in 3.0.x and 3.1.x, assuming that the
v6.27.3 version does not make any API or any semantic behavioral changes.
But I can only speak for myself, not the whole community. For older
versions as Bruno mentioned since there's compatibility issues we cannot
upgrade RocksDB any more.
Guozhang
On Thu, Feb 3, 2022 at 1:56 PM Jonathan Albrecht <Jo...@ibm.com>
wrote:
>
>
> Thanks Guozhang, yes the motivation is to support the s390x platform. It's
> not a critical bug for other platforms.
>
> Any chance that gaining platform support is also a valid reason? I was
> hoping it would be but I won't submit a PR if it isn't.
>
> Thanks,
>
> Jon
>
> "Guozhang Wang" <wa...@gmail.com> wrote on 2022-02-03 02:14:34 PM:
>
> > From: "Guozhang Wang" <wa...@gmail.com>
> > To: "dev" <de...@kafka.apache.org>
> > Date: 2022-02-03 02:15 PM
> > Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> >
> > Hello Jonathan,
> >
> > I think Bruno's point is that we can only upgrade in the bugfix releases
> if
> > the old version of rocksDB has a critical bug that the new version would
> > fix. For 6.27.3 it seems not fixing a critical bug but for a new feature,
> > and hence unless we change the policy we cannot upgrade in 3.0.1 / 3.1.1
> > releases.
> >
> >
> > Guozhang
> >
> > On Thu, Feb 3, 2022 at 9:22 AM Jonathan Albrecht
> <Jo...@ibm.com>
> > wrote:
> >
> > >
> > >
> > >
> > > Thanks for the info Bruno. In that case, if no other concerns, I'll try
> > > updating RocksDB to v6.27.3 on the 3.0 and 3.1 branches and file issues
> and
> > > PRs if everything looks good.
> > >
> > > Jon
> > >
> > >
> > > "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-03 10:40:00 AM:
> > >
> > > > From: "Bruno Cadonna" <ca...@apache.org>
> > > > To: dev@kafka.apache.org
> > > > Date: 2022-02-03 10:40 AM
> > > > Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> > > >
> > > > Hi Jonathan,
> > > >
> > > > We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to
> source
> > > > compatibility issue. More specifically, we expose RocksDB APIs in
> Kafka
> > > > Streams for configuring RocksDB and those RocksDB APIs changed. So
> > > > upgrading RocksDB was actually a compatibility breaking change. We
> had
> > > > to wait for the major release 3.0.0 to make the upgrade. That means,
> if
> > > > the policy allows to upgrade dependencies in bugfix releases we can
> only
> > > > upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading RocksDB
> in
> > > > earlier releases would break compatibility.
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > > > On 03.02.22 15:15, Jonathan Albrecht wrote:
> > > > >
> > > > >
> > > > > The rocksdbjni dependency has been upgraded to v6.27.3 on trunk and
> I
> > > > > wanted to ask if it would be ok to also upgrade it to v6.27.3 on
> the
> > > > > 3.1
> > > > > branch (and possibly earlier branches). I thought I should ask in
> case
> > > > > there are some policies around changing dependency versions in
> point
> > > > > releases.
> > > > >
> > > > > The motivation is that this is the first version of rocksdbjni that
> > > > > supports s390x and it allows kafka to be built out of the box on
> this
> > > > > platform. Having this support in earlier releases helps users on
> s390x
> > > > > that
> > > > > may need a specific minor release.
> > > > >
> > > > > If upgrading earlier releases is ok, how far back would be
> reasonable?
> > > > > I'm
> > > > > happy to create the issues and PRs and do the local testing, of
> course.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jonathan Albrecht
> > > > > Advisory Software Developer
> > > > > Linux on IBM Z Open Source Ecosystem
> > > > > 1 905 413 3577 Office
> > > > > jonathan.albrecht@ibm.com
> > > > >
> > > > > IBM
> > > > >
> > >
> >
> >
> > --
> > -- Guozhang
>
--
-- Guozhang
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Jonathan Albrecht <Jo...@ibm.com>.
Thanks Guozhang, yes the motivation is to support the s390x platform. It's
not a critical bug for other platforms.
Any chance that gaining platform support is also a valid reason? I was
hoping it would be but I won't submit a PR if it isn't.
Thanks,
Jon
"Guozhang Wang" <wa...@gmail.com> wrote on 2022-02-03 02:14:34 PM:
> From: "Guozhang Wang" <wa...@gmail.com>
> To: "dev" <de...@kafka.apache.org>
> Date: 2022-02-03 02:15 PM
> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>
> Hello Jonathan,
>
> I think Bruno's point is that we can only upgrade in the bugfix releases
if
> the old version of rocksDB has a critical bug that the new version would
> fix. For 6.27.3 it seems not fixing a critical bug but for a new feature,
> and hence unless we change the policy we cannot upgrade in 3.0.1 / 3.1.1
> releases.
>
>
> Guozhang
>
> On Thu, Feb 3, 2022 at 9:22 AM Jonathan Albrecht
<Jo...@ibm.com>
> wrote:
>
> >
> >
> >
> > Thanks for the info Bruno. In that case, if no other concerns, I'll try
> > updating RocksDB to v6.27.3 on the 3.0 and 3.1 branches and file issues
and
> > PRs if everything looks good.
> >
> > Jon
> >
> >
> > "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-03 10:40:00 AM:
> >
> > > From: "Bruno Cadonna" <ca...@apache.org>
> > > To: dev@kafka.apache.org
> > > Date: 2022-02-03 10:40 AM
> > > Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> > >
> > > Hi Jonathan,
> > >
> > > We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to
source
> > > compatibility issue. More specifically, we expose RocksDB APIs in
Kafka
> > > Streams for configuring RocksDB and those RocksDB APIs changed. So
> > > upgrading RocksDB was actually a compatibility breaking change. We
had
> > > to wait for the major release 3.0.0 to make the upgrade. That means,
if
> > > the policy allows to upgrade dependencies in bugfix releases we can
only
> > > upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading RocksDB
in
> > > earlier releases would break compatibility.
> > >
> > > Best,
> > > Bruno
> > >
> > > On 03.02.22 15:15, Jonathan Albrecht wrote:
> > > >
> > > >
> > > > The rocksdbjni dependency has been upgraded to v6.27.3 on trunk and
I
> > > > wanted to ask if it would be ok to also upgrade it to v6.27.3 on
the
> > > > 3.1
> > > > branch (and possibly earlier branches). I thought I should ask in
case
> > > > there are some policies around changing dependency versions in
point
> > > > releases.
> > > >
> > > > The motivation is that this is the first version of rocksdbjni that
> > > > supports s390x and it allows kafka to be built out of the box on
this
> > > > platform. Having this support in earlier releases helps users on
s390x
> > > > that
> > > > may need a specific minor release.
> > > >
> > > > If upgrading earlier releases is ok, how far back would be
reasonable?
> > > > I'm
> > > > happy to create the issues and PRs and do the local testing, of
course.
> > > >
> > > > Thanks,
> > > >
> > > > Jonathan Albrecht
> > > > Advisory Software Developer
> > > > Linux on IBM Z Open Source Ecosystem
> > > > 1 905 413 3577 Office
> > > > jonathan.albrecht@ibm.com
> > > >
> > > > IBM
> > > >
> >
>
>
> --
> -- Guozhang
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Jonathan,
I think Bruno's point is that we can only upgrade in the bugfix releases if
the old version of rocksDB has a critical bug that the new version would
fix. For 6.27.3 it seems not fixing a critical bug but for a new feature,
and hence unless we change the policy we cannot upgrade in 3.0.1 / 3.1.1
releases.
Guozhang
On Thu, Feb 3, 2022 at 9:22 AM Jonathan Albrecht <Jo...@ibm.com>
wrote:
>
>
>
> Thanks for the info Bruno. In that case, if no other concerns, I'll try
> updating RocksDB to v6.27.3 on the 3.0 and 3.1 branches and file issues and
> PRs if everything looks good.
>
> Jon
>
>
> "Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-03 10:40:00 AM:
>
> > From: "Bruno Cadonna" <ca...@apache.org>
> > To: dev@kafka.apache.org
> > Date: 2022-02-03 10:40 AM
> > Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
> >
> > Hi Jonathan,
> >
> > We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to source
> > compatibility issue. More specifically, we expose RocksDB APIs in Kafka
> > Streams for configuring RocksDB and those RocksDB APIs changed. So
> > upgrading RocksDB was actually a compatibility breaking change. We had
> > to wait for the major release 3.0.0 to make the upgrade. That means, if
> > the policy allows to upgrade dependencies in bugfix releases we can only
> > upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading RocksDB in
> > earlier releases would break compatibility.
> >
> > Best,
> > Bruno
> >
> > On 03.02.22 15:15, Jonathan Albrecht wrote:
> > >
> > >
> > > The rocksdbjni dependency has been upgraded to v6.27.3 on trunk and I
> > > wanted to ask if it would be ok to also upgrade it to v6.27.3 on the
> 3.1
> > > branch (and possibly earlier branches). I thought I should ask in case
> > > there are some policies around changing dependency versions in point
> > > releases.
> > >
> > > The motivation is that this is the first version of rocksdbjni that
> > > supports s390x and it allows kafka to be built out of the box on this
> > > platform. Having this support in earlier releases helps users on s390x
> that
> > > may need a specific minor release.
> > >
> > > If upgrading earlier releases is ok, how far back would be reasonable?
> I'm
> > > happy to create the issues and PRs and do the local testing, of course.
> > >
> > > Thanks,
> > >
> > > Jonathan Albrecht
> > > Advisory Software Developer
> > > Linux on IBM Z Open Source Ecosystem
> > > 1 905 413 3577 Office
> > > jonathan.albrecht@ibm.com
> > >
> > > IBM
> > >
>
--
-- Guozhang
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Jonathan Albrecht <Jo...@ibm.com>.
Thanks for the info Bruno. In that case, if no other concerns, I'll try
updating RocksDB to v6.27.3 on the 3.0 and 3.1 branches and file issues and
PRs if everything looks good.
Jon
"Bruno Cadonna" <ca...@apache.org> wrote on 2022-02-03 10:40:00 AM:
> From: "Bruno Cadonna" <ca...@apache.org>
> To: dev@kafka.apache.org
> Date: 2022-02-03 10:40 AM
> Subject: [EXTERNAL] Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
>
> Hi Jonathan,
>
> We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to source
> compatibility issue. More specifically, we expose RocksDB APIs in Kafka
> Streams for configuring RocksDB and those RocksDB APIs changed. So
> upgrading RocksDB was actually a compatibility breaking change. We had
> to wait for the major release 3.0.0 to make the upgrade. That means, if
> the policy allows to upgrade dependencies in bugfix releases we can only
> upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading RocksDB in
> earlier releases would break compatibility.
>
> Best,
> Bruno
>
> On 03.02.22 15:15, Jonathan Albrecht wrote:
> >
> >
> > The rocksdbjni dependency has been upgraded to v6.27.3 on trunk and I
> > wanted to ask if it would be ok to also upgrade it to v6.27.3 on the
3.1
> > branch (and possibly earlier branches). I thought I should ask in case
> > there are some policies around changing dependency versions in point
> > releases.
> >
> > The motivation is that this is the first version of rocksdbjni that
> > supports s390x and it allows kafka to be built out of the box on this
> > platform. Having this support in earlier releases helps users on s390x
that
> > may need a specific minor release.
> >
> > If upgrading earlier releases is ok, how far back would be reasonable?
I'm
> > happy to create the issues and PRs and do the local testing, of course.
> >
> > Thanks,
> >
> > Jonathan Albrecht
> > Advisory Software Developer
> > Linux on IBM Z Open Source Ecosystem
> > 1 905 413 3577 Office
> > jonathan.albrecht@ibm.com
> >
> > IBM
> >
Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?
Posted by Bruno Cadonna <ca...@apache.org>.
Hi Jonathan,
We had to wait until AK 3.0 to upgrade RocksDB to 6.19.3 due to source
compatibility issue. More specifically, we expose RocksDB APIs in Kafka
Streams for configuring RocksDB and those RocksDB APIs changed. So
upgrading RocksDB was actually a compatibility breaking change. We had
to wait for the major release 3.0.0 to make the upgrade. That means, if
the policy allows to upgrade dependencies in bugfix releases we can only
upgrade RocksDB in bugfix releases for 3.1 and 3.0. Upgrading RocksDB in
earlier releases would break compatibility.
Best,
Bruno
On 03.02.22 15:15, Jonathan Albrecht wrote:
>
>
> The rocksdbjni dependency has been upgraded to v6.27.3 on trunk and I
> wanted to ask if it would be ok to also upgrade it to v6.27.3 on the 3.1
> branch (and possibly earlier branches). I thought I should ask in case
> there are some policies around changing dependency versions in point
> releases.
>
> The motivation is that this is the first version of rocksdbjni that
> supports s390x and it allows kafka to be built out of the box on this
> platform. Having this support in earlier releases helps users on s390x that
> may need a specific minor release.
>
> If upgrading earlier releases is ok, how far back would be reasonable? I'm
> happy to create the issues and PRs and do the local testing, of course.
>
> Thanks,
>
> Jonathan Albrecht
> Advisory Software Developer
> Linux on IBM Z Open Source Ecosystem
> 1 905 413 3577 Office
> jonathan.albrecht@ibm.com
>
> IBM
>