You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Yanfei Lei <fr...@gmail.com> on 2022/12/12 14:02:03 UTC
[DISCUSS] Release new FRocksDB
Hi devs,
I'd like to bring up a discussion about releasing the new frocksdbjni
version, we are planning to adapt frocksdbjni to Apple M1 machines in
Flink 1.17[1], a new frocksdbjni jar needs to be released.
Thanks to the efforts of the community, FLINK-24932[1] has been merged into
FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a paid
CircleCI for testing, which ensures the correctness of the code.
The new frocksdbjni version is still based on 6.20.3[2], the code
interfaces and performance[3] will not change. In addition to
FLINK-24932[1], some fixes of vulnerabilities[4] will also be included in
the new version, see FLINK-30321[5].
Are there any minor fixes you would like to include in this release?
Welcome to create PR for FrocksDB[6].
Looking forward to your opinions.
[1] https://issues.apache.org/jira/browse/FLINK-24932
[2]
https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0
[3]
https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889
[4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr
[5] https://issues.apache.org/jira/browse/FLINK-30321
[6] https://github.com/ververica/frocksdb
--
Best,
Yanfei
Re: [DISCUSS] Release new FRocksDB
Posted by Yanfei Lei <fr...@gmail.com>.
Hi all,
Thanks for everyone's advice, FRocksDB 6.20.3-ververica-2.0[1] had
been released, compared with FRocksDB 6.20.3-ververica-1.0, there are
the following 4 points of improvement:
1. [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[2]
2. [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[3]
3. PR 55: Avoid expensive ToString call when not in Debug[4]
4. [FLINK-24932] Support build FRocksDB Java on Apple silicon[5]
Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
MaxOS arm, and windows are included in FRocksDB 6.20.3-ververica-2.0
jar, many thanks to @Yun Tang for the work done to support the Linux
ppc64le platform. And the FrocksDB in Flink 1.17 would be updated to
6.20.3-ververica-2.0, thanks again for everyone's suggestions and
efforts.
[1] https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-2.0
[2] https://issues.apache.org/jira/browse/FLINK-30457
[3] https://issues.apache.org/jira/browse/FLINK-30321
[4] https://github.com/ververica/frocksdb/pull/55
[5] https://issues.apache.org/jira/browse/FLINK-24932
Best,
Yanfei
ConradJam <ja...@gmail.com> 于2022年12月20日周二 15:16写道：
>
> +1 , I agree to open a 7.x branch to study whether the new version of
> rocksdb has brought something new changes
>
> Yun Tang <my...@live.com> 于2022年12月20日周二 12:47写道：
>
> > +1 to have a new FrocksDB release and thanks for the effors.
> >
> > For FLINK-24932, we planned to upgrade to RocksDB-6.29.x to support Apple
> > ARM chips. However, we found another performance regression between 6.20.3
> > and 6.29.5 [1]. That’s why we still stay in the 6.20.3 branch. Since we
> > already decided to create a new binary release, perhaps we could also try
> > RocksDB-7.x to see the performance change.
> >
> > For the improvements to include in the new release, we can consider
> > including [2] for better performance and FLINK-30457 [3] to support period
> > compaction once we set TTL in Flink, which is asked by some users.
> >
> >
> > [1]
> > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331
> > [2] https://github.com/ververica/frocksdb/pull/55
> > [3] https://issues.apache.org/jira/browse/FLINK-30457
> >
> > Best
> > Yun Tang
> >
> >
> >
> > From: Yu Li <ca...@gmail.com>
> > Date: Monday, December 19, 2022 at 22:01
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] Release new FRocksDB
> > +1 to have a new FRocksDB release. Thanks Yanfei and all for the efforts!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Fri, 16 Dec 2022 at 17:02, Martijn Visser <ma...@apache.org>
> > wrote:
> >
> > > Hi Yanfei,
> > >
> > > Big +1 for adding ARM support. It would be great to be able to get tests
> > to
> > > pass on Mac M1s.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi <balassi.marton@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Yanfei,
> > > >
> > > > Makes sense, arm architecture support would be great. Thanks for
> > looking
> > > > into this.
> > > >
> > > > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei <fr...@gmail.com>
> > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I'd like to bring up a discussion about releasing the new frocksdbjni
> > > > > version, we are planning to adapt frocksdbjni to Apple M1 machines
> > in
> > > > > Flink 1.17[1], a new frocksdbjni jar needs to be released.
> > > > >
> > > > > Thanks to the efforts of the community, FLINK-24932[1] has been
> > merged
> > > > into
> > > > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing
> > a
> > > > paid
> > > > > CircleCI for testing, which ensures the correctness of the code.
> > > > >
> > > > > The new frocksdbjni version is still based on 6.20.3[2], the code
> > > > > interfaces and performance[3] will not change. In addition to
> > > > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be
> > included
> > > > in
> > > > > the new version, see FLINK-30321[5].
> > > > >
> > > > > Are there any minor fixes you would like to include in this release?
> > > > > Welcome to create PR for FrocksDB[6].
> > > > > Looking forward to your opinions.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-24932
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> > https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0
> > > > > [3]
> > > > >
> > > > >
> > > >
> > >
> > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889
> > > > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr
> > > > > [5] https://issues.apache.org/jira/browse/FLINK-30321
> > > > > [6] https://github.com/ververica/frocksdb
> > > > >
> > > > > --
> > > > > Best,
> > > > > Yanfei
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] Release new FRocksDB
Posted by ConradJam <ja...@gmail.com>.
+1 , I agree to open a 7.x branch to study whether the new version of
rocksdb has brought something new changes
Yun Tang <my...@live.com> 于2022年12月20日周二 12:47写道：
> +1 to have a new FrocksDB release and thanks for the effors.
>
> For FLINK-24932, we planned to upgrade to RocksDB-6.29.x to support Apple
> ARM chips. However, we found another performance regression between 6.20.3
> and 6.29.5 [1]. That’s why we still stay in the 6.20.3 branch. Since we
> already decided to create a new binary release, perhaps we could also try
> RocksDB-7.x to see the performance change.
>
> For the improvements to include in the new release, we can consider
> including [2] for better performance and FLINK-30457 [3] to support period
> compaction once we set TTL in Flink, which is asked by some users.
>
>
> [1]
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331
> [2] https://github.com/ververica/frocksdb/pull/55
> [3] https://issues.apache.org/jira/browse/FLINK-30457
>
> Best
> Yun Tang
>
>
>
> From: Yu Li <ca...@gmail.com>
> Date: Monday, December 19, 2022 at 22:01
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] Release new FRocksDB
> +1 to have a new FRocksDB release. Thanks Yanfei and all for the efforts!
>
> Best Regards,
> Yu
>
>
> On Fri, 16 Dec 2022 at 17:02, Martijn Visser <ma...@apache.org>
> wrote:
>
> > Hi Yanfei,
> >
> > Big +1 for adding ARM support. It would be great to be able to get tests
> to
> > pass on Mac M1s.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi <balassi.marton@gmail.com
> >
> > wrote:
> >
> > > Hi Yanfei,
> > >
> > > Makes sense, arm architecture support would be great. Thanks for
> looking
> > > into this.
> > >
> > > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei <fr...@gmail.com>
> wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I'd like to bring up a discussion about releasing the new frocksdbjni
> > > > version, we are planning to adapt frocksdbjni to Apple M1 machines
> in
> > > > Flink 1.17[1], a new frocksdbjni jar needs to be released.
> > > >
> > > > Thanks to the efforts of the community, FLINK-24932[1] has been
> merged
> > > into
> > > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing
> a
> > > paid
> > > > CircleCI for testing, which ensures the correctness of the code.
> > > >
> > > > The new frocksdbjni version is still based on 6.20.3[2], the code
> > > > interfaces and performance[3] will not change. In addition to
> > > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be
> included
> > > in
> > > > the new version, see FLINK-30321[5].
> > > >
> > > > Are there any minor fixes you would like to include in this release?
> > > > Welcome to create PR for FrocksDB[6].
> > > > Looking forward to your opinions.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-24932
> > > > [2]
> > > >
> > > >
> > >
> >
> https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0
> > > > [3]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889
> > > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr
> > > > [5] https://issues.apache.org/jira/browse/FLINK-30321
> > > > [6] https://github.com/ververica/frocksdb
> > > >
> > > > --
> > > > Best,
> > > > Yanfei
> > > >
> > >
> >
>
Re: [DISCUSS] Release new FRocksDB
Posted by Yun Tang <my...@live.com>.
+1 to have a new FrocksDB release and thanks for the effors.
For FLINK-24932, we planned to upgrade to RocksDB-6.29.x to support Apple ARM chips. However, we found another performance regression between 6.20.3 and 6.29.5 [1]. That’s why we still stay in the 6.20.3 branch. Since we already decided to create a new binary release, perhaps we could also try RocksDB-7.x to see the performance change.
For the improvements to include in the new release, we can consider including [2] for better performance and FLINK-30457 [3] to support period compaction once we set TTL in Flink, which is asked by some users.
[1] https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331
[2] https://github.com/ververica/frocksdb/pull/55
[3] https://issues.apache.org/jira/browse/FLINK-30457
Best
Yun Tang
From: Yu Li <ca...@gmail.com>
Date: Monday, December 19, 2022 at 22:01
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [DISCUSS] Release new FRocksDB
+1 to have a new FRocksDB release. Thanks Yanfei and all for the efforts!
Best Regards,
Yu
On Fri, 16 Dec 2022 at 17:02, Martijn Visser <ma...@apache.org>
wrote:
> Hi Yanfei,
>
> Big +1 for adding ARM support. It would be great to be able to get tests to
> pass on Mac M1s.
>
> Best regards,
>
> Martijn
>
> On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Hi Yanfei,
> >
> > Makes sense, arm architecture support would be great. Thanks for looking
> > into this.
> >
> > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei <fr...@gmail.com> wrote:
> >
> > > Hi devs,
> > >
> > > I'd like to bring up a discussion about releasing the new frocksdbjni
> > > version, we are planning to adapt frocksdbjni to Apple M1 machines in
> > > Flink 1.17[1], a new frocksdbjni jar needs to be released.
> > >
> > > Thanks to the efforts of the community, FLINK-24932[1] has been merged
> > into
> > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a
> > paid
> > > CircleCI for testing, which ensures the correctness of the code.
> > >
> > > The new frocksdbjni version is still based on 6.20.3[2], the code
> > > interfaces and performance[3] will not change. In addition to
> > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be included
> > in
> > > the new version, see FLINK-30321[5].
> > >
> > > Are there any minor fixes you would like to include in this release?
> > > Welcome to create PR for FrocksDB[6].
> > > Looking forward to your opinions.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-24932
> > > [2]
> > >
> > >
> >
> https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0
> > > [3]
> > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889
> > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr
> > > [5] https://issues.apache.org/jira/browse/FLINK-30321
> > > [6] https://github.com/ververica/frocksdb
> > >
> > > --
> > > Best,
> > > Yanfei
> > >
> >
>
Re: [DISCUSS] Release new FRocksDB
Posted by Yu Li <ca...@gmail.com>.
+1 to have a new FRocksDB release. Thanks Yanfei and all for the efforts!
Best Regards,
Yu
On Fri, 16 Dec 2022 at 17:02, Martijn Visser <ma...@apache.org>
wrote:
> Hi Yanfei,
>
> Big +1 for adding ARM support. It would be great to be able to get tests to
> pass on Mac M1s.
>
> Best regards,
>
> Martijn
>
> On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Hi Yanfei,
> >
> > Makes sense, arm architecture support would be great. Thanks for looking
> > into this.
> >
> > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei <fr...@gmail.com> wrote:
> >
> > > Hi devs,
> > >
> > > I'd like to bring up a discussion about releasing the new frocksdbjni
> > > version, we are planning to adapt frocksdbjni to Apple M1 machines in
> > > Flink 1.17[1], a new frocksdbjni jar needs to be released.
> > >
> > > Thanks to the efforts of the community, FLINK-24932[1] has been merged
> > into
> > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a
> > paid
> > > CircleCI for testing, which ensures the correctness of the code.
> > >
> > > The new frocksdbjni version is still based on 6.20.3[2], the code
> > > interfaces and performance[3] will not change. In addition to
> > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be included
> > in
> > > the new version, see FLINK-30321[5].
> > >
> > > Are there any minor fixes you would like to include in this release?
> > > Welcome to create PR for FrocksDB[6].
> > > Looking forward to your opinions.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-24932
> > > [2]
> > >
> > >
> >
> https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0
> > > [3]
> > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889
> > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr
> > > [5] https://issues.apache.org/jira/browse/FLINK-30321
> > > [6] https://github.com/ververica/frocksdb
> > >
> > > --
> > > Best,
> > > Yanfei
> > >
> >
>
Re: [DISCUSS] Release new FRocksDB
Posted by Martijn Visser <ma...@apache.org>.
Hi Yanfei,
Big +1 for adding ARM support. It would be great to be able to get tests to
pass on Mac M1s.
Best regards,
Martijn
On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi <ba...@gmail.com>
wrote:
> Hi Yanfei,
>
> Makes sense, arm architecture support would be great. Thanks for looking
> into this.
>
> On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei <fr...@gmail.com> wrote:
>
> > Hi devs,
> >
> > I'd like to bring up a discussion about releasing the new frocksdbjni
> > version, we are planning to adapt frocksdbjni to Apple M1 machines in
> > Flink 1.17[1], a new frocksdbjni jar needs to be released.
> >
> > Thanks to the efforts of the community, FLINK-24932[1] has been merged
> into
> > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a
> paid
> > CircleCI for testing, which ensures the correctness of the code.
> >
> > The new frocksdbjni version is still based on 6.20.3[2], the code
> > interfaces and performance[3] will not change. In addition to
> > FLINK-24932[1], some fixes of vulnerabilities[4] will also be included
> in
> > the new version, see FLINK-30321[5].
> >
> > Are there any minor fixes you would like to include in this release?
> > Welcome to create PR for FrocksDB[6].
> > Looking forward to your opinions.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-24932
> > [2]
> >
> >
> https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0
> > [3]
> >
> >
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889
> > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr
> > [5] https://issues.apache.org/jira/browse/FLINK-30321
> > [6] https://github.com/ververica/frocksdb
> >
> > --
> > Best,
> > Yanfei
> >
>
Re: [DISCUSS] Release new FRocksDB
Posted by Márton Balassi <ba...@gmail.com>.
Hi Yanfei,
Makes sense, arm architecture support would be great. Thanks for looking
into this.
On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei <fr...@gmail.com> wrote:
> Hi devs,
>
> I'd like to bring up a discussion about releasing the new frocksdbjni
> version, we are planning to adapt frocksdbjni to Apple M1 machines in
> Flink 1.17[1], a new frocksdbjni jar needs to be released.
>
> Thanks to the efforts of the community, FLINK-24932[1] has been merged into
> FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a paid
> CircleCI for testing, which ensures the correctness of the code.
>
> The new frocksdbjni version is still based on 6.20.3[2], the code
> interfaces and performance[3] will not change. In addition to
> FLINK-24932[1], some fixes of vulnerabilities[4] will also be included in
> the new version, see FLINK-30321[5].
>
> Are there any minor fixes you would like to include in this release?
> Welcome to create PR for FrocksDB[6].
> Looking forward to your opinions.
>
> [1] https://issues.apache.org/jira/browse/FLINK-24932
> [2]
>
> https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0
> [3]
>
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889
> [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr
> [5] https://issues.apache.org/jira/browse/FLINK-30321
> [6] https://github.com/ververica/frocksdb
>
> --
> Best,
> Yanfei
>