You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Zheng Yu Chen <ja...@gmail.com> on 2022/11/22 03:39:50 UTC
FRocksDB Some Question
Hi community, I want to ask a question about frocksdb
I noticed that frocksdb seems to have been inactive for a long time.
flink-statebackend-rocksdb is depend https://github.com/ververica/frocksdb
to run state backend
This Rocksdb is stuck at 6.20.3.But the current upstream version
https://github.com/facebook/rocksdb is already at 7.7.3.As the basic state
backend of Flink, this part
seems to be ignored by people (although users cannot directly perceive it)
I don't know how often this library is maintained and updated and the
differences between them and Rocksdb. It would be best if there are
relevant documents to tell me
If the community is willing to upgrade this underlying component, I am
willing to assist in this work as one of the volunteers.
At the same time, I would like to ask if there is any chance for this
project to be moved to the Apache Flink sub-project to allow more people to
participate?
--
Best
ConradJam
Re: FRocksDB Some Question
Posted by Yun Tang <my...@live.com>.
Hi ConradJam,
Thanks for taking a look at the FRocksDB project.
From my point of view, the reasons why Flink community would not upgrade the FRocksDB too frequently are mainly due to:
1. We have suffered from the performance regression of new versions of RocksDB compared with current versions[1]. Unfortunately, the RocksDB community (or we can say the RocksDB team at Meta Inc) does not have clear benchmark testing when releasing the new version each time. I spent some time to figure out why RocksDB-6.20.x behaves worse than RocksDB-5.17, however, we just found one major performance regression reason [2] and cannot advance more. That's why Flink community launches an ANNOUNCE of this problem [3].
2. Releasing RocksDB-java binaries is really a bit complicated as RocksDB has supported more and more platforms (linux32/64, windows, apple silicon, arm, ppc64, s390). However, we don't have so many environments to release these binaries within one jar package. We even considered launching discussions to drop some platform support.
Thus, if we don't have strong demands, we would not upgrade the FRocksDB too frequently. BTW, we aim to make the FRocksDB could run on Apple silicon chips in Flink-1.17 [4]
Last but not least, RocksDB actually has an apache-2.0 license. Apart from the possible license problem, FRocksDB [5] was created in 2019, in which the community does have so many Flink sub-projects yet.
[1] https://issues.apache.org/jira/browse/FLINK-14482
[2] https://issues.apache.org/jira/browse/FLINK-19710
[3] https://lists.apache.org/thread/v9w4dm6wdqgn5b6jqjdrzfycxsdh38vj
[4] https://issues.apache.org/jira/browse/FLINK-24932
[5] https://issues.apache.org/jira/browse/FLINK-10471
Best
Yun Tang
________________________________
From: Zheng Yu Chen <ja...@gmail.com>
Sent: Tuesday, November 22, 2022 17:02
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: FRocksDB Some Question
Thank you for your reply. I will join the discussion after careful review
Yanfei Lei <fr...@gmail.com> 于2022年11月22日周二 12:20写道：
> Hi ConradJam,
>
> Thanks for bringing this up.
> In fact, @Sebastian has tried to update RocksDB to 6.29.5[1], but there are
> some regressions on MapState[2]. If you want to use the newest feature of
> RocksDB, I think you can refer to this release document[3] to release your
> own version.
>
> > I would like to ask if there is any chance for this project to be moved
> to the Apache Flink sub-project to allow more people to participate?
> IMO, the open-source licenses of RocksDB and Apache Flink are different,
> moving FRocksDB to the Apache Flink sub-project may be hindered.
>
> [1]
>
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17560179&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17560179
>
> [2]
>
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331
> [3]
>
> https://github.com/ververica/frocksdb/blob/FRocksDB-6.20.3/FROCKSDB-RELEASE.md
>
>
> Zheng Yu Chen <ja...@gmail.com> 于2022年11月22日周二 11:40写道：
>
> > Hi community, I want to ask a question about frocksdb
> >
> > I noticed that frocksdb seems to have been inactive for a long time.
> > flink-statebackend-rocksdb is depend
> https://github.com/ververica/frocksdb
> > to run state backend
> > This Rocksdb is stuck at 6.20.3.But the current upstream version
> > https://github.com/facebook/rocksdb is already at 7.7.3.As the basic
> state
> > backend of Flink, this part
> > seems to be ignored by people (although users cannot directly perceive
> it)
> >
> > I don't know how often this library is maintained and updated and the
> > differences between them and Rocksdb. It would be best if there are
> > relevant documents to tell me
> >
> > If the community is willing to upgrade this underlying component, I am
> > willing to assist in this work as one of the volunteers.
> > At the same time, I would like to ask if there is any chance for this
> > project to be moved to the Apache Flink sub-project to allow more people
> to
> > participate?
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
>
> --
> Best,
> Yanfei
>
--
Best
ConradJam
Re: FRocksDB Some Question
Posted by Zheng Yu Chen <ja...@gmail.com>.
Thank you for your reply. I will join the discussion after careful review
Yanfei Lei <fr...@gmail.com> 于2022年11月22日周二 12:20写道：
> Hi ConradJam,
>
> Thanks for bringing this up.
> In fact, @Sebastian has tried to update RocksDB to 6.29.5[1], but there are
> some regressions on MapState[2]. If you want to use the newest feature of
> RocksDB, I think you can refer to this release document[3] to release your
> own version.
>
> > I would like to ask if there is any chance for this project to be moved
> to the Apache Flink sub-project to allow more people to participate?
> IMO, the open-source licenses of RocksDB and Apache Flink are different,
> moving FRocksDB to the Apache Flink sub-project may be hindered.
>
> [1]
>
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17560179&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17560179
>
> [2]
>
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331
> [3]
>
> https://github.com/ververica/frocksdb/blob/FRocksDB-6.20.3/FROCKSDB-RELEASE.md
>
>
> Zheng Yu Chen <ja...@gmail.com> 于2022年11月22日周二 11:40写道：
>
> > Hi community, I want to ask a question about frocksdb
> >
> > I noticed that frocksdb seems to have been inactive for a long time.
> > flink-statebackend-rocksdb is depend
> https://github.com/ververica/frocksdb
> > to run state backend
> > This Rocksdb is stuck at 6.20.3.But the current upstream version
> > https://github.com/facebook/rocksdb is already at 7.7.3.As the basic
> state
> > backend of Flink, this part
> > seems to be ignored by people (although users cannot directly perceive
> it)
> >
> > I don't know how often this library is maintained and updated and the
> > differences between them and Rocksdb. It would be best if there are
> > relevant documents to tell me
> >
> > If the community is willing to upgrade this underlying component, I am
> > willing to assist in this work as one of the volunteers.
> > At the same time, I would like to ask if there is any chance for this
> > project to be moved to the Apache Flink sub-project to allow more people
> to
> > participate?
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
>
> --
> Best,
> Yanfei
>
--
Best
ConradJam
Re: FRocksDB Some Question
Posted by Yanfei Lei <fr...@gmail.com>.
Hi ConradJam,
Thanks for bringing this up.
In fact, @Sebastian has tried to update RocksDB to 6.29.5[1], but there are
some regressions on MapState[2]. If you want to use the newest feature of
RocksDB, I think you can refer to this release document[3] to release your
own version.
> I would like to ask if there is any chance for this project to be moved
to the Apache Flink sub-project to allow more people to participate?
IMO, the open-source licenses of RocksDB and Apache Flink are different,
moving FRocksDB to the Apache Flink sub-project may be hindered.
[1]
https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17560179&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17560179
[2]
https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331
[3]
https://github.com/ververica/frocksdb/blob/FRocksDB-6.20.3/FROCKSDB-RELEASE.md
Zheng Yu Chen <ja...@gmail.com> 于2022年11月22日周二 11:40写道：
> Hi community, I want to ask a question about frocksdb
>
> I noticed that frocksdb seems to have been inactive for a long time.
> flink-statebackend-rocksdb is depend https://github.com/ververica/frocksdb
> to run state backend
> This Rocksdb is stuck at 6.20.3.But the current upstream version
> https://github.com/facebook/rocksdb is already at 7.7.3.As the basic state
> backend of Flink, this part
> seems to be ignored by people (although users cannot directly perceive it)
>
> I don't know how often this library is maintained and updated and the
> differences between them and Rocksdb. It would be best if there are
> relevant documents to tell me
>
> If the community is willing to upgrade this underlying component, I am
> willing to assist in this work as one of the volunteers.
> At the same time, I would like to ask if there is any chance for this
> project to be moved to the Apache Flink sub-project to allow more people to
> participate?
>
>
>
> --
> Best
>
> ConradJam
>
--
Best,
Yanfei