You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2017/04/09 23:58:36 UTC
[GitHub] flink pull request #3704: [FLINK-5756] Replace RocksDB dependency with FRock...
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/3704
[FLINK-5756] Replace RocksDB dependency with FRocksDB
@StefanRRichter has created a custom RocksDB release that fixes FLINK-5756.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink flink5756
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3704.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3704
----
commit b05c595a37ea5b3a08ef4a11d9259eb7aabee005
Author: Robert Metzger <rm...@apache.org>
Date: 2017-04-09T20:05:08Z
[FLINK-5756] Replace RocksDB dependency with FRocksDB
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by StefanRRichter <gi...@git.apache.org>.
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/3704
Unfortunately, all newer versions of RocksDB are broken for some functions when used through the Java API and will segfault when using merge operators. This is the same version as we previously used in Flink, so there is no regression. Furthermore, it is possible to use a different compression scheme. We might at some point try to build our custom version against zlib-1.2.9.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by StefanRRichter <gi...@git.apache.org>.
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/3704
LGTM. +1 for merging this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3704
I'll merge the change now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on the issue:
https://github.com/apache/flink/pull/3704
I hope there is a plan to contribute back to rocksdb and we could come back to bring back dependencies on the fixed version. Relying on modified lib never good practice
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on the issue:
https://github.com/apache/flink/pull/3704
Yeah totally understand the reason. Just want to make sure we have plan to move back as soon as RocksDB usable for Flink.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/3704
@SyinChwunLeo We will try and contribute the patch to RocksDB and will also soon try and move to a newer RocksDB version, as soon as its Java API works again for the required functions. The RocksDB folks mentioned that the next release of RocksDB is quite soon and should fix that.
That will hopefully address the issue. Until then, we cannot upgrade, unfortunately :-(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by SyinChwunLeo <gi...@git.apache.org>.
Github user SyinChwunLeo commented on the issue:
https://github.com/apache/flink/pull/3704
I notice that RocksDB in 4.11.2 version is dependent on zlib-1.2.8, unfortunately, there is a security leak in zlib-1.2.8. https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2016-9840\uff0cand fixed in 1.2.9 version. Only RocksDB's latest version 5.2.1 using zlib-1.2.9. Is it possible to update rocksDB to 5.2.1 in flink ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3704: [FLINK-5756] Replace RocksDB dependency with FRock...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3704
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by SyinChwunLeo <gi...@git.apache.org>.
Github user SyinChwunLeo commented on the issue:
https://github.com/apache/flink/pull/3704
OK\uff0c FRocksDB is only available in master branch or we can also use it in flink-1.2.0?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3704
@SyinChwunLeo: we'll merge the fix to both master and the release 1.2 branch.
But you can start using FrocksDB right away, because its available in maven central.
You just have to exclude the rocksdb dependency from Flink and include frocksdb:
```diff
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb_${scala.binary.version}</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.rocksdb</groupId>
+ <artifactId>rocksdbjni</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>com.data-artisans</groupId>
+ <artifactId>frocksdbjni</artifactId>
+ <version>4.11.2-artisans</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/flink/pull/3704
Could you tell what modifications are done in "FRocksDB" and post the url of source code repository?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by StefanRRichter <gi...@git.apache.org>.
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/3704
We are using the newer StringAppendTESTOperator over the StringAppendOperator in rocksdbjni, because the merge algorithm used in StringAppendOperator has quadratic complexity in the number of merged elements. The modification is just instantiating the other implementation by providing a different parameter to a factory method.
The source code is here: https://github.com/dataArtisans/frocksdb/tree/master <https://github.com/dataArtisans/frocksdb/tree/master>
> Am 09.04.2017 um 18:51 schrieb Tao Wang <no...@github.com>:
>
> Could you tell what modifications are done in "FRocksDB" and post the url of source code repository?
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/apache/flink/pull/3704#issuecomment-292831352>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACB6n1O1jAlsbSln9KuaKonQMqP8VZloks5ruYsugaJpZM4M4M1I>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/3704
@hsaputra There is a plan to do that and we are in touch with the RocksDB folks. The latest RocksDB master does not work for Flink though, currently, so we needed a custom backport to an earlier RocksDB version...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---