You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/04/22 11:59:00 UTC
[GitHub] [kafka] cadonna opened a new pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
cadonna opened a new pull request #10587:
URL: https://github.com/apache/kafka/pull/10587
RocksDB 6.8.1 is the newest version I could upgrade without
running into a SIGABRT issue with error message
"Pure virtual function called!" during Gradle builds.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#issuecomment-831220570
See also in the logs of the ARM build of this PR: https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka-pr/branches/PR-10587/runs/1/nodes/15/steps/132/log/?start=0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
chia7712 commented on pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#issuecomment-831223249
> What java version do you use? This error was observed with Java 8.
JDK8
Did you run tests to produce such error? I tested it by ‘./gradle clean build -x test’ and there is no error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#issuecomment-831224159
Yes, you have to run tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#issuecomment-832173985
Superseeded by https://github.com/apache/kafka/pull/10568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#discussion_r626469074
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
##########
@@ -99,35 +101,7 @@ public Env getEnv() {
@Override
public Options prepareForBulkLoad() {
- /* From https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ
- *
- * Q: What's the fastest way to load data into RocksDB?
- *
- * A: A fast way to direct insert data to the DB:
- *
- * 1. using single writer thread and insert in sorted order
- * 2. batch hundreds of keys into one write batch
- * 3. use vector memtable
- * 4. make sure options.max_background_flushes is at least 4
- * 5. before inserting the data,
- * disable automatic compaction,
- * set options.level0_file_num_compaction_trigger,
- * options.level0_slowdown_writes_trigger
- * and options.level0_stop_writes_trigger to very large.
- * After inserting all the data, issue a manual compaction.
- *
- * 3-5 will be automatically done if you call Options::PrepareForBulkLoad() to your option
- */
- // (1) not in our control
- // (2) is done via bulk-loading API
- // (3) skipping because, not done in actual PrepareForBulkLoad() code in https://github.com/facebook/rocksdb/blob/master/options/options.cc
- //columnFamilyOptions.setMemTableConfig(new VectorMemTableConfig());
- // (4-5) below:
- dbOptions.setMaxBackgroundFlushes(4);
- columnFamilyOptions.setDisableAutoCompactions(true);
- columnFamilyOptions.setLevel0FileNumCompactionTrigger(1 << 30);
- columnFamilyOptions.setLevel0SlowdownWritesTrigger(1 << 30);
- columnFamilyOptions.setLevel0StopWritesTrigger(1 << 30);
+ super.prepareForBulkLoad();
Review comment:
TBH, I do not understand why we set the options here instead of simply calling `super.prepareForBulkLoad()`. Also in the version that we currently use (5.18.4) the options are set like here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna closed pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
cadonna closed pull request #10587:
URL: https://github.com/apache/kafka/pull/10587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a change in pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on a change in pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#discussion_r619416799
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
##########
@@ -99,35 +101,7 @@ public Env getEnv() {
@Override
public Options prepareForBulkLoad() {
- /* From https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ
- *
- * Q: What's the fastest way to load data into RocksDB?
- *
- * A: A fast way to direct insert data to the DB:
- *
- * 1. using single writer thread and insert in sorted order
- * 2. batch hundreds of keys into one write batch
- * 3. use vector memtable
- * 4. make sure options.max_background_flushes is at least 4
- * 5. before inserting the data,
- * disable automatic compaction,
- * set options.level0_file_num_compaction_trigger,
- * options.level0_slowdown_writes_trigger
- * and options.level0_stop_writes_trigger to very large.
- * After inserting all the data, issue a manual compaction.
- *
- * 3-5 will be automatically done if you call Options::PrepareForBulkLoad() to your option
- */
- // (1) not in our control
- // (2) is done via bulk-loading API
- // (3) skipping because, not done in actual PrepareForBulkLoad() code in https://github.com/facebook/rocksdb/blob/master/options/options.cc
- //columnFamilyOptions.setMemTableConfig(new VectorMemTableConfig());
- // (4-5) below:
- dbOptions.setMaxBackgroundFlushes(4);
- columnFamilyOptions.setDisableAutoCompactions(true);
- columnFamilyOptions.setLevel0FileNumCompactionTrigger(1 << 30);
- columnFamilyOptions.setLevel0SlowdownWritesTrigger(1 << 30);
- columnFamilyOptions.setLevel0StopWritesTrigger(1 << 30);
+ super.prepareForBulkLoad();
Review comment:
Maybe still valuable to leave a comment on which rocksDB version the bulk loading behavior is updated to meet our needs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#issuecomment-831195589
@chia7712 What java version do you use? This error was observed with Java 8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
chia7712 commented on pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#issuecomment-831151970
> RocksDB 6.8.1 is the newest version I could upgrade without
running into a SIGABRT issue with error message
"Pure virtual function called!" during Gradle builds.
How to reproduce the error? I update the RocksDB JNI to 6.19.3 (with this patch) and the error message "Pure virtual function called!" does not appear when running gradle build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a change in pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1
Posted by GitBox <gi...@apache.org>.
cadonna commented on a change in pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#discussion_r626469825
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
##########
@@ -99,35 +101,7 @@ public Env getEnv() {
@Override
public Options prepareForBulkLoad() {
- /* From https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ
- *
- * Q: What's the fastest way to load data into RocksDB?
- *
- * A: A fast way to direct insert data to the DB:
- *
- * 1. using single writer thread and insert in sorted order
- * 2. batch hundreds of keys into one write batch
- * 3. use vector memtable
- * 4. make sure options.max_background_flushes is at least 4
- * 5. before inserting the data,
- * disable automatic compaction,
- * set options.level0_file_num_compaction_trigger,
- * options.level0_slowdown_writes_trigger
- * and options.level0_stop_writes_trigger to very large.
- * After inserting all the data, issue a manual compaction.
- *
- * 3-5 will be automatically done if you call Options::PrepareForBulkLoad() to your option
- */
- // (1) not in our control
- // (2) is done via bulk-loading API
- // (3) skipping because, not done in actual PrepareForBulkLoad() code in https://github.com/facebook/rocksdb/blob/master/options/options.cc
- //columnFamilyOptions.setMemTableConfig(new VectorMemTableConfig());
- // (4-5) below:
- dbOptions.setMaxBackgroundFlushes(4);
- columnFamilyOptions.setDisableAutoCompactions(true);
- columnFamilyOptions.setLevel0FileNumCompactionTrigger(1 << 30);
- columnFamilyOptions.setLevel0SlowdownWritesTrigger(1 << 30);
- columnFamilyOptions.setLevel0StopWritesTrigger(1 << 30);
+ super.prepareForBulkLoad();
Review comment:
What value would it have to leave a comment with the RocksDB version? I guess, we will not downgrade RocksDB less than the version we currently use, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org