You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by GitBox <gi...@apache.org> on 2020/02/12 00:40:48 UTC
[GitHub] [samza] PanTheMan opened a new pull request #1275: SAMZA-2456:
Upgrade Rocksdb to 6.5.3
PanTheMan opened a new pull request #1275: SAMZA-2456: Upgrade Rocksdb to 6.5.3
URL: https://github.com/apache/samza/pull/1275
Symptom: Currently Samza is using RocksDB 5.7.3 version. This would cause a segment fault if the dependency order is not properly provisioned. This was reported in https://github.com/tensorflow/tensorflow/issues/22307
Cause: For example, if RockDB library is loaded before Tensorflow library, segment fault is triggered. In contrast, if Tensorflow library is loaded first, then there is no run-time error. Please see the following experiments for more details.
public class RocksDBTest{
public static void main(String[] args) throws InterruptedException
{ //System.load("/home/yangzhan/libs/librocksdbjni7259590262328036607.so"); // RocksDB v=5.0.1 => Failed // System.load("/home/yangzhan/libs/librocksdbjni575105474288895339.so"); // RocksDB v=5.7.3 => Failed System.load("/home/yangzhan/libs/librocksdbjni6484631531885492614.so"); // RockdsDB v=5.15.10 => Passed System.load("/home/yangzhan/libs/libtensorflow_framework.so"); }
}
Changes: I upgraded RocksDB to the latest (6.5.3) to fix this issue. I tested (see below) to ensure that this upgrade doesn't change any current RocksDB use. A list of RocksDB changes since 5.7.3 can be found here https://github.com/facebook/rocksdb/releases.
Tests: Ran the script above to verify this fixes the issue. Also tested to verify that this upgrade doesn't require any code changes to our RocksDbKeyValueStore by running:
./gradlew clean test
./gradlew build test
bin/integration-tests.sh for standalone and yarn
mint build && mint pcl in samza-li
Ran Brett's store workload job in samza release tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] mynameborat commented on issue #1275: SAMZA-2456: Upgrade
Rocksdb to 6.5.3
Posted by GitBox <gi...@apache.org>.
mynameborat commented on issue #1275: SAMZA-2456: Upgrade Rocksdb to 6.5.3
URL: https://github.com/apache/samza/pull/1275#issuecomment-587992960
Thanks for attaching the report. It will be good to summarize and add it to the upcoming release notes to make sure the community is aware of it.
Looks good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] PanTheMan commented on issue #1275: SAMZA-2456: Upgrade
Rocksdb to 6.5.3
Posted by GitBox <gi...@apache.org>.
PanTheMan commented on issue #1275: SAMZA-2456: Upgrade Rocksdb to 6.5.3
URL: https://github.com/apache/samza/pull/1275#issuecomment-587991579
Good point, attached is the performance results
[RocksDB performance.xlsx](https://github.com/apache/samza/files/4222687/RocksDB.performance.xlsx). The results are from using TestKeyValuePerformance.scala code and Brett's store workload job vs the results from a release test email for the master branch. Overall it seems like for an average use case, the new RocksDB version is faster however when more messages are written of more than 1kb size, there is some performance lost.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] shanthoosh merged pull request #1275: SAMZA-2456: Upgrade
Rocksdb to 6.5.3
Posted by GitBox <gi...@apache.org>.
shanthoosh merged pull request #1275: SAMZA-2456: Upgrade Rocksdb to 6.5.3
URL: https://github.com/apache/samza/pull/1275
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] mynameborat commented on issue #1275: SAMZA-2456: Upgrade
Rocksdb to 6.5.3
Posted by GitBox <gi...@apache.org>.
mynameborat commented on issue #1275: SAMZA-2456: Upgrade Rocksdb to 6.5.3
URL: https://github.com/apache/samza/pull/1275#issuecomment-585903268
@PanTheMan Did we notice any changes to the store perf metrics with the version change? It will be good to attach the graphs or reports if available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services