You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/02/20 03:43:32 UTC
[GitHub] [spark] LuciferYang opened a new pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
LuciferYang opened a new pull request #35580:
URL: https://github.com/apache/spark/pull/35580
### What changes were proposed in this pull request?
The main change of this pr is upgrade `rockdbjni` to 6.29.3.
### Why are the changes needed?
This is the first version of `rockdbjni` that supports **Apple silicon/M1**
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079546619
Manually restarted GA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4.1
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079830794
[ac2f6f3](https://github.com/apache/spark/pull/35580/commits/ac2f6f30047f8372f55d24ae766ccc1e1d56682c) try updating rocksdbjni to 7.0.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35580:
URL: https://github.com/apache/spark/pull/35580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35580:
URL: https://github.com/apache/spark/pull/35580#discussion_r836251591
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -177,7 +177,7 @@ class RocksDB(
numKeysOnWritingVersion -= 1
}
}
- writeBatch.remove(key)
+ writeBatch.delete(key)
Review comment:
thanks @adamretter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1047202966
Since this is `RocksDB` update, cc @viirya and @HeartSaVioR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adamretter commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4.1
Posted by GitBox <gi...@apache.org>.
adamretter commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079774799
@LuciferYang I have also now released RocksJava 7.0.3 to Maven Central
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1050421671
Is there any update, @LuciferYang ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1058909472
@dongjoon-hyun Thank you for syncing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adamretter commented on a change in pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
Posted by GitBox <gi...@apache.org>.
adamretter commented on a change in pull request #35580:
URL: https://github.com/apache/spark/pull/35580#discussion_r836249719
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -177,7 +177,7 @@ class RocksDB(
numKeysOnWritingVersion -= 1
}
}
- writeBatch.remove(key)
+ writeBatch.delete(key)
Review comment:
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1046419218
> Before touching those parts, we need to make it sure that this RocksDB has no regression. Could you run benchmarks and include them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1046291756
Thank you, @LuciferYang .
Before touching those parts, we need to make it sure that this RocksDB has no regression. Could you run benchmarks and include them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adamretter commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
Posted by GitBox <gi...@apache.org>.
adamretter commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1080441722
@LuciferYang Just to mention that - most methods that move bytes now have Java `ByteBuffer` versions as well as the older `byte[]` versions. If you are not using the ByteBuffer versions, then you might like to try them, as we have seen improved performance with those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079897094
Leave these data to revert to the previous version
6.29.4.1 on Linux
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.356 0.281 0.519 0.519
dbCreation 4 77.093 3.549 300.771 300.771
naturalIndexCreateIterator 1024 0.006 0.002 1.429 0.008
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.071 0.008
naturalIndexDescendingIteration 1024 0.006 0.004 0.259 0.008
naturalIndexIteration 1024 0.006 0.004 0.063 0.009
randomDeleteIndexed 1024 0.026 0.019 0.319 0.035
randomDeletesNoIndex 1024 0.015 0.012 0.038 0.017
randomUpdatesIndexed 1024 0.081 0.032 30.418 0.087
randomUpdatesNoIndex 1024 0.063 0.059 0.629 0.068
randomWritesIndexed 1024 0.119 0.034 52.506 0.125
randomWritesNoIndex 1024 0.071 0.063 1.647 0.078
refIndexCreateIterator 1024 0.004 0.004 0.017 0.006
refIndexDescendingCreateIterator 1024 0.003 0.003 0.031 0.005
refIndexDescendingIteration 1024 0.007 0.005 0.048 0.009
refIndexIteration 1024 0.007 0.005 0.074 0.010
sequentialDeleteIndexed 1024 0.021 0.017 0.124 0.024
sequentialDeleteNoIndex 1024 0.015 0.012 0.050 0.017
sequentialUpdatesIndexed 1024 0.070 0.051 1.724 0.082
sequentialUpdatesNoIndex 1024 0.067 0.050 0.790 0.079
sequentialWritesIndexed 1024 0.073 0.058 2.074 0.083
sequentialWritesNoIndex 1024 0.057 0.051 3.193 0.062
```
6.29.4.1 on M1
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.563 0.428 0.861 0.861
dbCreation 4 312.788 134.860 737.943 737.943
naturalIndexCreateIterator 1024 0.003 0.001 0.963 0.003
naturalIndexDescendingCreateIterator 1024 0.003 0.002 0.048 0.003
naturalIndexDescendingIteration 1024 0.003 0.003 0.018 0.004
naturalIndexIteration 1024 0.004 0.003 0.084 0.011
randomDeleteIndexed 1024 0.017 0.013 0.306 0.020
randomDeletesNoIndex 1024 0.010 0.009 0.032 0.011
randomUpdatesIndexed 1024 0.044 0.020 15.237 0.041
randomUpdatesNoIndex 1024 0.019 0.015 0.533 0.022
randomWritesIndexed 1024 0.091 0.023 50.713 0.088
randomWritesNoIndex 1024 0.036 0.023 7.722 0.035
refIndexCreateIterator 1024 0.002 0.002 0.013 0.002
refIndexDescendingCreateIterator 1024 0.001 0.001 0.010 0.002
refIndexDescendingIteration 1024 0.010 0.003 6.005 0.005
refIndexIteration 1024 0.005 0.004 0.233 0.005
sequentialDeleteIndexed 1024 0.015 0.013 0.088 0.018
sequentialDeleteNoIndex 1024 0.013 0.011 0.325 0.015
sequentialUpdatesIndexed 1024 0.023 0.018 0.512 0.029
sequentialUpdatesNoIndex 1024 0.026 0.021 0.536 0.031
sequentialWritesIndexed 1024 0.041 0.023 1.817 0.082
sequentialWritesNoIndex 1024 0.030 0.024 2.137 0.039
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4.1
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079900938
@dongjoon-hyun Both 6.29.4.1 and 7.0.3 can pass GA. I chose 7.x because in addition to supporting M1, it is also a version compiled by Java 8, while 6.x is still compiled by Java 7.
<img width="824" alt="image" src="https://user-images.githubusercontent.com/1475305/160277315-2bfc8ba9-95c0-4f32-89b1-ab6b73822702.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1047351885
> Is it released?
From the release notes , it has been released for 3 days.
I'll go to the Facebook community to find the reason and I'll ping you when this pr ready for review.
Sorry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1056103458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079544671
Thank you, @adamretter !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4.1
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079819866
7.0.3
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.332 0.235 0.557 0.557
dbCreation 4 75.354 3.301 290.771 290.771
naturalIndexCreateIterator 1024 0.005 0.002 1.418 0.007
naturalIndexDescendingCreateIterator 1024 0.006 0.006 0.068 0.008
naturalIndexDescendingIteration 1024 0.006 0.004 0.261 0.007
naturalIndexIteration 1024 0.006 0.004 0.061 0.009
randomDeleteIndexed 1024 0.026 0.020 0.231 0.035
randomDeletesNoIndex 1024 0.014 0.012 0.040 0.017
randomUpdatesIndexed 1024 0.081 0.032 31.135 0.085
randomUpdatesNoIndex 1024 0.037 0.033 0.528 0.042
randomWritesIndexed 1024 0.119 0.034 54.771 0.122
randomWritesNoIndex 1024 0.042 0.036 1.448 0.048
refIndexCreateIterator 1024 0.005 0.004 0.020 0.006
refIndexDescendingCreateIterator 1024 0.003 0.003 0.029 0.005
refIndexDescendingIteration 1024 0.006 0.005 0.046 0.008
refIndexIteration 1024 0.007 0.005 0.072 0.009
sequentialDeleteIndexed 1024 0.021 0.017 0.100 0.026
sequentialDeleteNoIndex 1024 0.014 0.012 0.039 0.017
sequentialUpdatesIndexed 1024 0.047 0.040 0.747 0.058
sequentialUpdatesNoIndex 1024 0.040 0.029 0.730 0.049
sequentialWritesIndexed 1024 0.050 0.044 1.842 0.057
sequentialWritesNoIndex 1024 0.041 0.031 2.213 0.044
```
Let me compile with 7.0.3 locally first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1046419218
> Before touching those parts, we need to make it sure that this RocksDB has no regression. Could you run benchmarks and include them?
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1068775819
wait https://github.com/facebook/rocksdb/issues/9689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1050501939
> Is there any update, @LuciferYang ?
Unfortunately, I didn't get more useful information. Shall I close this pr and reopen it after the problem is solved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1046169599
After upgrade `rockdbjni` from 6.20.2 to 6.29.3 in this pr, then we can enable all UTs related to `rocksdb` on Apple silicon, is it better to enable these UTs in a new pr or to complete it together in this pr? what do you think @dongjoon-hyun ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1080444407
> @LuciferYang Just to mention that - most methods that move bytes now have Java `ByteBuffer` versions as well as the older `byte[]` versions. If you are not using the ByteBuffer versions, then you might like to try them, as we have seen improved performance with those.
Thanks for your advice. I'll try this later, maybe a new pr for Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adamretter commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
adamretter commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1059186913
@LuciferYang I think you missed an `@` when you mentioned me - so I did not get any notification. There is a problem at the moment with performing the releases due to a regression that was introduced by someone else trying to introduce Apple Silicon support - I am looking into this but it will take a little time. Details are here - https://github.com/facebook/rocksdb/pull/9254#issuecomment-1054598516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adamretter commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4
Posted by GitBox <gi...@apache.org>.
adamretter commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079494128
I just added 6.29.4.1 as the full-fat jar for 6.29.4 was missing the native windows binary. If you don't use the full fat jar or don't use Windows, then it's not an issue. If you do use Windows, you could use the Windows specific jar even then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079488090
Thank you, @adamretter !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1060985545
@dongjoon-hyun I found @adamretter is fixing the issue of java version release:
- [Fix RocksJava releases for macOS](https://github.com/facebook/rocksdb/pull/9662)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4.1
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079621888
> I just added 6.29.4.1 as the full-fat jar for 6.29.4 was missing the native windows binary. If you don't use the full fat jar or don't use Windows, then it's not an issue. If you do use Windows, you could use the Windows specific jar even then
Yes, 6.29.4 is missing `librocksdbjni-win64.dll`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1067536695
> For a record, `RocksDB` 7.0.2 is released.
>
> * https://github.com/facebook/rocksdb/releases/tag/v7.0.2
Yes, but 7.0.2 also cannot be downloaded through Maven repository, and it seems that @pdillinger and @adamretter wantsto release 6.29.4 first:
- https://github.com/facebook/rocksdb/pull/9678
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1050504265
Up to you. If that looks going to be released soon, I don't mind leaving it open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4.1
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079810337
Thanks @adamretter !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079489599
It seems that 6.29.4.1 exists too. Are we supposed to use `6.29.4.1`, @adamretter ?
- https://repo1.maven.org/maven2/org/rocksdb/rocksdbjni/6.29.4/
- https://repo1.maven.org/maven2/org/rocksdb/rocksdbjni/6.29.4.1/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1080891799
thanks all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1050530463
> Up to you. If that looks going to be released soon, I don't mind leaving it open.
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1056103458
> BTW, Apache Spark 3.3 branch cut is planned on May 15th.
@adamretter has not replied to the comments on GitHub for the time being. I just sent an email to him and hope to get a reply. If there is no release plan in the short term, I will close this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1056106870
@dongjoon-hyun OK ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1067522909
For a record, `RocksDB` 7.0.2 is released.
- https://github.com/facebook/rocksdb/releases/tag/v7.0.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1047230674
> [error] sbt.librarymanagement.ResolveException: Error downloading org.rocksdb:rocksdbjni:6.29.3
Is it released?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1046153708
It seems that the Maven central repository has not been synchronized to the release, and it will take a few days. Set this pr to `Draft` first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1055780525
BTW, Apache Spark 3.3 branch cut is planned on May 15th.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1056105798
Thank you for new information. You don't need to close this PR~
I just want to share our due date for Apache Spark 3.3.0.
If we miss 3.3, we can try this at 3.4, @LuciferYang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.3
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1058540179
We got the official confirmation like the following. `6.29.3` fails. We need a new version of rocksDB and rocksDB JNI will arrive after that one or two months.
- https://github.com/facebook/rocksdb/issues/9550#issuecomment-1058515642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adamretter commented on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4
Posted by GitBox <gi...@apache.org>.
adamretter commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079286010
@LuciferYang 6.29.4 is now available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #35580: [WIP][SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079546619
Manually restarted GA, I will update other files later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4.1
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35580:
URL: https://github.com/apache/spark/pull/35580#issuecomment-1079897094
6.29.4.1 on Linux
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.356 0.281 0.519 0.519
dbCreation 4 77.093 3.549 300.771 300.771
naturalIndexCreateIterator 1024 0.006 0.002 1.429 0.008
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.071 0.008
naturalIndexDescendingIteration 1024 0.006 0.004 0.259 0.008
naturalIndexIteration 1024 0.006 0.004 0.063 0.009
randomDeleteIndexed 1024 0.026 0.019 0.319 0.035
randomDeletesNoIndex 1024 0.015 0.012 0.038 0.017
randomUpdatesIndexed 1024 0.081 0.032 30.418 0.087
randomUpdatesNoIndex 1024 0.063 0.059 0.629 0.068
randomWritesIndexed 1024 0.119 0.034 52.506 0.125
randomWritesNoIndex 1024 0.071 0.063 1.647 0.078
refIndexCreateIterator 1024 0.004 0.004 0.017 0.006
refIndexDescendingCreateIterator 1024 0.003 0.003 0.031 0.005
refIndexDescendingIteration 1024 0.007 0.005 0.048 0.009
refIndexIteration 1024 0.007 0.005 0.074 0.010
sequentialDeleteIndexed 1024 0.021 0.017 0.124 0.024
sequentialDeleteNoIndex 1024 0.015 0.012 0.050 0.017
sequentialUpdatesIndexed 1024 0.070 0.051 1.724 0.082
sequentialUpdatesNoIndex 1024 0.067 0.050 0.790 0.079
sequentialWritesIndexed 1024 0.073 0.058 2.074 0.083
sequentialWritesNoIndex 1024 0.057 0.051 3.193 0.062
```
6.29.4.1 on M1
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.563 0.428 0.861 0.861
dbCreation 4 312.788 134.860 737.943 737.943
naturalIndexCreateIterator 1024 0.003 0.001 0.963 0.003
naturalIndexDescendingCreateIterator 1024 0.003 0.002 0.048 0.003
naturalIndexDescendingIteration 1024 0.003 0.003 0.018 0.004
naturalIndexIteration 1024 0.004 0.003 0.084 0.011
randomDeleteIndexed 1024 0.017 0.013 0.306 0.020
randomDeletesNoIndex 1024 0.010 0.009 0.032 0.011
randomUpdatesIndexed 1024 0.044 0.020 15.237 0.041
randomUpdatesNoIndex 1024 0.019 0.015 0.533 0.022
randomWritesIndexed 1024 0.091 0.023 50.713 0.088
randomWritesNoIndex 1024 0.036 0.023 7.722 0.035
refIndexCreateIterator 1024 0.002 0.002 0.013 0.002
refIndexDescendingCreateIterator 1024 0.001 0.001 0.010 0.002
refIndexDescendingIteration 1024 0.010 0.003 6.005 0.005
refIndexIteration 1024 0.005 0.004 0.233 0.005
sequentialDeleteIndexed 1024 0.015 0.013 0.088 0.018
sequentialDeleteNoIndex 1024 0.013 0.011 0.325 0.015
sequentialUpdatesIndexed 1024 0.023 0.018 0.512 0.029
sequentialUpdatesNoIndex 1024 0.026 0.021 0.536 0.031
sequentialWritesIndexed 1024 0.041 0.023 1.817 0.082
sequentialWritesNoIndex 1024 0.030 0.024 2.137 0.039
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35580: [SPARK-38257][BUILD] Upgrade `rockdbjni` to 6.29.4.1
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35580:
URL: https://github.com/apache/spark/pull/35580#discussion_r835884770
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -177,7 +177,7 @@ class RocksDB(
numKeysOnWritingVersion -= 1
}
}
- writeBatch.remove(key)
+ writeBatch.delete(key)
Review comment:
The `remove` series method of `WriteBatch` is identified as `@Deprecated` in 6.x and [deleted in 7.x](https://github.com/facebook/rocksdb/pull/9576), here use `delete` instead
Am I right @adamretter ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org