You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/26 09:27:38 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
dongjoon-hyun opened a new pull request #30934:
URL: https://github.com/apache/spark/pull/30934
### What changes were proposed in this pull request?
This PR aims to fix offset bug and improve compression codec test coverage.
### Why are the changes needed?
When the user choose a non-default codec, it causes a failure.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the extended test suite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751441766
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133400/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751439366
**[Test build #133400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133400/testReport)** for PR 30934 at commit [`1550b55`](https://github.com/apache/spark/commit/1550b55091d770232f73b88e00ed53e579bc2c7a).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751339673
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37986/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751408387
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37991/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751339986
**[Test build #133395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133395/testReport)** for PR 30934 at commit [`2c54d10`](https://github.com/apache/spark/commit/2c54d10ea133d4dd3956409f7caa726e245e07b8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751382025
**[Test build #133395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133395/testReport)** for PR 30934 at commit [`2c54d10`](https://github.com/apache/spark/commit/2c54d10ea133d4dd3956409f7caa726e245e07b8).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30934:
URL: https://github.com/apache/spark/pull/30934#discussion_r549526603
##########
File path: core/src/main/scala/org/apache/spark/storage/FallbackStorage.scala
##########
@@ -158,7 +158,7 @@ object FallbackStorage extends Logging {
val name = ShuffleDataBlockId(shuffleId, mapId, NOOP_REDUCE_ID).name
val dataFile = new Path(fallbackPath, s"$appId/$shuffleId/$name")
val f = fallbackFileSystem.open(dataFile)
- val size = nextOffset - 1 - offset
+ val size = nextOffset - offset
logDebug(s"To byte array $size")
Review comment:
Sounds good, thanks @dongjoon-hyun !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751408713
**[Test build #133400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133400/testReport)** for PR 30934 at commit [`1550b55`](https://github.com/apache/spark/commit/1550b55091d770232f73b88e00ed53e579bc2c7a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30934:
URL: https://github.com/apache/spark/pull/30934#discussion_r549527676
##########
File path: core/src/main/scala/org/apache/spark/storage/FallbackStorage.scala
##########
@@ -158,7 +158,7 @@ object FallbackStorage extends Logging {
val name = ShuffleDataBlockId(shuffleId, mapId, NOOP_REDUCE_ID).name
val dataFile = new Path(fallbackPath, s"$appId/$shuffleId/$name")
val f = fallbackFileSystem.open(dataFile)
- val size = nextOffset - 1 - offset
+ val size = nextOffset - offset
logDebug(s"To byte array $size")
Review comment:
Thank you, @mridulm !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751339986
**[Test build #133395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133395/testReport)** for PR 30934 at commit [`2c54d10`](https://github.com/apache/spark/commit/2c54d10ea133d4dd3956409f7caa726e245e07b8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751597528
Thank you for review and approval, @HyukjinKwon .
Please let me know if you have other comments, @MaxGekk and @mridulm .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751385084
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133395/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751408713
**[Test build #133400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133400/testReport)** for PR 30934 at commit [`1550b55`](https://github.com/apache/spark/commit/1550b55091d770232f73b88e00ed53e579bc2c7a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30934:
URL: https://github.com/apache/spark/pull/30934#discussion_r549049921
##########
File path: core/src/main/scala/org/apache/spark/storage/FallbackStorage.scala
##########
@@ -158,7 +158,7 @@ object FallbackStorage extends Logging {
val name = ShuffleDataBlockId(shuffleId, mapId, NOOP_REDUCE_ID).name
val dataFile = new Path(fallbackPath, s"$appId/$shuffleId/$name")
val f = fallbackFileSystem.open(dataFile)
- val size = nextOffset - 1 - offset
+ val size = nextOffset - offset
logDebug(s"To byte array $size")
Review comment:
Given this bug, I am wondering if we want to refactor `IndexShuffleBlockResolver.read` such that we can reuse it here as well ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751408387
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37991/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751905642
Thank you all!
Merged to master/3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751597528
Thank you for review and approval, @HyukjinKwon .
Please let me know if you have other comments, @MaxGekk , @viirya , @mridulm .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #30934:
URL: https://github.com/apache/spark/pull/30934
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751407988
The `lz4` (the default) compression codec seems to be robust on it while the other compression code is sensitive in some cases, @viirya .
> Hm, the bug is caused by wrong offset calculation or non-default codec?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751441766
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133400/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30934:
URL: https://github.com/apache/spark/pull/30934#discussion_r549043658
##########
File path: core/src/test/scala/org/apache/spark/storage/FallbackStorageSuite.scala
##########
@@ -227,43 +228,45 @@ class FallbackStorageSuite extends SparkFunSuite with LocalSparkContext {
}
}
- test("Newly added executors should access old data from remote storage") {
- sc = new SparkContext(getSparkConf(2, 0))
- withSpark(sc) { sc =>
- TestUtils.waitUntilExecutorsUp(sc, 2, 60000)
- val rdd1 = sc.parallelize(1 to 10, 2)
- val rdd2 = rdd1.map(x => (x % 2, 1))
- val rdd3 = rdd2.reduceByKey(_ + _)
- assert(rdd3.collect() === Array((0, 5), (1, 5)))
+ Seq("lz4", "snappy", "zstd").foreach { codec =>
Review comment:
Sure, I'll add that, @MaxGekk .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30934:
URL: https://github.com/apache/spark/pull/30934#issuecomment-751339673
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37986/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30934:
URL: https://github.com/apache/spark/pull/30934#discussion_r549049921
##########
File path: core/src/main/scala/org/apache/spark/storage/FallbackStorage.scala
##########
@@ -158,7 +158,7 @@ object FallbackStorage extends Logging {
val name = ShuffleDataBlockId(shuffleId, mapId, NOOP_REDUCE_ID).name
val dataFile = new Path(fallbackPath, s"$appId/$shuffleId/$name")
val f = fallbackFileSystem.open(dataFile)
- val size = nextOffset - 1 - offset
+ val size = nextOffset - offset
logDebug(s"To byte array $size")
Review comment:
Given this bug, I am wondering if we want to refactor `IndexShuffleBlockResolver` such that we can reuse it here as well ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30934:
URL: https://github.com/apache/spark/pull/30934#discussion_r549218772
##########
File path: core/src/main/scala/org/apache/spark/storage/FallbackStorage.scala
##########
@@ -158,7 +158,7 @@ object FallbackStorage extends Logging {
val name = ShuffleDataBlockId(shuffleId, mapId, NOOP_REDUCE_ID).name
val dataFile = new Path(fallbackPath, s"$appId/$shuffleId/$name")
val f = fallbackFileSystem.open(dataFile)
- val size = nextOffset - 1 - offset
+ val size = nextOffset - offset
logDebug(s"To byte array $size")
Review comment:
Maybe, for Apache Spark 3.2.0, @mridulm ? Currently, this PR is aiming to provide this fix for Apache Spark 3.1.0 RC before next Monday.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30934: [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30934:
URL: https://github.com/apache/spark/pull/30934#discussion_r548998380
##########
File path: core/src/test/scala/org/apache/spark/storage/FallbackStorageSuite.scala
##########
@@ -227,43 +228,45 @@ class FallbackStorageSuite extends SparkFunSuite with LocalSparkContext {
}
}
- test("Newly added executors should access old data from remote storage") {
- sc = new SparkContext(getSparkConf(2, 0))
- withSpark(sc) { sc =>
- TestUtils.waitUntilExecutorsUp(sc, 2, 60000)
- val rdd1 = sc.parallelize(1 to 10, 2)
- val rdd2 = rdd1.map(x => (x % 2, 1))
- val rdd3 = rdd2.reduceByKey(_ + _)
- assert(rdd3.collect() === Array((0, 5), (1, 5)))
+ Seq("lz4", "snappy", "zstd").foreach { codec =>
Review comment:
The doc of `IO_COMPRESSION_CODEC` says that it supports `lz4, lzf, snappy, and zstd`. Should we test `lzf` too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org