You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/13 21:27:57 UTC
[GitHub] [spark] ankurdave opened a new pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
ankurdave opened a new pull request #29744:
URL: https://github.com/apache/spark/pull/29744
### What changes were proposed in this pull request?
When BytesToBytesMap is close to `MAX_CAPACITY` and needs to grow, `numKeys >= growthThreshold` is true but `longArray.size() / 2 < MAX_CAPACITY` is false. This correctly prevents the map from growing, but `canGrowArray` incorrectly remains true. Therefore the map keeps accepting new keys and exceeds its growth threshold. If we attempt to spill the map in this state, the UnsafeKVExternalSorter will not be able to reuse the long array for sorting. By this point the task has typically consumed all available memory, so the allocation of the new pointer array is likely to fail.
This PR fixes the issue by setting `canGrowArray` to false in this case. This prevents the map from accepting new elements when it cannot grow to accommodate them.
### Why are the changes needed?
Without this change, hash aggregations will fail when the number of groups per task is close to `MAX_CAPACITY = 2^29` (approximately 500 million), and when the grouping aggregation is the only memory-consuming operator in its stage.
For example, the final aggregation in `SELECT COUNT(DISTINCT id) FROM tbl` fails when `tbl` contains 1 billion distinct values and when `spark.sql.shuffle.partitions=1`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Reproducing this issue requires building a very large BytesToBytesMap. Because this is infeasible to do in a unit test, this PR was tested manually by adding the following test to AbstractBytesToBytesMapSuite. Before this PR, the test failed in 8.5 minutes. With this PR, the test passes in 1.5 minutes.
```java
public abstract class AbstractBytesToBytesMapSuite {
// ...
@Test
public void stopAcceptingNewKeysNearMaxCapacity() {
TestMemoryManager memoryManager2 =
new TestMemoryManager(
new SparkConf()
.set(package$.MODULE$.MEMORY_OFFHEAP_ENABLED(), true)
.set(package$.MODULE$.MEMORY_OFFHEAP_SIZE(), 25600 * 1024 * 1024L)
.set(package$.MODULE$.SHUFFLE_SPILL_COMPRESS(), false)
.set(package$.MODULE$.SHUFFLE_COMPRESS(), false));
TaskMemoryManager taskMemoryManager2 = new TaskMemoryManager(memoryManager2, 0);
final long pageSizeBytes = 8000000 + 8; // 8 bytes for end-of-page marker
final BytesToBytesMap map = new BytesToBytesMap(taskMemoryManager2, 1024, pageSizeBytes);
try {
// Fill the map up to its maximum capacity.
for (long i = 0; i < BytesToBytesMap.MAX_CAPACITY; i++) {
if (i % (1024 * 1024) == 0) System.out.println("Inserting element " + i);
final long[] value = new long[]{i};
BytesToBytesMap.Location loc = map.lookup(value, Platform.LONG_ARRAY_OFFSET, 8);
Assert.assertFalse(loc.isDefined());
boolean success =
loc.append(value, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8);
if (!success) break;
}
// The map should stop accepting new keys once it has reached its growth threshold.
long capacity = map.getArray().size() / 2;
Assert.assertTrue(map.numKeys() <= capacity / 2);
map.free();
} finally {
map.free();
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487584414
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map cannot grow because doing so would cause it to exceed MAX_CAPACITY. Instead, we
+ // prevent the map from accepting any more new elements.
+ canGrowArray = false;
Review comment:
I agree. Once we reach `MAX_CAPACITY`, we have a choice: we can either (a) exceed the load factor, or (b) we can spill.
The code currently does (a). If the number of groups ultimately turns out to be less than `MAX_CAPACITY`, then this may avoid spilling. (Performance will degrade as the load factor approaches 1.0, but it will still be better than spilling.) However, if the number of groups is larger than `MAX_CAPACITY`, or if we need more memory to store the records, then we will be unable to spill and the query will fail.
It seems to me that it would be better to prevent queries from failing, as (b) does. This does mean we would spill earlier for a narrow range of queries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691747199
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692277460
> As for rerunning the GH tests: Unfortunately the only way currently is to reopen the PR: Close it in Github and reopen it -- that will force all the GH tests to be rerun.
I think we can rerun GitHub Actions without closing the PR. Just triggered the rerun.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691729631
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691728107
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692276747
@ankurdave
1. For this PR, one of them(Jenkins/GithubAction) is enough for merging.
2. For Guthub Action retrigger, all comitter can retrigger at Github action detail page
3. It would be great if we can have this at branch-3.0/2.4 because this bug has been existing since 2.0 or older.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691776643
**[Test build #128609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128609/testReport)** for PR 29744 at commit [`ec51e65`](https://github.com/apache/spark/commit/ec51e65585e3fb5fa982ea51b7d371672808f583).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691777161
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487656777
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,12 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
Review comment:
+1 too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691743230
**[Test build #128609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128609/testReport)** for PR 29744 at commit [`ec51e65`](https://github.com/apache/spark/commit/ec51e65585e3fb5fa982ea51b7d371672808f583).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487591356
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map has reached its growth threshold, but is already at MAX_CAPACITY and cannot
+ // grow. Instead, we prevent the map from accepting any more new elements.
Review comment:
Can we be more clear in the comment, like:
```
... Instead, we prevent the map from accepting any more new elements to
make sure we don't exceed load factor, so we can reuse the array in sorting later.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691869773
**[Test build #128626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128626/testReport)** for PR 29744 at commit [`f610fbe`](https://github.com/apache/spark/commit/f610fbee9398b59e8bad7b8b2ec72fb72603352d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691742562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691742406
**[Test build #128608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128608/testReport)** for PR 29744 at commit [`00e4431`](https://github.com/apache/spark/commit/00e443106672b3214f83fe1fb32eca04c28d8ebf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691729631
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691775497
**[Test build #128608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128608/testReport)** for PR 29744 at commit [`00e4431`](https://github.com/apache/spark/commit/00e443106672b3214f83fe1fb32eca04c28d8ebf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692148091
Just curious - changing the guard condition at the beginning of `append` from `numKeys == MAX_CAPACITY - 1` to `numKeys >= MAX_CAPACITY - 1` would suffice ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692278740
@agrawaldevesh . Not every CI has a large enough memory. It can lead us more flakiness situation. You can add the test in your downstream CI environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691815973
Thank you for pinging me, @ankurdave . And, thank you for providing the test example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
ankurdave commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691728640
cc @viirya @cloud-fan @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691965900
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691729520
**[Test build #128606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128606/testReport)** for PR 29744 at commit [`de0f9fe`](https://github.com/apache/spark/commit/de0f9fe895bc9a6c14e69a8d4a4717a6d8faaa71).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691771714
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692257967
Thanks all for the quick reviews! A couple questions for maintainers of core:
1. The Jenkins tests passed but the GitHub Actions tests seem flaky (50% success rate so far). Do we require both to pass? If so, is there a good way to rerun the GitHub Actions tests?
2. What is the feeling on backporting this fix? I don't feel strongly either way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691765020
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691749164
**[Test build #128605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128605/testReport)** for PR 29744 at commit [`c3afb09`](https://github.com/apache/spark/commit/c3afb09f942e0465b05155fda18fb24eabc212f8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691739021
**[Test build #128607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128607/testReport)** for PR 29744 at commit [`7d76463`](https://github.com/apache/spark/commit/7d76463e94b62c31b563c00e68752139e34183b7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691747201
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128606/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691776030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487588391
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map cannot grow because doing so would cause it to exceed MAX_CAPACITY. Instead, we
+ // prevent the map from accepting any more new elements.
+ canGrowArray = false;
Review comment:
Regarding the memory setting: I agree that whether or not an OOM will occur is dependent on the memory setting, but I would argue that (1) the user should not be required to tune memory settings to avoid failing queries, and (2) many reasonable settings will cause an OOM.
The problem arises when there are between 2^28+1 and 2^29 keys. If we need to spill, `UnsafeKVExternalSorter` will attempt to allocate between 8 and 16 GiB. Therefore the user will either need to increase the memory per task by approximately this amount so that this allocation will succeed, or decrease the memory per task sufficiently that the map will never reach `MAX_CAPACITY` in the first place. This seems to create quite a wide range of problematic settings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691728807
**[Test build #128605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128605/testReport)** for PR 29744 at commit [`c3afb09`](https://github.com/apache/spark/commit/c3afb09f942e0465b05155fda18fb24eabc212f8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691742406
**[Test build #128608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128608/testReport)** for PR 29744 at commit [`00e4431`](https://github.com/apache/spark/commit/00e443106672b3214f83fe1fb32eca04c28d8ebf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691965900
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692322905
Oops. @ankurdave . Sorry, but could you make a backporting PR to branch-2.4 once more?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691758270
@SparkQA retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692310011
Merged to master/3.0/2.4. Thank you, @ankurdave and all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692158060
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691728107
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692332890
Opened https://github.com/apache/spark/pull/29753 to backport to branch-2.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692148091
Sratch that. ~~Just curious - changing the guard condition at the beginning of `append` from `numKeys == MAX_CAPACITY - 1` to `numKeys >= MAX_CAPACITY - 1` would suffice ?~~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691749592
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692157435
**[Test build #128661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128661/testReport)** for PR 29744 at commit [`680c4cf`](https://github.com/apache/spark/commit/680c4cf9437f8d8878de46446d4d4c18f31b88e9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691870634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691758270
@SparkQA retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487655942
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,12 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
Review comment:
+1 for @cloud-fan 's refactoring suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692278740
@agrawaldevesh . Not every CI has a large enough memory. It can lead us more flaky situation. You can add the test in your downstream CI environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692250924
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691870634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691739021
**[Test build #128607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128607/testReport)** for PR 29744 at commit [`7d76463`](https://github.com/apache/spark/commit/7d76463e94b62c31b563c00e68752139e34183b7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691771182
**[Test build #128607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128607/testReport)** for PR 29744 at commit [`7d76463`](https://github.com/apache/spark/commit/7d76463e94b62c31b563c00e68752139e34183b7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691739204
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691728807
**[Test build #128605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128605/testReport)** for PR 29744 at commit [`c3afb09`](https://github.com/apache/spark/commit/c3afb09f942e0465b05155fda18fb24eabc212f8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487582849
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map cannot grow because doing so would cause it to exceed MAX_CAPACITY. Instead, we
+ // prevent the map from accepting any more new elements.
+ canGrowArray = false;
Review comment:
I think we won't exceed `MAX_CAPACITY` at all. We have some restrictions to prevent it.
The root cause, I think, is once the map reaches `MAX_CAPACITY`, we won't grow the map even the `numKeys` is more than `growthThreshold`. So we could add elements to the map to more than load factor (0.5) of it. The map cannot be used for sorting later in `UnsafeKVExternalSorter`. Although `UnsafeKVExternalSorter` will allocate new array in the case, it could hit OOM and fails the query.
This change favors reusing the array for sorting, but it prevents the map accepting new elements more early, half of the array is not used after the map capacity reaches `MAX_CAPACITY`. But by doing this, we will do spilling early too.
When I was fixing the customer application before, we don't see the OOM even the payload reached `MAX_CAPACITY`. Is it possibly due to the memory setting on your payload/cluster?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487650024
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,12 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
Review comment:
it should be more readable to combine the common condition:
```
// The map has reached its growth threshold, try to grow it
if (numKeys >= growthThreshold) {
// We use two array entries per key ...
if (longArray.size() / 2 < MAX_CAPACITY) {
...
} else {
// It's already at MAX_CAPACITY and cannot grow ...
canGrowArray = false;
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691765020
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691749592
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691747199
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691869773
**[Test build #128626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128626/testReport)** for PR 29744 at commit [`f610fbe`](https://github.com/apache/spark/commit/f610fbee9398b59e8bad7b8b2ec72fb72603352d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691964726
**[Test build #128626 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128626/testReport)** for PR 29744 at commit [`f610fbe`](https://github.com/apache/spark/commit/f610fbee9398b59e8bad7b8b2ec72fb72603352d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487580161
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map needs to grow, but this would cause it to exceed MAX_CAPACITY. Prevent the map
+ // from accepting any more new elements.
+ canGrowArray = false;
Review comment:
The map won't exceed `MAX_CAPACITY`, I think. This just prevents the map accepting new elements more early.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692249729
**[Test build #128661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128661/testReport)** for PR 29744 at commit [`680c4cf`](https://github.com/apache/spark/commit/680c4cf9437f8d8878de46446d4d4c18f31b88e9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487592464
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map has reached its growth threshold, but is already at MAX_CAPACITY and cannot
+ // grow. Instead, we prevent the map from accepting any more new elements.
Review comment:
Okay, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692157435
**[Test build #128661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128661/testReport)** for PR 29744 at commit [`680c4cf`](https://github.com/apache/spark/commit/680c4cf9437f8d8878de46446d4d4c18f31b88e9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
ankurdave commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487580646
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map needs to grow, but this would cause it to exceed MAX_CAPACITY. Prevent the map
+ // from accepting any more new elements.
+ canGrowArray = false;
Review comment:
Hmm, perhaps the comment's wording is confusing. I intended to convey that, if the map were to grow, then its capacity would exceed MAX_CAPACITY. Therefore it cannot grow and we must prevent it from accepting any more new elements.
I reworded the comment to clarify this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691742562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691777161
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691739204
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692270160
@ankurdave ... Great fix !.
I am curious if the test you mentioned in the PR description should actually be checked in ? The only downside seems to be that it would take 1.5 minutes to pass. Perhaps there is a mode for "slow tests".
As for rerunning the GH tests: Unfortunately the only way currently is to reopen the PR: Close it in Github and reopen it -- that will force all the GH tests to be rerun.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691743230
**[Test build #128609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128609/testReport)** for PR 29744 at commit [`ec51e65`](https://github.com/apache/spark/commit/ec51e65585e3fb5fa982ea51b7d371672808f583).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692158060
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-692250924
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29744:
URL: https://github.com/apache/spark/pull/29744
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487582849
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map cannot grow because doing so would cause it to exceed MAX_CAPACITY. Instead, we
+ // prevent the map from accepting any more new elements.
+ canGrowArray = false;
Review comment:
I think we won't exceed `MAX_CAPACITY` at all. We have some restrictions to prevent it.
The root cause, I think, is once the map reaches `MAX_CAPACITY`, we won't grow the map even the `numKeys` is more than `growthThreshold`. So we could add elements to the map to more than load factor (0.5) of it. The map cannot be used for sorting later in `UnsafeKVExternalSorter`. Although `UnsafeKVExternalSorter` will allocate new array in the case, it could hit OOM and fails the query.
This change favors reusing the array for sorting, but it prevents the map accepting new elements more early, half of the array is not used after the map capacity reaches `MAX_CAPACITY`. But by doing this, we will do spilling early too.
When I was fixing the customer application before, we don't see the OOM even the payload reached `MAX_CAPACITY`. Is it possibly due to the memory setting on your payload?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691776030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691771714
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487699572
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,12 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
Review comment:
Thanks, done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap from exceeding growth threshold near MAX_CAPACITY
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691729520
**[Test build #128606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128606/testReport)** for PR 29744 at commit [`de0f9fe`](https://github.com/apache/spark/commit/de0f9fe895bc9a6c14e69a8d4a4717a6d8faaa71).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ankurdave commented on a change in pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
ankurdave commented on a change in pull request #29744:
URL: https://github.com/apache/spark/pull/29744#discussion_r487592358
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -816,6 +816,10 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
} catch (SparkOutOfMemoryError oom) {
canGrowArray = false;
}
+ } else if (numKeys >= growthThreshold && longArray.size() / 2 >= MAX_CAPACITY) {
+ // The map has reached its growth threshold, but is already at MAX_CAPACITY and cannot
+ // grow. Instead, we prevent the map from accepting any more new elements.
Review comment:
Done. I also changed the wording to reference UnsafeKVExternalSorter explicitly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29744: [SPARK-32872][CORE] Prevent BytesToBytesMap at MAX_CAPACITY from exceeding growth threshold
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29744:
URL: https://github.com/apache/spark/pull/29744#issuecomment-691746970
**[Test build #128606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128606/testReport)** for PR 29744 at commit [`de0f9fe`](https://github.com/apache/spark/commit/de0f9fe895bc9a6c14e69a8d4a4717a6d8faaa71).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org