You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/26 08:38:34 UTC
[GitHub] [spark] LuciferYang opened a new pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
LuciferYang opened a new pull request #33848:
URL: https://github.com/apache/spark/pull/33848
### What changes were proposed in this pull request?
The Guava Cache use in Spark with weight eviction mechanism has risk of memory leak because `LocalCache` weight eviction does not work when `maxSegmentWeight is >= Int.MAX_VALUE` [Guava#1761](https://github.com/google/guava/issues/1761)
Now these pr is WIP and only give a new UT can reproduce this issue.
### Why are the changes needed?
Bug fix
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919125546
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47759/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909323721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912222480
**[Test build #142956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142956/testReport)** for PR 33848 at commit [`3b965c2`](https://github.com/apache/spark/commit/3b965c2248d35ff2c3c12fb81124edb8f9e303ba).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906434766
**[Test build #142818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142818/testReport)** for PR 33848 at commit [`8ec7525`](https://github.com/apache/spark/commit/8ec752568f6ce075bf3eae9a5b856ddea44708b5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909373840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142888/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912240845
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47456/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906600928
**[Test build #142819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142819/testReport)** for PR 33848 at commit [`6c4b06d`](https://github.com/apache/spark/commit/6c4b06d3359be03cda877296dd0f169df025c87d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912249095
**[Test build #142960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142960/testReport)** for PR 33848 at commit [`5524c68`](https://github.com/apache/spark/commit/5524c685991e417fc287d444356b684b24dc03ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906538739
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142818/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912249095
**[Test build #142960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142960/testReport)** for PR 33848 at commit [`5524c68`](https://github.com/apache/spark/commit/5524c685991e417fc287d444356b684b24dc03ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r701575224
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
@srowen @dongjoon-hyun
As comments above:
1. Upgrading the guava version will have a potential interfere with other libs like Hadoop
2. Use Caffeine will introduce new dependent libs. (#33784)
Therefore, I added a maximum limit to the related configurations to avoid the potential bug of Guava Cache [Guava#1761](https://github.com/google/guava/issues/1761)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919134630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47759/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906215520
Maybe there the following ways can solve the problem:
1. Upgrade the Guava version can fix this bug
2. Or we can avoid this issue by limiting the configuration related to `maximumWeight` to the security range
I will try to upgrade Guava version first after 98458fd build successful
Do you have any better suggestions ? @HyukjinKwon @dongjoon-hyun @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r697134008
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
Ya, I agree with @srowen 's comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r696426276
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java
##########
@@ -110,6 +118,48 @@ public void testSortShuffleBlocks() throws IOException {
}
}
+ @Test
+ public void testShuffleIndexCacheEvictionBehavior() throws IOException, ExecutionException {
+ Map<String, String> config = new HashMap<>();
+ String indexCacheSize = "8192m";
+ config.put("spark.shuffle.service.index.cache.size", indexCacheSize);
Review comment:
configure `spark.shuffle.service.index.cache.size` with 8g and the default concurrencyLevel is 4, so the `maxSegmentWeight` Local Cache is `Int.MAX_VALUE`, these condition will trigger the bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906324133
**[Test build #142802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142802/testReport)** for PR 33848 at commit [`98458fd`](https://github.com/apache/spark/commit/98458fd364927627166a376ee1a145dcad1f2693).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #33848:
URL: https://github.com/apache/spark/pull/33848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906412169
8ec7525 upgrade Guava to 30.1.1-jre to fix this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909220519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912222480
**[Test build #142956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142956/testReport)** for PR 33848 at commit [`3b965c2`](https://github.com/apache/spark/commit/3b965c2248d35ff2c3c12fb81124edb8f9e303ba).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906214460
from the debug view we found that there are 2 segment.totalWeight is a negative value:

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912237140
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47456/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906469129
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906532320
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47321/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906412169
8ec7525 upgrade Guava to 30.1.1-jre to fix `LocalCache weight eviction does not work when maxSegmentWeight is >= Int.MAX_VALUE` issue and change UT to check the eviction behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906258003
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47303/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906339231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142802/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919184857
**[Test build #143256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143256/testReport)** for PR 33848 at commit [`f200546`](https://github.com/apache/spark/commit/f200546c1e6df47210d9d68bb55d11a9cc529035).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class NettyLogger `
* `public final class LZ4Compressor `
* `public final class LZ4Factory `
* `public final class LZ4SafeDecompressor `
* `class DatetimeNTZOps(DatetimeOps):`
* `class DatetimeNTZConverter(object):`
* `public final class AlwaysFalse extends Filter `
* `public final class AlwaysTrue extends Filter `
* `public final class And extends BinaryFilter `
* `abstract class BinaryComparison extends Filter `
* `abstract class BinaryFilter extends Filter `
* `public final class EqualNullSafe extends BinaryComparison `
* `public final class EqualTo extends BinaryComparison `
* `public abstract class Filter implements Expression `
* `public final class GreaterThan extends BinaryComparison `
* `public final class GreaterThanOrEqual extends BinaryComparison `
* `public final class In extends Filter `
* `public final class IsNotNull extends Filter `
* `public final class IsNull extends Filter `
* `public final class LessThan extends BinaryComparison `
* `public final class LessThanOrEqual extends BinaryComparison `
* `public final class Not extends Filter `
* `public final class Or extends BinaryFilter `
* `public final class StringContains extends StringPredicate `
* `public final class StringEndsWith extends StringPredicate `
* `abstract class StringPredicate extends Filter `
* `public final class StringStartsWith extends StringPredicate `
* `case class CastTimestampNTZToLong(child: Expression) extends TimestampToLongBase `
* `case class ILike(`
* `class SQLOpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](`
* `case class OptimizeSkewedJoin(`
* `case class SkewJoinAwareCost(`
* `case class SimpleCostEvaluator(forceOptimizeSkewedJoin: Boolean) extends CostEvaluator `
* `case class EnsureRequirements(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906482213
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909260228
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47391/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r701575224
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
@srowen @dongjoon-hyun
As comments above:
1. Upgrading the Guava version will have a potential interfere with other libs like Hadoop
2. Use Caffeine will introduce new dependent libs. (#33784)
Therefore, I added a maximum limit to the related configurations to avoid the potential bug of Guava Cache [Guava#1761](https://github.com/google/guava/issues/1761)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912318759
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142960/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906481210
Through https://github.com/apache/spark/pull/30022 , I found that upgrading guava has some impact on `hadoop-2.x`, so I will revet the change of 8ec7525 and waiting for others advice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r698124577
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
If upgradingGuava version is not a good way, it seems that it is not easy to fix this potential bug. I have manually verified that `Caffeine` does not have this problem, but maybe we are not willing to accept it because https://github.com/apache/spark/pull/33784#event-5190314002 .
So can we prevent users from using a problematic config by adding some config value check?
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
If upgradingGuava version is not a good way, it seems that it is not easy to fix this potential bug. I have manually verified that `Caffeine` does not have this problem, but maybe we are not willing to accept it because https://github.com/apache/spark/pull/33784
So can we prevent users from using a problematic config by adding some config value check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912277433
**[Test build #142956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142956/testReport)** for PR 33848 at commit [`3b965c2`](https://github.com/apache/spark/commit/3b965c2248d35ff2c3c12fb81124edb8f9e303ba).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909220519
**[Test build #142888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142888/testReport)** for PR 33848 at commit [`0fade2a`](https://github.com/apache/spark/commit/0fade2abb70ff13776275cc3eec4f4e5286bc5f0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906532320
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47321/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919186519
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143256/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906617684
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142819/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r696684100
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
@HyukjinKwon @dongjoon-hyun @srowen It seems that upgrading the Guava version can solve the problem, but it will also add additional dependencies. I'm not sure whether it is valuable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909220519
**[Test build #142888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142888/testReport)** for PR 33848 at commit [`0fade2a`](https://github.com/apache/spark/commit/0fade2abb70ff13776275cc3eec4f4e5286bc5f0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906215520
Maybe there the following ways can solve the problem:
1. Upgrade the Guava version can fix this bug
2. Or we can avoid this issue by limiting the configuration related to `maximumWeight` to the security range
I will try to upgrade Guava version first after 98458fd build successful
Do you have any better suggestions ? @HyukjinKwon @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906434766
**[Test build #142818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142818/testReport)** for PR 33848 at commit [`8ec7525`](https://github.com/apache/spark/commit/8ec752568f6ce075bf3eae9a5b856ddea44708b5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906486196
**[Test build #142819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142819/testReport)** for PR 33848 at commit [`6c4b06d`](https://github.com/apache/spark/commit/6c4b06d3359be03cda877296dd0f169df025c87d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906259745
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47303/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906481210
Through https://github.com/apache/spark/pull/30022 , I found that upgrading guava has some impact on `hadoop-2.x`, so I will revet the change of 8ec7525 and waiting for your advice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906339231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142802/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906486196
**[Test build #142819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142819/testReport)** for PR 33848 at commit [`6c4b06d`](https://github.com/apache/spark/commit/6c4b06d3359be03cda877296dd0f169df025c87d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909220519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909373840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142888/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912318759
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142960/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r701575224
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
@srowen @dongjoon-hyun
As discussed above:
1. Upgrading the guava version will have a potential interfere with other libs like Hadoop
2. Use Caffeine will introduce new dependent libs. (#33784)
Therefore, I added a maximum limit to the related configurations to avoid the potential bug of Guava Cache (Guava#1761)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909323721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919083355
**[Test build #143256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143256/testReport)** for PR 33848 at commit [`f200546`](https://github.com/apache/spark/commit/f200546c1e6df47210d9d68bb55d11a9cc529035).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-1000573479
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919186519
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143256/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919134630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47759/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919134583
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47759/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r696421330
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java
##########
@@ -110,6 +118,48 @@ public void testSortShuffleBlocks() throws IOException {
}
}
+ @Test
+ public void testShuffleIndexCacheEvictionBehavior() throws IOException, ExecutionException {
+ Map<String, String> config = new HashMap<>();
+ String indexCacheSize = "8192m";
+ config.put("spark.shuffle.service.index.cache.size", indexCacheSize);
+ TransportConf transportConf = new TransportConf("shuffle", new MapConfigProvider(config));
+ ExternalShuffleBlockResolver resolver = new ExternalShuffleBlockResolver(transportConf, null);
+ resolver.registerExecutor("app0", "exec0", dataContext.createExecutorInfo(SORT_MANAGER));
+
+ LoadingCache<File, ShuffleIndexInformation> shuffleIndexCache = resolver.shuffleIndexCache;
+
+ // 8g -> 8589934592 bytes
+ long maximumWeight = JavaUtils.byteStringAsBytes(indexCacheSize);
+ int unitSize = 1048575;
+ // CacheBuilder.DEFAULT_CONCURRENCY_LEVEL
+ int concurrencyLevel = 4;
+ int totalGetCount = 16384;
+ // maxCacheCount is 8192
+ long maxCacheCount = maximumWeight / concurrencyLevel / unitSize * concurrencyLevel;
+ for (int i = 0; i < totalGetCount; i++) {
+ File indexFile = new File("shuffle_" + 0 + "_" + i + "_0.index");
+ ShuffleIndexInformation indexInfo = Mockito.mock(ShuffleIndexInformation.class);
+ Mockito.when(indexInfo.getSize()).thenReturn(unitSize);
+ shuffleIndexCache.get(indexFile, () -> indexInfo);
+ }
+
+ long totalWeight =
+ shuffleIndexCache.asMap().values().stream().mapToLong(ShuffleIndexInformation::getSize).sum();
+ long size = shuffleIndexCache.size();
+ try{
+ Assert.assertTrue(size <= maxCacheCount);
+ Assert.assertTrue(totalWeight < maximumWeight);
+ fail("The tests code should not enter this line now.");
Review comment:
Should enter line 153, but it won't now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r696418942
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java
##########
@@ -110,6 +118,48 @@ public void testSortShuffleBlocks() throws IOException {
}
}
+ @Test
+ public void testShuffleIndexCacheEvictionBehavior() throws IOException, ExecutionException {
+ Map<String, String> config = new HashMap<>();
+ String indexCacheSize = "8192m";
+ config.put("spark.shuffle.service.index.cache.size", indexCacheSize);
+ TransportConf transportConf = new TransportConf("shuffle", new MapConfigProvider(config));
+ ExternalShuffleBlockResolver resolver = new ExternalShuffleBlockResolver(transportConf, null);
+ resolver.registerExecutor("app0", "exec0", dataContext.createExecutorInfo(SORT_MANAGER));
+
+ LoadingCache<File, ShuffleIndexInformation> shuffleIndexCache = resolver.shuffleIndexCache;
+
+ // 8g -> 8589934592 bytes
+ long maximumWeight = JavaUtils.byteStringAsBytes(indexCacheSize);
+ int unitSize = 1048575;
+ // CacheBuilder.DEFAULT_CONCURRENCY_LEVEL
+ int concurrencyLevel = 4;
+ int totalGetCount = 16384;
+ // maxCacheCount is 8192
+ long maxCacheCount = maximumWeight / concurrencyLevel / unitSize * concurrencyLevel;
+ for (int i = 0; i < totalGetCount; i++) {
+ File indexFile = new File("shuffle_" + 0 + "_" + i + "_0.index");
+ ShuffleIndexInformation indexInfo = Mockito.mock(ShuffleIndexInformation.class);
+ Mockito.when(indexInfo.getSize()).thenReturn(unitSize);
+ shuffleIndexCache.get(indexFile, () -> indexInfo);
Review comment:
Not easy to mock the init behavior of ShuffleIndexInformation, so there re-write CacheLoader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906220672
**[Test build #142802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142802/testReport)** for PR 33848 at commit [`98458fd`](https://github.com/apache/spark/commit/98458fd364927627166a376ee1a145dcad1f2693).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909323721
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47391/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r701575224
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
@srowen @dongjoon-hyun
As discussed above:
1. Upgrading the guava version will have a potential interfere with other libs like Hadoop
2. Use Caffeine will introduce new dependent libs. (#33784)
Therefore, I added a maximum limit to the related configurations to avoid the potential bug of Guava Cache [Guava#1761](https://github.com/google/guava/issues/1761)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909350638
**[Test build #142888 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142888/testReport)** for PR 33848 at commit [`0fade2a`](https://github.com/apache/spark/commit/0fade2abb70ff13776275cc3eec4f4e5286bc5f0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906482213
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906523452
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47321/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r696684100
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
@HyukjinKwon @dongjoon-hyun It seems that upgrading the Guava version can solve the problem, but it will also add additional dependencies. I'm not sure whether it is valuable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r696723119
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
This new dep is fine, it's ALv2 https://github.com/google/error-prone
yes the problem is that it can potentially interfere with other libs like Hadoop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906538739
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142818/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909323721
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47391/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906617684
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142819/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak when cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906537419
**[Test build #142818 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142818/testReport)** for PR 33848 at commit [`8ec7525`](https://github.com/apache/spark/commit/8ec752568f6ce075bf3eae9a5b856ddea44708b5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906259745
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47303/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912241138
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47456/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906220672
**[Test build #142802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142802/testReport)** for PR 33848 at commit [`98458fd`](https://github.com/apache/spark/commit/98458fd364927627166a376ee1a145dcad1f2693).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-919083355
**[Test build #143256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143256/testReport)** for PR 33848 at commit [`f200546`](https://github.com/apache/spark/commit/f200546c1e6df47210d9d68bb55d11a9cc529035).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912264115
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912241138
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47456/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #33848: [WIP][SPARK-36598][SHUFFLE][SQL] Fix memory leak in cache use weight eviction mechanism
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-906215520
Maybe there the following ways can solve the problem:
1. Upgrade the Guava version can fix this bug
2. Or we can avoid this issue by limiting the configuration related to `maximumWeight` to the security range
Do you have any better suggestions ? @HyukjinKwon @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-909220519
**[Test build #142888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142888/testReport)** for PR 33848 at commit [`0fade2a`](https://github.com/apache/spark/commit/0fade2abb70ff13776275cc3eec4f4e5286bc5f0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912317896
**[Test build #142960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142960/testReport)** for PR 33848 at commit [`5524c68`](https://github.com/apache/spark/commit/5524c685991e417fc287d444356b684b24dc03ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class TimestampNTZType(AtomicType, metaclass=DataTypeSingleton):`
* `abstract class AbstractPodsAllocator `
* `class ExecutorPodsAllocator(`
* `case class GenerateExecID(podName: String) extends Serializable`
* ` case class Arguments(`
* `class StatefulsetPodsAllocator(`
* `case class UnresolvedDBObjectName(nameParts: Seq[String], isNamespace: Boolean) extends LeafNode `
* `case class ResolvedDBObjectName(catalog: CatalogPlugin, nameParts: Seq[String]) extends LeafNode `
* `trait SupportsPushDownCatalystFilters `
* `class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)`
* `class AvailableNowMicroBatchStreamWrapper(delegate: MicroBatchStream)`
* `class AvailableNowSourceWrapper(delegate: Source)`
* `case class SingleBatchExecutor() extends TriggerExecutor `
* `case class MultiBatchExecutor() extends TriggerExecutor `
* `class HiveThriftServer2AppStatusStore(store: KVStore) `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #33848:
URL: https://github.com/apache/spark/pull/33848#discussion_r701575224
##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -53,10 +54,12 @@ datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+error_prone_annotations/2.5.1//error_prone_annotations-2.5.1.jar
Review comment:
@srowen @dongjoon-hyun
As comments above:
1. Upgrading the Guava version will have potential interfere with other libs like Hadoop
2. Use Caffeine will introduce new dependent libs. (#33784)
Therefore, I added a maximum limit to the related configurations to avoid the potential bug of Guava Cache [Guava#1761](https://github.com/google/guava/issues/1761)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912285293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912285293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33848: [SPARK-36598][SHUFFLE][SQL] Avoid the memory leak of Guava cache by add maximumWeight limit
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33848:
URL: https://github.com/apache/spark/pull/33848#issuecomment-912282615
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org