You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/11 16:53:23 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
dongjoon-hyun opened a new pull request #34560:
URL: https://github.com/apache/spark/pull/34560
### What changes were proposed in this pull request?
This PR aims to remove the explicit GC options in Scala tests.
### Why are the changes needed?
At Apache Spark 3.0, SPARK-29282 introduced the explicit GC options in Scala tests to see the consistent result between Java8/Java11. Now, we have Java 17 additionally. This PR aims to use the default GC options for each JVM.
This is a partial revert of SPARK-29282.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the GitHub Action CIs without memory issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966588245
**[Test build #145117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145117/testReport)** for PR 34560 at commit [`a3103a2`](https://github.com/apache/spark/commit/a3103a2134e04683ff06231cb2a2d2d712b7dfb7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966468539
Hi, @HyukjinKwon and @LuciferYang .
The daily Java11 and 17 runs seems to work, but the only exception looks like the known the memory issue.
So I hope this try can mitigate the situation at least the following warning.
```
[warn] In the last 3393 seconds, 6.757 (0.2%) were spent in GC. [Heap: 3.49GB free of 4.00GB, max 4.00GB] Consider increasing the JVM heap using `-Xmx` or try a different collector, e.g. `-XX:+UseG1GC`, for better performance.
```
- https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966777214
Thanks. Yes, GA jobs are flaky due to the memory limits and I'll monitor this carefully.
Note that this will affect only Java 11/Java 17 which means
- Java11 build job (without tests)
- Java17 build job (without tests)
- Daily Java11/17 build and test jobs which is added recently.
The other jobs are running Java 8 whose default GC is `ParallelGC`. In addition, the flakiness mainly occurs at the `build&test` jobs.
Especially, this PR aims to stablize the daily Java11 and Java17 jobs. I'll keep improving those test coverage for Spark 3.3..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966554262
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49586/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966608810
Could you review this, @mridulm ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966777265
Thank you, @LuciferYang , too. Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966503976
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49586/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966590786
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966546633
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49586/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #34560:
URL: https://github.com/apache/spark/pull/34560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966554262
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49586/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966471636
**[Test build #145117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145117/testReport)** for PR 34560 at commit [`a3103a2`](https://github.com/apache/spark/commit/a3103a2134e04683ff06231cb2a2d2d712b7dfb7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966468539
Hi, @HyukjinKwon and @LuciferYang .
The daily Java11 and 17 runs seems to work, but the only exception looks like the known the memory issue.
So I hope this try can mitigate the situation.
- https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966590786
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966468539
Hi, @HyukjinKwon and @LuciferYang .
The daily Java11 and 17 runs seems to work, but the only exception looks like the known the memory issue.
So I hope this try can mitigate the situation at least the following warning.
```
[warn] In the last 3393 seconds, 6.757 (0.2%) were spent in GC. [Heap: 3.49GB free of 4.00GB, max 4.00GB]
Consider increasing the JVM heap using `-Xmx` or try a different collector, e.g. `-XX:+UseG1GC`, for better performance.
```
- https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966471636
**[Test build #145117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145117/testReport)** for PR 34560 at commit [`a3103a2`](https://github.com/apache/spark/commit/a3103a2134e04683ff06231cb2a2d2d712b7dfb7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34560: [SPARK-37293][TESTS] Remove explicit GC options from Scala tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34560:
URL: https://github.com/apache/spark/pull/34560#issuecomment-966777214
Thanks. Yes, GA are flaky due to the memory limits and I'll monitor this carefully.
Note that this will affect only Java 11/Java 17 which means
- Java11 build job (without tests)
- Java17 build job (without tests)
- Daily Java11/17 build and test jobs which is added recently.
The other jobs are running Java 8 whose default GC is `ParallelGC`. In addition, the flakiness mainly occurs at the `build&test` jobs.
Especially, this PR aims to stablize the daily Java11 and Java17 jobs. I'll keep improving those test coverage for Spark 3.3..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org