You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2025/08/08 05:53:33 UTC
[PR] [SPARK-53193][DOCS] Add advanced JVM optimization parameters to tuning guide [spark]
wangyum opened a new pull request, #51920:
URL: https://github.com/apache/spark/pull/51920
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53193][DOCS] Add advanced JVM optimization parameters to tuning guide [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #51920:
URL: https://github.com/apache/spark/pull/51920#discussion_r2268802961
##########
docs/tuning.md:
##########
@@ -232,8 +232,12 @@ temporary objects created during task execution. Some steps which may be useful
value of the JVM's `NewRatio` parameter. Many JVMs default this to 2, meaning that the Old generation
occupies 2/3 of the heap. It should be large enough such that this fraction exceeds `spark.memory.fraction`.
-* Since 4.0.0, Spark uses JDK 17 by default, which also makes the G1GC garbage collector the default. Note that with
- large executor heap sizes, it may be important to increase the G1 region size with `-XX:G1HeapRegionSize`.
+* Since 4.0.0, Spark uses JDK 17 by default, which also makes the G1GC garbage collector the default,
+ Some applications may encounter out-of-memory (OOM) issues or experience slower performance.
+ To mitigate these problems:
+ - Increase region size with `-XX:G1HeapRegionSize=<size>` when using large executor heap sizes.
+ - Optimize GC locker behavior with `-XX:+UnlockDiagnosticVMOptions -XX:GCLockerRetryAllocationCount=100`.
Review Comment:
Might be worth mentioning that the purpose of this suggestion is to mitigate https://openjdk.org/jeps/423, and is not required when running Spark with JDK 22+
PS: I have verified some internal Spark workloads, which were easy to OOM with JDK 21 and G1, that work well on JDK 22. Unfortunately, JDK 22 is EOL, life will be easier if the openJDK community can backport JEP 423 to JDK 21 LTS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53193][DOCS] Add advanced JVM optimization parameters to tuning guide [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #51920: [SPARK-53193][DOCS] Add advanced JVM optimization parameters to tuning guide
URL: https://github.com/apache/spark/pull/51920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53193][DOCS] Add advanced JVM optimization parameters to tuning guide [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51920:
URL: https://github.com/apache/spark/pull/51920#issuecomment-3560770581
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53193][DOCS] Add advanced JVM optimization parameters to tuning guide [spark]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #51920:
URL: https://github.com/apache/spark/pull/51920#discussion_r2262170046
##########
docs/tuning.md:
##########
@@ -232,8 +232,12 @@ temporary objects created during task execution. Some steps which may be useful
value of the JVM's `NewRatio` parameter. Many JVMs default this to 2, meaning that the Old generation
occupies 2/3 of the heap. It should be large enough such that this fraction exceeds `spark.memory.fraction`.
-* Since 4.0.0, Spark uses JDK 17 by default, which also makes the G1GC garbage collector the default. Note that with
- large executor heap sizes, it may be important to increase the G1 region size with `-XX:G1HeapRegionSize`.
+* Since 4.0.0, Spark uses JDK 17 by default, which also makes the G1GC garbage collector the default,
+ Some applications may encounter out-of-memory (OOM) issues or experience slower performance.
+ To mitigate these problems:
+ - Increase region size with `-XX:G1HeapRegionSize=<size>` when using large executor heap sizes.
+ - Optimize GC locker behavior with `-XX:+UnlockDiagnosticVMOptions -XX:GCLockerRetryAllocationCount=100`.
+ - Consider switching to ParallelGC by specifying `-XX:+UseParallelGC` for better throughput with large workloads.
Review Comment:
JDK21 >=21.0.4 + Parallel GC may exceed memory, causing Executor to be killed by YARN.
---
8328744: Parallel: Parallel GC throws OOM before heap is fully expanded
https://bugs.openjdk.org/browse/JDK-8328744
https://bugs.openjdk.org/browse/JDK-8338920
An existing bug may have prevented the full usage of the Java heap allotted with the command line flag -Xmx. That bug has been fixed with JDK-8328744. As a side effect of this fix, installations may experience an increased heap usage when using Parallel GC. Customers should, if necessary, adjust the maximum heap size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org