You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/04/27 21:58:50 UTC

[PR] [SPARK-56644] Support periodic GC [spark-kubernetes-operator]

dongjoon-hyun opened a new pull request, #644:
URL: https://github.com/apache/spark-kubernetes-operator/pull/644

   ### What changes were proposed in this pull request?
   
   This PR adds a periodic `System.gc()` invocation in the Spark Operator JVM, controlled by a new configuration:
   
   - `spark.kubernetes.operator.periodicGC.intervalSeconds` (default: `120`)
     - Set to `0` or a negative value to disable.
     - Note that `System.gc()` is a no-op when the JVM is started with `-XX:+DisableExplicitGC`.
   
   ### Why are the changes needed?
   
   The Spark Operator JVM is a long-running process that continuously allocates objects through JOSDK reconcilers and `SentinelManager` health checks. Without an explicit Full GC, fragmentation and old-generation garbage can accumulate over time, leading to unpredictable Full GC pauses. Triggering `System.gc()` periodically gives operators a deterministic, observable point at which old generation is reclaimed, which is helpful for steady-state memory behavior in long-running deployments.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. A new operator configuration `spark.kubernetes.operator.periodicGC.intervalSeconds` is introduced and is enabled by default (`120` seconds). Operators that wish to opt out can set the value to `0` or a negative number. New `INFO`-level log lines are emitted at startup and on every GC cycle.
   
   ### How was this patch tested?
   
   Pass the CIs. And manually install and check the log.
   
   ```
   26/04/27 21:51:30 INFO   o.a.s.k.o.SparkOperator Version: 0.9.0-SNAPSHOT
   26/04/27 21:51:30 INFO   o.a.s.k.o.SparkOperator Java Version: 26.0.1+8                                                                                                                                                                              26/04/27 21:51:30 INFO   o.a.s.k.o.SparkOperator Built-in Spark Version: 4.2.0-preview4
   ...
   26/04/27 21:51:30 INFO   o.a.s.k.o.SparkOperator Periodic System.gc() enabled with interval 120s                                                                                                                                                     
   ...
   26/04/27 21:53:30 INFO   o.a.s.k.o.SparkOperator System.gc() finished in 41 ms. used: 31 MB -> 23 MB, total: 31 MB -> 206 MB
   26/04/27 21:55:30 INFO   o.a.s.k.o.SparkOperator System.gc() finished in 48 ms. used: 31 MB -> 22 MB, total: 206 MB -> 53 MB
   26/04/27 21:57:30 INFO   o.a.s.k.o.SparkOperator System.gc() finished in 43 ms. used: 25 MB -> 22 MB, total: 53 MB -> 53 MB
   ```
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56644] Support periodic GC [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #644: [SPARK-56644] Support periodic GC
URL: https://github.com/apache/spark-kubernetes-operator/pull/644


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56644] Support periodic GC [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #644:
URL: https://github.com/apache/spark-kubernetes-operator/pull/644#issuecomment-4336944502

   Merged to main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56644] Support periodic GC [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #644:
URL: https://github.com/apache/spark-kubernetes-operator/pull/644#issuecomment-4336937578

   Thank you, @peter-toth !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org