You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/04/07 18:01:12 UTC

(spark-kubernetes-operator) branch main updated: [SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git


The following commit(s) were added to refs/heads/main by this push:
     new bc4e3fe  [SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory
bc4e3fe is described below

commit bc4e3fe9195d1c52374e45a7d64b075ee32cfd25
Author: Jiwon Park <jp...@outlook.kr>
AuthorDate: Tue Apr 7 11:01:05 2026 -0700

    [SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory
    
    ### What changes were proposed in this pull request?
    
    - Use try-with-resources for `OutputStreamWriter` in `createLocalFileForPodTemplateSpec` to
      guarantee streams are closed even when an exception is thrown during the write operation.
    - Replace the `"UTF-8"` string literal with `StandardCharsets.UTF_8` for type safety.
    - Narrow the `catch (Throwable t)` clause in `deleteLocalFileFromPathKey` to
      `catch (SecurityException e)`, which is the only exception `File.delete()` can throw.
    
    ### Why are the changes needed?
    
    `FileOutputStream` and `OutputStreamWriter` were opened without try-with-resources.
    If `writer.write()` or `ModelUtils.asJsonString()` threw an exception, the streams would
    remain open, causing file descriptor leaks. Over time this could exhaust the OS file
    descriptor limit in long-running operator deployments.
    
    Catching `Throwable` in `deleteLocalFileFromPathKey` is an anti-pattern that swallows
    JVM-level errors such as `OutOfMemoryError` and `VirtualMachineError`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    - Added `testBuildResourceSpecWritesAndCleansTempFilesForPodTemplateSpec` to
      `SparkAppResourceSpecFactoryTest` to verify that temp files created for driver and executor
      pod template specs are deleted after `buildResourceSpec` completes.
    - Ran `./gradlew :spark-operator:test`, `checkstyleMain`, `checkstyleTest`, `pmdMain`,
      `spotbugsMain`, and `spotlessCheck` — all passed.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Sonnet 4.6
    
    Closes #597 from j1wonpark/fix/stream-resource-leak.
    
    Authored-by: Jiwon Park <jp...@outlook.kr>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../operator/reconciler/SparkAppResourceSpecFactory.java    | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkAppResourceSpecFactory.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkAppResourceSpecFactory.java
index bd7e33e..e137faa 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkAppResourceSpecFactory.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkAppResourceSpecFactory.java
@@ -29,6 +29,7 @@ import java.io.File;
 import java.io.FileOutputStream;
 import java.io.IOException;
 import java.io.OutputStreamWriter;
+import java.nio.charset.StandardCharsets;
 import java.util.HashMap;
 import java.util.Map;
 import java.util.Optional;
@@ -144,8 +145,8 @@ public final class SparkAppResourceSpecFactory {
       } else {
         log.warn("Local temp file not found at {}", pathKey);
       }
-    } catch (Throwable t) {
-      log.warn("Failed to delete temp file. Attempting delete upon exit.", t);
+    } catch (SecurityException e) {
+      log.warn("Failed to delete temp file. Attempting delete upon exit.", e);
     } finally {
       if (!deleted && localFile.isPresent() && localFile.get().exists()) {
         localFile.get().deleteOnExit();
@@ -212,10 +213,10 @@ public final class SparkAppResourceSpecFactory {
       final PodTemplateSpec podTemplateSpec, final String tempFilePrefix) {
     try {
       File tmpFile = File.createTempFile(tempFilePrefix, ".json");
-      FileOutputStream fileStream = new FileOutputStream(tmpFile);
-      OutputStreamWriter writer = new OutputStreamWriter(fileStream, "UTF-8");
-      writer.write(ModelUtils.asJsonString(ModelUtils.getPodFromTemplateSpec(podTemplateSpec)));
-      writer.close();
+      try (OutputStreamWriter writer =
+          new OutputStreamWriter(new FileOutputStream(tmpFile), StandardCharsets.UTF_8)) {
+        writer.write(ModelUtils.asJsonString(ModelUtils.getPodFromTemplateSpec(podTemplateSpec)));
+      }
       String path = tmpFile.getAbsolutePath();
       if (log.isDebugEnabled()) {
         log.debug("Temp file wrote to {}", tmpFile.getAbsolutePath());


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org