You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/04/07 18:01:12 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new bc4e3fe [SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory
bc4e3fe is described below
commit bc4e3fe9195d1c52374e45a7d64b075ee32cfd25
Author: Jiwon Park <jp...@outlook.kr>
AuthorDate: Tue Apr 7 11:01:05 2026 -0700
[SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory
### What changes were proposed in this pull request?
- Use try-with-resources for `OutputStreamWriter` in `createLocalFileForPodTemplateSpec` to
guarantee streams are closed even when an exception is thrown during the write operation.
- Replace the `"UTF-8"` string literal with `StandardCharsets.UTF_8` for type safety.
- Narrow the `catch (Throwable t)` clause in `deleteLocalFileFromPathKey` to
`catch (SecurityException e)`, which is the only exception `File.delete()` can throw.
### Why are the changes needed?
`FileOutputStream` and `OutputStreamWriter` were opened without try-with-resources.
If `writer.write()` or `ModelUtils.asJsonString()` threw an exception, the streams would
remain open, causing file descriptor leaks. Over time this could exhaust the OS file
descriptor limit in long-running operator deployments.
Catching `Throwable` in `deleteLocalFileFromPathKey` is an anti-pattern that swallows
JVM-level errors such as `OutOfMemoryError` and `VirtualMachineError`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added `testBuildResourceSpecWritesAndCleansTempFilesForPodTemplateSpec` to
`SparkAppResourceSpecFactoryTest` to verify that temp files created for driver and executor
pod template specs are deleted after `buildResourceSpec` completes.
- Ran `./gradlew :spark-operator:test`, `checkstyleMain`, `checkstyleTest`, `pmdMain`,
`spotbugsMain`, and `spotlessCheck` — all passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
Closes #597 from j1wonpark/fix/stream-resource-leak.
Authored-by: Jiwon Park <jp...@outlook.kr>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../operator/reconciler/SparkAppResourceSpecFactory.java | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkAppResourceSpecFactory.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkAppResourceSpecFactory.java
index bd7e33e..e137faa 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkAppResourceSpecFactory.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkAppResourceSpecFactory.java
@@ -29,6 +29,7 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStreamWriter;
+import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
@@ -144,8 +145,8 @@ public final class SparkAppResourceSpecFactory {
} else {
log.warn("Local temp file not found at {}", pathKey);
}
- } catch (Throwable t) {
- log.warn("Failed to delete temp file. Attempting delete upon exit.", t);
+ } catch (SecurityException e) {
+ log.warn("Failed to delete temp file. Attempting delete upon exit.", e);
} finally {
if (!deleted && localFile.isPresent() && localFile.get().exists()) {
localFile.get().deleteOnExit();
@@ -212,10 +213,10 @@ public final class SparkAppResourceSpecFactory {
final PodTemplateSpec podTemplateSpec, final String tempFilePrefix) {
try {
File tmpFile = File.createTempFile(tempFilePrefix, ".json");
- FileOutputStream fileStream = new FileOutputStream(tmpFile);
- OutputStreamWriter writer = new OutputStreamWriter(fileStream, "UTF-8");
- writer.write(ModelUtils.asJsonString(ModelUtils.getPodFromTemplateSpec(podTemplateSpec)));
- writer.close();
+ try (OutputStreamWriter writer =
+ new OutputStreamWriter(new FileOutputStream(tmpFile), StandardCharsets.UTF_8)) {
+ writer.write(ModelUtils.asJsonString(ModelUtils.getPodFromTemplateSpec(podTemplateSpec)));
+ }
String path = tmpFile.getAbsolutePath();
if (log.isDebugEnabled()) {
log.debug("Temp file wrote to {}", tmpFile.getAbsolutePath());
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org