[PR] [SPARK-XXXXX][K8S] Write Spark driver stderr to Kubernetes termination log on failure [spark]

["turboFei (via GitHub)" <gi...@apache.org>]

turboFei opened a new pull request, #56262:
URL: https://github.com/apache/spark/pull/56262

   ### What changes were proposed in this pull request?
   
   When running Spark on Kubernetes, driver pod failures are difficult to diagnose because error details are only available in `kubectl logs`, which may have rotated by the time an operator investigates.
   
   Kubernetes provides a standard mechanism for surfacing container exit reasons: writing to `/dev/termination-log` (max 4096 bytes). This message is preserved in pod status and visible via `kubectl describe pod → Last State > Message`, even after the pod is deleted.
   
   This patch adds termination log support to the Spark driver entrypoint:
   
   - On non-zero exit, the last 4KB of driver stderr is written to `/dev/termination-log`
   - stderr is streamed through an **awk ring buffer via named pipe**: fully visible in `kubectl logs` in real-time, with no unbounded `/tmp` disk growth regardless of how long the driver runs
   - **SIGTERM is forwarded to tini** so Kubernetes graceful shutdown still reaches the Spark driver (without `exec`, bash as PID 1 ignores SIGTERM by default)
   - Only the `driver` case is affected; `executor` and pass-through modes are unchanged
   - The `TERMINATION_LOG` env var can override the path (useful for local testing)
   
   **How it works:**
   ```
   driver stderr ──→ named pipe ──→ awk ring buffer (memory only, max 4KB)
                                         └──→ container stderr (kubectl logs, unchanged)
   on exit code != 0:
     cat last4k_buffer → /dev/termination-log
   ```
   
   ### Why are the changes needed?
   
   Without this change, operators investigating a failed Spark driver pod on Kubernetes must:
   1. Have captured `kubectl logs` before the pod was cleaned up, or
   2. Have external log aggregation configured
   
   With this change, `kubectl describe pod <driver-pod>` always shows the last 4KB of driver stderr in the `Last State > Message` field, providing immediate diagnostic context.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes — `kubectl describe pod` for a failed Spark driver pod will now populate the `Last State > Message` field with the last 4KB of driver stderr.
   
   ### How was this patch tested?
   
   - Verified locally that the awk ring buffer correctly captures the last 4096 bytes of stderr output (tested with >56KB of simulated log output)
   - Verified that `kubectl logs` output is unchanged (stderr still streams in real-time)
   - Verified that SIGTERM forwarding works correctly for graceful shutdown
   - Verified that named pipe is cleaned up on both normal and abnormal exit (via `trap ... EXIT`)
   
   ### Was this patch authored or co-authored by a generative AI?
   
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57204][K8S] Write Spark driver stderr to Kubernetes termination log on failure [spark]

["turboFei (via GitHub)" <gi...@apache.org>]

turboFei closed pull request #56262: [SPARK-57204][K8S] Write Spark driver stderr to Kubernetes termination log on failure
URL: https://github.com/apache/spark/pull/56262


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org