You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sharma-0311 (via GitHub)" <gi...@apache.org> on 2026/05/21 05:56:36 UTC
[PR] [SPARK-56966][SPARK-56967][CORE] Auto-create event log and history se… [spark]
sharma-0311 opened a new pull request, #56028:
URL: https://github.com/apache/spark/pull/56028
### What changes were proposed in this pull request?
Auto-create the event log directory (`spark.eventLog.dir`) and the
history server log directory (`spark.history.fs.logDirectory`) if they
do not exist at startup, instead of failing with a `FileNotFoundException`.
**FsHistoryProvider** (`SPARK-56966`): In `startPolling()`, the
`FileNotFoundException` catch block now calls `FileSystem.mkdirs` with
`LOG_FOLDER_PERMISSIONS` before giving up. If creation succeeds the
directory is treated as valid; if it fails, the original warning-and-skip
behavior is preserved.
**EventLogFileWriters** (`SPARK-56967`): `requireLogBaseDirAsDirectory()`
now checks `fileSystem.exists` first and calls `FileSystem.mkdirs` if the
path is absent. This works for local FS, HDFS, and S3-compatible paths.
### Why are the changes needed?
Users frequently see startup failures because the log directory has not
been pre-created, even when the path and permissions are otherwise correct.
Auto-creating it reduces operational friction with no behavioral regression
for directories that already exist.
### Does this PR introduce _any_ user-facing change?
Yes — previously both code paths threw/warned on a missing directory.
Now they silently create it. The behavior when a directory exists is
unchanged.
### How was this patch tested?
- [ ] Unit tests for `FsHistoryProviderSuite` covering the new mkdir path
- [ ] Unit tests for `EventLogFileWritersSuite` covering auto-creation
- [ ] Manual test with a non-existent local path and an S3 path
Closes #<PR number>
Resolves https://issues.apache.org/jira/browse/SPARK-56966
Resolves https://issues.apache.org/jira/browse/SPARK-56967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56966][SPARK-56967][CORE] Auto-create event log and history se… [spark]
Posted by "sharma-0311 (via GitHub)" <gi...@apache.org>.
sharma-0311 closed pull request #56028: [SPARK-56966][SPARK-56967][CORE] Auto-create event log and history se…
URL: https://github.com/apache/spark/pull/56028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org