You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/11/22 01:51:29 UTC
(spark) branch master updated: [SPARK-46012][CORE][FOLLOWUP] Invoke `fs.listStatus` once and reuse the result
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6be4a035826 [SPARK-46012][CORE][FOLLOWUP] Invoke `fs.listStatus` once and reuse the result
6be4a035826 is described below
commit 6be4a0358265fb81f68a27589f9940bd726c8ee7
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Tue Nov 21 17:51:10 2023 -0800
[SPARK-46012][CORE][FOLLOWUP] Invoke `fs.listStatus` once and reuse the result
### What changes were proposed in this pull request?
This PR is a follow-up of #43914 and aims to invoke `fs.listStatus` once and reuse the result.
### Why are the changes needed?
This will prevent the increase of the number of `listStatus` invocation .
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs with the existing test case.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43944 from dongjoon-hyun/SPARK-46012-2.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../scala/org/apache/spark/deploy/history/EventLogFileReaders.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
index 714987a8eb8..8c3dda47277 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
@@ -119,8 +119,9 @@ object EventLogFileReader extends Logging {
if (isSingleEventLog(status)) {
Some(new SingleFileEventLogFileReader(fs, status.getPath, Option(status)))
} else if (isRollingEventLogs(status)) {
- if (fs.listStatus(status.getPath).exists(RollingEventLogFilesWriter.isEventLogFile) &&
- fs.listStatus(status.getPath).exists(RollingEventLogFilesWriter.isAppStatusFile)) {
+ val files = fs.listStatus(status.getPath)
+ if (files.exists(RollingEventLogFilesWriter.isEventLogFile) &&
+ files.exists(RollingEventLogFilesWriter.isAppStatusFile)) {
Some(new RollingEventLogFilesFileReader(fs, status.getPath))
} else {
logDebug(s"Rolling event log directory have no event log file at ${status.getPath}")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org