You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yang Jie (Jira)" <ji...@apache.org> on 2022/10/12 04:03:00 UTC

[jira] [Created] (SPARK-40765) Optimize redundant fs operations in `CommandUtils#calculateSingleLocationSize#getPathSize` method

Yang Jie created SPARK-40765:
--------------------------------

             Summary: Optimize redundant fs operations in `CommandUtils#calculateSingleLocationSize#getPathSize` method
                 Key: SPARK-40765
                 URL: https://issues.apache.org/jira/browse/SPARK-40765
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.4.0
            Reporter: Yang Jie


{code:java}
def getPathSize(fs: FileSystem, path: Path): Long = {
      val fileStatus = fs.getFileStatus(path)
      val size = if (fileStatus.isDirectory) {
        fs.listStatus(path)
          .map { status =>
            if (isDataPath(status.getPath, stagingDir)) {
              getPathSize(fs, status.getPath)
            } else {
              0L
            }
          }.sum
      } else {
        fileStatus.getLen
      }      size
    } {code}
Change input parameter from `Path` to `FileStatus`,  there is no need to do `fs.getFileStatus(path)` after each recursive call.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org