You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yang Jie (Jira)" <ji...@apache.org> on 2022/10/12 04:03:00 UTC
[jira] [Created] (SPARK-40765) Optimize redundant fs operations in `CommandUtils#calculateSingleLocationSize#getPathSize` method
Yang Jie created SPARK-40765:
--------------------------------
Summary: Optimize redundant fs operations in `CommandUtils#calculateSingleLocationSize#getPathSize` method
Key: SPARK-40765
URL: https://issues.apache.org/jira/browse/SPARK-40765
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.4.0
Reporter: Yang Jie
{code:java}
def getPathSize(fs: FileSystem, path: Path): Long = {
val fileStatus = fs.getFileStatus(path)
val size = if (fileStatus.isDirectory) {
fs.listStatus(path)
.map { status =>
if (isDataPath(status.getPath, stagingDir)) {
getPathSize(fs, status.getPath)
} else {
0L
}
}.sum
} else {
fileStatus.getLen
} size
} {code}
Change input parameter from `Path` to `FileStatus`, there is no need to do `fs.getFileStatus(path)` after each recursive call.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org