You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/17 11:15:59 UTC

[GitHub] [spark] steveloughran commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

steveloughran commented on a change in pull request #31495:
URL: https://github.com/apache/spark/pull/31495#discussion_r595923118



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManager.scala
##########
@@ -277,7 +277,6 @@ class FileSystemBasedCheckpointFileManager(path: Path, hadoopConf: Configuration
       fs.delete(path, true)
     } catch {
       case e: FileNotFoundException =>
-        logInfo(s"Failed to delete $path as it does not exist")

Review comment:
       > Calling delete blindly is really a speedup.
   
   mmm. Those HEAD requests add up, don't they?
   
   FWIW, Hadoop-3.3.1 adds a lot of IOStatistics collection on s3a and abfs operations, for which there's an interface implemented by, amongst other things: input streams, output streams, filesystems, remote iterators. But toString() logs this stuff too. So if you log an FS.toString() at debug you will see these. We are doing more than just counting #of calls, we are collecting min/max/mean performance. LIST is the killer -especially if you have versioning turned on on S3 and then do lots of overwrite/delete of objects. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org