[GitHub] [spark] wayneguow commented on a diff in pull request #38024: [SPARK-40591][SQL] Fix data loss caused by ignoreCorruptFiles

[GitBox <gi...@apache.org>]

wayneguow commented on code in PR #38024:
URL: https://github.com/apache/spark/pull/38024#discussion_r981991786


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FilePartitionReader.scala:
##########
@@ -36,8 +36,15 @@ class FilePartitionReader[T](
 
   private def ignoreMissingFiles = options.ignoreMissingFiles
   private def ignoreCorruptFiles = options.ignoreCorruptFiles
+  private def ignoreCorruptFilesAfterRetries = options.ignoreCorruptFilesAfterRetries
 
   override def next(): Boolean = {
+
+    def shouldSkipCorruptFiles(): Boolean = {

Review Comment:
   If network is unreachable or the file storage is broken down in a period of time temporarily, a task which has been retried beyond TASK_MAX_FAILURES would cause data lost.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org