You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/12/23 16:35:40 UTC

[I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]

comphead opened a new issue, #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970

   ### Describe the bug
   
   Comet fails if the folder exists and cannot overwrite it
   
   ### Steps to reproduce
   
   Folder `output2` exists
   ```
   spark.read.parquet("/tmp/writertest/input").repartition(1).write.mode("overwrite").parquet("/tmp/writertest/output2")
   
   25/12/23 08:31:40 ERROR CometExecIterator: Native execution for task 17 failed
   org.apache.comet.CometNativeException: Failed to create output file '/tmp/writertest/output2/part-00000-00017.parquet': Is a directory (os error 21)
   
   ```
   
   ### Expected behavior
   
   Should overwrite the folder with new content
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-3858958581

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-3697596047

   some more details on implementation
   
   The flow is like
   
   ```User calls: df.write.mode("overwrite").save(path)
       ↓
   DataFrameWriter.saveInternal() 
       ↓
   InsertIntoHadoopFsRelationCommand.run()
       ↓
   [Line 131] deleteMatchingPartitions(fs, qualifiedOutputPath, customPartitionLocations, committer)
       ↓
   [Line 238] committer.deleteWithJob(fs, staticPrefixPath, true)
       ↓
   [Line 183] fs.delete(path, recursive=true)  ← ACTUAL DELETION HAPPENS HERE
   ```
   This Spark implementation now
   ```
       val doInsertion = if (mode == SaveMode.Append) {
         true
       } else {
         val pathExists = fs.exists(qualifiedOutputPath)
         (mode, pathExists) match {
           case (SaveMode.ErrorIfExists, true) =>
             throw QueryCompilationErrors.outputPathAlreadyExistsError(qualifiedOutputPath)
           case (SaveMode.Overwrite, true) =>
             if (ifPartitionNotExists && matchingPartitions.nonEmpty) {
               false
             } else if (dynamicPartitionOverwrite) {
               // For dynamic partition overwrite, do not delete partition directories ahead.
               true
             } else {
               deleteMatchingPartitions(fs, qualifiedOutputPath, customPartitionLocations, committer)
               true
             }
           case (SaveMode.Overwrite, _) | (SaveMode.ErrorIfExists, false) =>
             true
           case (SaveMode.Ignore, exists) =>
             !exists
           case (s, exists) =>
             throw QueryExecutionErrors.saveModeUnsupportedError(s, exists)
         }
       }
   ```
   
   The keypoint here, we need to support `dynamicPartitionOverwrite` or fallback to Spark in this case


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-6218851668

   Overwrite mode works on `main`, as @dwsmith1983 found. I checked it at c230bf4e98: a native Parquet write with `mode("overwrite")` into a directory that already held an earlier write replaced it, on Spark 3.5.9 (`CometNativeWriteExec`) and on Spark 4.1.3 (`CometWriteFilesExec`). The second write succeeded, and the directory read back only its 5 rows.
   
   #4946 added the `SaveMode.Overwrite` handling on Spark 3.x, which `SaveMode.Overwrite replaces existing parquet data` in `CometParquetWriterSuite` covers, and on Spark 4.0+ #5763 leaves the save mode to Spark. Closing.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-5976226710

   Overwrite mode works on main now: #4946 added `SaveMode.Overwrite` handling to the native writer (`CometNativeWriteExec.scala`, test "SaveMode.Overwrite replaces existing parquet data" in `CometParquetWriterSuite`), and on Spark 4.0+ #5763 leaves SaveMode to Spark. I think this can be closed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-3688823290

   Taking a look at this 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2970: Comet writer doesn't respect `OVERWRITE` mode
URL: https://github.com/apache/datafusion-comet/issues/2970


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org