You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/12/23 16:35:40 UTC
[I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]
comphead opened a new issue, #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970
### Describe the bug
Comet fails if the folder exists and cannot overwrite it
### Steps to reproduce
Folder `output2` exists
```
spark.read.parquet("/tmp/writertest/input").repartition(1).write.mode("overwrite").parquet("/tmp/writertest/output2")
25/12/23 08:31:40 ERROR CometExecIterator: Native execution for task 17 failed
org.apache.comet.CometNativeException: Failed to create output file '/tmp/writertest/output2/part-00000-00017.parquet': Is a directory (os error 21)
```
### Expected behavior
Should overwrite the folder with new content
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-3858958581
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-3697596047
some more details on implementation
The flow is like
```User calls: df.write.mode("overwrite").save(path)
↓
DataFrameWriter.saveInternal()
↓
InsertIntoHadoopFsRelationCommand.run()
↓
[Line 131] deleteMatchingPartitions(fs, qualifiedOutputPath, customPartitionLocations, committer)
↓
[Line 238] committer.deleteWithJob(fs, staticPrefixPath, true)
↓
[Line 183] fs.delete(path, recursive=true) ← ACTUAL DELETION HAPPENS HERE
```
This Spark implementation now
```
val doInsertion = if (mode == SaveMode.Append) {
true
} else {
val pathExists = fs.exists(qualifiedOutputPath)
(mode, pathExists) match {
case (SaveMode.ErrorIfExists, true) =>
throw QueryCompilationErrors.outputPathAlreadyExistsError(qualifiedOutputPath)
case (SaveMode.Overwrite, true) =>
if (ifPartitionNotExists && matchingPartitions.nonEmpty) {
false
} else if (dynamicPartitionOverwrite) {
// For dynamic partition overwrite, do not delete partition directories ahead.
true
} else {
deleteMatchingPartitions(fs, qualifiedOutputPath, customPartitionLocations, committer)
true
}
case (SaveMode.Overwrite, _) | (SaveMode.ErrorIfExists, false) =>
true
case (SaveMode.Ignore, exists) =>
!exists
case (s, exists) =>
throw QueryExecutionErrors.saveModeUnsupportedError(s, exists)
}
}
```
The keypoint here, we need to support `dynamicPartitionOverwrite` or fallback to Spark in this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-6218851668
Overwrite mode works on `main`, as @dwsmith1983 found. I checked it at c230bf4e98: a native Parquet write with `mode("overwrite")` into a directory that already held an earlier write replaced it, on Spark 3.5.9 (`CometNativeWriteExec`) and on Spark 4.1.3 (`CometWriteFilesExec`). The second write succeeded, and the directory read back only its 5 rows.
#4946 added the `SaveMode.Overwrite` handling on Spark 3.x, which `SaveMode.Overwrite replaces existing parquet data` in `CometParquetWriterSuite` covers, and on Spark 4.0+ #5763 leaves the save mode to Spark. Closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-5976226710
Overwrite mode works on main now: #4946 added `SaveMode.Overwrite` handling to the native writer (`CometNativeWriteExec.scala`, test "SaveMode.Overwrite replaces existing parquet data" in `CometParquetWriterSuite`), and on Spark 4.0+ #5763 leaves SaveMode to Spark. I think this can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #2970:
URL: https://github.com/apache/datafusion-comet/issues/2970#issuecomment-3688823290
Taking a look at this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet writer doesn't respect `OVERWRITE` mode [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2970: Comet writer doesn't respect `OVERWRITE` mode
URL: https://github.com/apache/datafusion-comet/issues/2970
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org