You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/14 19:54:33 UTC
[PR] fix: suppress spurious WriteFilesExec fallback reason on native writes [datafusion-comet]
andygrove opened a new pull request, #4928:
URL: https://github.com/apache/datafusion-comet/pull/4928
## Summary
Suppress a spurious `WriteFilesExec is not supported` fallback reason emitted during native Parquet writes.
Spark's `V1Writes` rule wraps native writes as `DataWritingCommandExec(cmd, WriteFilesExec(child))`. `CometExecRule` converts the outer `DataWritingCommandExec` to `CometNativeWriteExec` and unwraps the inner `WriteFilesExec` inside `convertToComet`, so the write runs natively end-to-end.
However, `transformUp` visits `WriteFilesExec` before its parent. No case in the outer `match` handles it, so it falls through to the generic `${op.nodeName} is not supported` tag at the bottom of `convertNode`. The `WriteFilesExec` instance is then discarded when the parent converts, so the executed plan looks correct — but with `spark.comet.logFallbackReasons.enabled=true` the log emits a misleading warning:
```
Comet cannot accelerate WriteFilesExec because: WriteFilesExec is not supported
```
Add `WriteFilesExec` to the list of operators that should never be tagged with a generic fallback reason (alongside `CometPlan`, `AQEShuffleReadExec`, `BroadcastExchangeExec`, `BroadcastQueryStageExec`, `AdaptiveSparkPlanExec`, `ExecutedCommandExec`, `V2CommandExec`).
## Test plan
- [ ] Enable `spark.comet.logFallbackReasons.enabled=true` and a native Parquet write via `df.write.parquet(...)`; confirm the `WriteFilesExec is not supported` warning no longer appears.
- [ ] Confirm existing `CometParquetWriterSuite` tests still pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: suppress spurious WriteFilesExec fallback reason on native writes [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4928:
URL: https://github.com/apache/datafusion-comet/pull/4928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org