You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/03/15 15:42:19 UTC
[I] `native_datafusion` scan is only enabled when `spark.comet.exec.enabled` is set [datafusion-comet]
andygrove opened a new issue, #1536:
URL: https://github.com/apache/datafusion-comet/issues/1536
### Describe the bug
`native_datafusion` scan is only enabled when `spark.comet.exec.enabled` is set.
In `CometScanRule` we add `CometScanExec` as a placeholder for a `native_datafusion` scan:
```rust
case scanExec @ FileSourceScanExec(
HadoopFsRelation(_, partitionSchema, _, _, fileFormat, _),
_: Seq[_],
requiredSchema,
_,
_,
_,
_,
_,
_)
if CometScanExec.isFileFormatSupported(fileFormat)
&& CometNativeScanExec.isSchemaSupported(requiredSchema)
&& CometNativeScanExec.isSchemaSupported(partitionSchema)
// TODO we only enable full native scan if COMET_EXEC_ENABLED is enabled
// but this is not really what we want .. we currently insert `CometScanExec`
// here and then it gets replaced with `CometNativeScanExec` in `CometExecRule`
// but that only happens if `COMET_EXEC_ENABLED` is enabled
&& COMET_EXEC_ENABLED.get()
&& COMET_NATIVE_SCAN_IMPL.get() == CometConf.SCAN_NATIVE_DATAFUSION =>
logInfo("Comet extension enabled for v1 full native Scan")
CometScanExec(scanExec, session)
```
and then in `CometExecRule` we replace it:
```
plan.transformUp {
// Fully native scan for V1
case scan: CometScanExec
if COMET_NATIVE_SCAN_IMPL.get().equals(CometConf.SCAN_NATIVE_DATAFUSION) =>
val nativeOp = QueryPlanSerde.operator2Proto(scan).get
CometNativeScanExec(nativeOp, scan.wrapped, scan.session)
```
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `native_datafusion` scan is only enabled when `spark.comet.exec.enabled` is set [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1536:
URL: https://github.com/apache/datafusion-comet/issues/1536#issuecomment-2727044778
Thanks @viirya. Yes, fully native scan is only used with native execution. Maybe the design is correct as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `native_datafusion` scan is only enabled when `spark.comet.exec.enabled` is set [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1536:
URL: https://github.com/apache/datafusion-comet/issues/1536#issuecomment-2730761195
I will close this issue since the design does seem correct. We should remove the TODO though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `native_datafusion` scan is only enabled when `spark.comet.exec.enabled` is set [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1536: `native_datafusion` scan is only enabled when `spark.comet.exec.enabled` is set
URL: https://github.com/apache/datafusion-comet/issues/1536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `native_datafusion` scan is only enabled when `spark.comet.exec.enabled` is set [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1536:
URL: https://github.com/apache/datafusion-comet/issues/1536#issuecomment-2726748501
@viirya Do you remember why the design is like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `native_datafusion` scan is only enabled when `spark.comet.exec.enabled` is set [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #1536:
URL: https://github.com/apache/datafusion-comet/issues/1536#issuecomment-2726904269
Hmm, I think it is a mistake. In the commit 1cca8d6f7bd2dfb8e1996bdd55ebe09d08eb8221, I transformed `CometScanExec` added for fully native scan in `CometScanRule` to `CometNativeScanExec` in `CometExecRule`.
Maybe at the moment, I thought that this fully native scan is only used with native execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org