You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2024/11/11 23:10:21 UTC
[PR] wip - CometNativeScan [datafusion-comet]
parthchandra opened a new pull request, #1076:
URL: https://github.com/apache/datafusion-comet/pull/1076
Replaces hacked `CometScan` with `CometNativeScan` so we can switch between original and experimental without rebuilding.
I tried removing a lot of the functions in CometNativeScan but too many test in CometExecSuite fail and I ended up including almost everything back.
Three tests in CometExecScan are still failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] wip - CometNativeScan [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #1076:
URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1837328909
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -77,6 +77,15 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(true)
+ val COMET_FULL_NATIVE_SCAN_ENABLED: ConfigEntry[Boolean] = conf(
+ "spark.comet.native.scan.enabled")
+ .doc(
+ "Whether to enable the fully native scan. When this is turned on, Spark will use Comet to " +
+ "read supported data sources (currently only Parquet is supported natively)." +
+ " By default, this config is true.")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
No, I meant I tried to do similar native scan change into this branch. I was trying to enable DataFusion native Parquet scan yesterday. I found this branch this morning so tried to apply initial change onto this branch.
I will see if any difference can be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] wip - CometNativeScan [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #1076:
URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1837318445
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -77,6 +77,15 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(true)
+ val COMET_FULL_NATIVE_SCAN_ENABLED: ConfigEntry[Boolean] = conf(
+ "spark.comet.native.scan.enabled")
+ .doc(
+ "Whether to enable the fully native scan. When this is turned on, Spark will use Comet to " +
+ "read supported data sources (currently only Parquet is supported natively)." +
+ " By default, this config is true.")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
If it was was something related, you can merge it here. This is an experimental branch. If it is for something different, then we can change our name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] wip - CometNativeScan [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1076:
URL: https://github.com/apache/datafusion-comet/pull/1076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] wip - CometNativeScan [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #1076:
URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1837300980
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -77,6 +77,15 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(true)
+ val COMET_FULL_NATIVE_SCAN_ENABLED: ConfigEntry[Boolean] = conf(
+ "spark.comet.native.scan.enabled")
+ .doc(
+ "Whether to enable the fully native scan. When this is turned on, Spark will use Comet to " +
+ "read supported data sources (currently only Parquet is supported natively)." +
+ " By default, this config is true.")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
I added a similar config locally. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] wip - CometNativeScan [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #1076:
URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1837328909
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -77,6 +77,15 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(true)
+ val COMET_FULL_NATIVE_SCAN_ENABLED: ConfigEntry[Boolean] = conf(
+ "spark.comet.native.scan.enabled")
+ .doc(
+ "Whether to enable the fully native scan. When this is turned on, Spark will use Comet to " +
+ "read supported data sources (currently only Parquet is supported natively)." +
+ " By default, this config is true.")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
No, I meant I tried to do similar native scan change into this branch. I was trying to enable DataFusion native Parquet scan yesterday. I found this branch this morning so tried to apply onto this branch.
I will see if any difference can be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] wip - CometNativeScan [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1076:
URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1838709715
##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -189,6 +189,22 @@ class CometSparkSessionExtensions
}
// data source V1
+ case scanExec @ FileSourceScanExec(
+ HadoopFsRelation(_, partitionSchema, _, _, _: ParquetFileFormat, _),
+ _: Seq[_],
+ requiredSchema,
+ _,
+ _,
+ _,
+ _,
+ _,
+ _)
+ if CometNativeScanExec.isSchemaSupported(requiredSchema)
Review Comment:
we could also fall back for bucketed scans and other unsupported cases (can be a separate PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org