You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2025/01/17 00:31:44 UTC
[PR] Comet parquet exec merge 20240116 [datafusion-comet]
parthchandra opened a new pull request, #1299:
URL: https://github.com/apache/datafusion-comet/pull/1299
Merge from main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: [comet-parquet-exec] merge from main 20240116 [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #1299:
URL: https://github.com/apache/datafusion-comet/pull/1299#discussion_r1920791563
##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -190,7 +190,7 @@ class CometSparkSessionExtensions
// data source V1
case scanExec @ FileSourceScanExec(
- HadoopFsRelation(_, partitionSchema, _, _, _: ParquetFileFormat, _),
+ HadoopFsRelation(_, partitionSchema, _, _, _, _),
Review Comment:
Oh, right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: [comet-parquet-exec] merge from main 20240116 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1299:
URL: https://github.com/apache/datafusion-comet/pull/1299#discussion_r1920620850
##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -190,7 +190,7 @@ class CometSparkSessionExtensions
// data source V1
case scanExec @ FileSourceScanExec(
- HadoopFsRelation(_, partitionSchema, _, _, _: ParquetFileFormat, _),
+ HadoopFsRelation(_, partitionSchema, _, _, _, _),
Review Comment:
This doesn't seem correct, because it would make us replace non-parquet scans as well.
main branch has:
```scala
// data source V1
case scanExec @ FileSourceScanExec(
HadoopFsRelation(_, partitionSchema, _, _, fileFormat, _),
_: Seq[_],
requiredSchema,
_,
_,
_,
_,
_,
_)
if CometScanExec.isFileFormatSupported(fileFormat)
&& CometScanExec.isSchemaSupported(requiredSchema)
&& CometScanExec.isSchemaSupported(partitionSchema) =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: [comet-parquet-exec] merge from main 20240116 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1299:
URL: https://github.com/apache/datafusion-comet/pull/1299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: [comet-parquet-exec] merge from main 20240116 [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #1299:
URL: https://github.com/apache/datafusion-comet/pull/1299#issuecomment-2599460209
@andygrove I added another fix since you approved. This was also a merge issue and caused 6 test failures if native_datafusion was enabled.
Latest count :
```
native_comet: Tests: succeeded 797, failed 0, canceled 2, ignored 50, pending 0
native_datafusion: Tests: succeeded 628, failed 40, canceled 2, ignored 50, pending 0
native_iceberg_compat: Tests: succeeded 623, failed 45, canceled 2, ignored 50, pending 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org