You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2025/05/07 18:56:55 UTC
[I] feat: bucketed scan for native_datafusion Parquet scan [datafusion-comet]
mbutrovich opened a new issue, #1719:
URL: https://github.com/apache/datafusion-comet/issues/1719
### What is the problem the feature request solves?
The native_datafusion Parquet scan does not support bucketed scan, and fails most of the tests in Spark's BucketedReadSuite without a fallback. With a bucketed scan, some partitions end up without a file to read so their `PartitionedFile` is empty.
### Describe the potential solution
I don't think DataSourceExec will take no file at construction. We might need to replace that node with a different no-op node that generates an empty data set with the correct schema in the case of a bucketed scan when a partition has no corresponding file.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] feat: bucketed scan for native_datafusion Parquet scan [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed issue #1719: feat: bucketed scan for native_datafusion Parquet scan
URL: https://github.com/apache/datafusion-comet/issues/1719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] feat: bucketed scan for native_datafusion Parquet scan [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1719:
URL: https://github.com/apache/datafusion-comet/issues/1719#issuecomment-2863289065
Interestingly, native_datafusion currently passes the "bucketed table" Comet test. I suspect the Spark SQL tests are doing some bucket pruning, which is where we end up with empty file paths for some of the partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org