You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pgwhalen (via GitHub)" <gi...@apache.org> on 2023/12/23 04:23:02 UTC
[I] Re-enable support for ListingTables with custom file formats [arrow-datafusion]
pgwhalen opened a new issue, #8637:
URL: https://github.com/apache/arrow-datafusion/issues/8637
### Is your feature request related to a problem or challenge?
After I upgraded my application from 28.0.0 to 34.0.0, I noticed that as a result of https://github.com/apache/arrow-datafusion/pull/7390, it is no longer possible to register your own listing table with a custom file format, because the code that supports writing assumes a fixed list of [FileTypes](https://github.com/apache/arrow-datafusion/blob/03c2ef46f2d88fb015ee305ab67df6d930b780e2/datafusion/common/src/file_options/file_type.rs#L45). Specifically, the when implementing [FileFormat](https://github.com/apache/arrow-datafusion/blob/03c2ef46f2d88fb015ee305ab67df6d930b780e2/datafusion/core/src/datasource/file_format/mod.rs#L109), the enum must be provided.
### Describe the solution you'd like
Ideally `FileFormat` would not have to return a `FileType`, and the ability to write to listing tables would be:
- optional
- not coupled to an enumerated list of file types
As it stands currently, it looks like this method is only used to help out code that supports writing, so perhaps it can be refactored to be more coupled with that functionality. I don't have any specific proposal at the moment, but it seems achievable (if it's something maintainers would be okay with).
### Describe alternatives you've considered
Several alternatives seem to work for me:
- Give up on `ListingTable` as an extensible API and just use `TableProvider` instead. This would be disappointing, because it does provide a lot of nice functionality that would have to reimplemented.
- Provide a garbage `FileType` in `FileFormat::file_type` because it doesn't seem to cause problems unless you're trying to write. This is what I'm doing for now.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Re-enable support for ListingTables with custom file formats [arrow-datafusion]
Posted by "pgwhalen (via GitHub)" <gi...@apache.org>.
pgwhalen commented on issue #8637:
URL: https://github.com/apache/arrow-datafusion/issues/8637#issuecomment-1871714722
That looks great, I'll keep an eye on it, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Re-enable support for ListingTables with custom file formats [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8637:
URL: https://github.com/apache/arrow-datafusion/issues/8637#issuecomment-1869554139
Thank you for this comment @pgwhalen
I think another idea that has come up (e.g. in https://github.com/apache/arrow-datafusion/issues/8345) is to combine `FileType` into the `FileFormat` -- I filed https://github.com/apache/arrow-datafusion/issues/8657 with a more specific proposal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Re-enable support for ListingTables with custom file formats [datafusion]
Posted by "pgwhalen (via GitHub)" <gi...@apache.org>.
pgwhalen closed issue #8637: Re-enable support for ListingTables with custom file formats
URL: https://github.com/apache/datafusion/issues/8637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Re-enable support for ListingTables with custom file formats [datafusion]
Posted by "pgwhalen (via GitHub)" <gi...@apache.org>.
pgwhalen commented on issue #8637:
URL: https://github.com/apache/datafusion/issues/8637#issuecomment-2113374715
Closing due to fix: https://github.com/apache/datafusion/pull/10499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org