You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/08/17 18:02:06 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue, #7323: Consolidate `OutputFileFormat` and `FileType`
alamb opened a new issue, #7323:
URL: https://github.com/apache/arrow-datafusion/issues/7323
### Is your feature request related to a problem or challenge?
https://github.com/apache/arrow-datafusion/pull/7283#discussion_r1294561258 added a `OutputFileFormat` enum that is very similar to the existing [`FileType`](https://docs.rs/datafusion/latest/datafusion/datasource/file_format/file_type/enum.FileType.html) enum:
https://github.com/apache/arrow-datafusion/blob/f2c0100a5a10bf3ea166a1a590d94b8c9b6cf673/datafusion/expr/src/logical_plan/dml.rs#L45-L53
### Describe the solution you'd like
Move `FileType` into `datafusion_common` and remove `OutputFileFormat` so it could be used by both the logical plan and datasource?
The tricky bit of this PR will be disentangling FileType from the code that instantiates compression: https://github.com/apache/arrow-datafusion/blob/f2c0100a5a10bf3ea166a1a590d94b8c9b6cf673/datafusion/core/src/datasource/file_format/file_type.rs#L287
I think we could move `FileType` over, however, and define some trait with `get_ext_with_compression` for the appropriate code
### Describe alternatives you've considered
_No response_
### Additional context
@devinjdangelo and I discussed this here: https://github.com/apache/arrow-datafusion/pull/7283#discussion_r1294561258
I think this would be a good first issue as it is primarily a code movement thing, doesn't require intimate knowledge of DataFusion's code, and would be a good introduction to the development workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #7323: Consolidate `OutputFileFormat` and `FileType`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #7323: Consolidate `OutputFileFormat` and `FileType`
URL: https://github.com/apache/arrow-datafusion/issues/7323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] devinjdangelo commented on issue #7323: Consolidate `OutputFileFormat` and `FileType`
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #7323:
URL: https://github.com/apache/arrow-datafusion/issues/7323#issuecomment-1684414826
> The tricky bit of this PR will be disentangling FileType from the code that instantiates compression:
The idea I went with here is moving the entire file_type.rs file to common i.e.:
`git mv datafusion/core/src/datasource/file_format/file_type.rs datafusion/common/src/`
Loads of file imports to clean up after that, but it seems like it worked (PR: #7336 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org