You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "NGA-TRAN (via GitHub)" <gi...@apache.org> on 2023/10/19 21:37:51 UTC
[I] Make data type of FileScanConfig.table_partition_cols a vector of Field [arrow-datafusion]
NGA-TRAN opened a new issue, #7875:
URL: https://github.com/apache/arrow-datafusion/issues/7875
### Is your feature request related to a problem or challenge?
Currently, the `table_partition_cols` of `FileScanConfig` is a vector of `(String, DataType)` to store only column name and data type https://github.com/apache/arrow-datafusion/blob/37d6bf08c948418fe6c72d072d988c2875d81e02/datafusion/core/src/datasource/physical_plan/file_scan_config.rs#L104
In our IOx case, we use a column has more meaning than just `name` and `data type` and need the [Field](https://github.com/apache/arrow-rs/blob/7e134f4d277c0b62c27529fc15a4739de3ad0afd/arrow-schema/src/field.rs#L37C5-L37C5) data type to convey that.
### Describe the solution you'd like
Replace `pub table_partition_cols: Vec<(String, DataType)>,`
with `pub table_partition_cols: Vec<Field>,`
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make data type of FileScanConfig.table_partition_cols a vector of Field [arrow-datafusion]
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on issue #7875:
URL: https://github.com/apache/arrow-datafusion/issues/7875#issuecomment-1772312399
That sounds reasonable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make data type of FileScanConfig.table_partition_cols a vector of Field [arrow-datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #7875:
URL: https://github.com/apache/arrow-datafusion/issues/7875#issuecomment-1772563761
> It is reasonable in theory, however, this will likely be a non trivial API change, so we should take significant care needs to be taken on the API to make migration as easy as possible (for example, a function that takes the old data type and makes a new one), or maybe a with_partition_columns type function
Agreed. I will keep the current API and add one more to make sure it still works for current folks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make data type of FileScanConfig.table_partition_cols a vector of Field [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan closed issue #7875: Make data type of FileScanConfig.table_partition_cols a vector of Field
URL: https://github.com/apache/arrow-datafusion/issues/7875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make data type of FileScanConfig.table_partition_cols a vector of Field [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7875:
URL: https://github.com/apache/arrow-datafusion/issues/7875#issuecomment-1772455875
It is reasonable in theory, however, this will likely be a non trivial API change, so we should take significant care needs to be taken on the API to make migration as easy as possible (for example, a function that takes the old data type and makes a new one), or maybe a `with_partition_columns` type function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make data type of FileScanConfig.table_partition_cols a vector of Field [arrow-datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #7875:
URL: https://github.com/apache/arrow-datafusion/issues/7875#issuecomment-1772811713
I start working on this today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make data type of FileScanConfig.table_partition_cols a vector of Field [arrow-datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #7875:
URL: https://github.com/apache/arrow-datafusion/issues/7875#issuecomment-1771746072
@alamb
If there is no objection, I will start working on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org