You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Omega359 (via GitHub)" <gi...@apache.org> on 2024/08/28 17:51:45 UTC
[I] Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary [datafusion]
Omega359 opened a new issue, #12221:
URL: https://github.com/apache/datafusion/issues/12221
### Is your feature request related to a problem or challenge?
Currently in demux::compute_partition_keys_by_row the only supported types for writing out partitions seems to be just `DataType::Utf8` and `DataType::Dictionary(_, _)`. I think there is opportunity to support a number of other `DataType`s such as int/uint 8/32/64 types, Date32 (with a fixed format 'yyyy-MM-dd') and bool.
### Describe the solution you'd like
Code and tests for writing out hive-style partitions includes additional datatypes beyond just utf8 and Dictionary
### Describe alternatives you've considered
Cast field to utf8 prior to output.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary [datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #12221:
URL: https://github.com/apache/datafusion/issues/12221#issuecomment-2323450374
The partitioning is tested in sqllogictests. See https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/test_files/copy.slt#L83
The files are written to partitioned directories and then a specific directory is queried to make sure it was partitioned as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #12221:
URL: https://github.com/apache/datafusion/issues/12221#issuecomment-2322730390
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #12221:
URL: https://github.com/apache/datafusion/issues/12221#issuecomment-2318249089
> I think this is a good idea. The only thing to watch out for is what happens when you read the partitions back using datafusion.
>
> E.g. if you write out a date column as the partition into 'some/path/some_col=2023-01-01/...', will datafusion parse this back into a date32 column? I am not as familiar with the read path, but we would want to make sure type handling is consistent between reading and writing partitioned columns.
Ah, good point. I'll verify that when I put in a PR for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #12221:
URL: https://github.com/apache/datafusion/issues/12221#issuecomment-2323441801
I've made the changes to a branch in my [repo for this ](https://github.com/Omega359/arrow-datafusion/blob/aa17e4d30a45c30e30230c5ae46471888dcaef78/datafusion/core/src/datasource/file_format/write/demux.rs#L324) however I can't seem to find any place where this fn (or hive partitioning at all) is actually tested. Any suggestions as to where to add a test for this would be appreciated. The demux functionality is quite complex so I was thinking that maybe something at a much higher level would be best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary [datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #12221:
URL: https://github.com/apache/datafusion/issues/12221#issuecomment-2318244348
I think this is a good idea. The only thing to watch out for is what happens when you read the partitioned back using datafusion.
E.g. if you write out a date column as the partition into 'some/path/some_col=2023-01-01/...', will datafusion parse this back into a date32 column? I am not as familiar with the read path, but we would want to make sure type handling is consistent between reading and writing partitioned columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #12221: Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary
URL: https://github.com/apache/datafusion/issues/12221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org