You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/06/05 17:37:24 UTC
[I] Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray` [arrow-rs]
alamb opened a new issue, #5847:
URL: https://github.com/apache/arrow-rs/issues/5847
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
As we worked on filling out support for converting parquet statistics into ArrayRefs in DataFusion (see https://github.com/apache/datafusion/issues/10453) @marvinlanhenke discovered that
**Describe the solution you'd like**
Support writing the following types of data with statistics in the parquet [ArrowWriter](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html#)
* [IntervalDayTimeArray](https://docs.rs/arrow/latest/arrow/array/type.IntervalDayTimeArray.html)
* [IntervalMonthDayNanoArray](https://docs.rs/arrow/latest/arrow/array/type.IntervalMonthDayNanoArray.html)
* [IntervalYearMonthArray](https://docs.rs/arrow/latest/arrow/array/type.IntervalYearMonthArray.html)
This might be the location: https://github.com/apache/arrow-rs/blob/fa8d3502388d7cfac724f7b9fae92abc3a716b6f/parquet/src/arrow/arrow_writer/mod.rs#L747
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
https://github.com/apache/datafusion/pull/10801#issuecomment-2149452632
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #5847: Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray`
URL: https://github.com/apache/arrow-rs/issues/5847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray` [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5847:
URL: https://github.com/apache/arrow-rs/issues/5847#issuecomment-2160626641
It is incorrect to write statistics for interval types, as per the format specification - https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#interval
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5847:
URL: https://github.com/apache/arrow-rs/issues/5847#issuecomment-2152334490
@marvinlanhenke also points out that to completely support this ticket probably requires completing https://github.com/apache/arrow-rs/issues/5849 first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #5847: Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray`
URL: https://github.com/apache/arrow-rs/issues/5847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5847:
URL: https://github.com/apache/arrow-rs/issues/5847#issuecomment-2161417901
That makes sense. Thank you @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write parquet statistics for `IntervalDayTimeArray` , `IntervalMonthDayNanoArray` and `IntervalYearMonthArray` [arrow-rs]
Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on issue #5847:
URL: https://github.com/apache/arrow-rs/issues/5847#issuecomment-2150628259
for additional context and possibly [code pointers](https://github.com/apache/datafusion/issues/10752#issuecomment-2150024521) where Interval support is missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org