You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/06/01 20:28:12 UTC

[I] Extract parquet statistics from `Interval` columns [datafusion]

alamb opened a new issue, #10752:
URL: https://github.com/apache/datafusion/issues/10752

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Extract parquet statistics from `Interval` columns [datafusion]

Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on issue #10752:
URL: https://github.com/apache/datafusion/issues/10752#issuecomment-2148202795

   sure I can do that; from the top of my mind - the `fn run()` from the `struct Test` panics if we can't extract any statistics, which is the case here. So I'd prepare as much as possible (creating record batches, adding a Scenario, writing those tests) but for now would assert `should panic` - does this make any sense to you @alamb? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Extract parquet statistics from `Interval` columns [datafusion]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #10752: Extract parquet statistics from `Interval` columns
URL: https://github.com/apache/datafusion/issues/10752


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Extract parquet statistics from `Interval` columns [datafusion]

Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on issue #10752:
URL: https://github.com/apache/datafusion/issues/10752#issuecomment-2143979816

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Extract parquet statistics from `Interval` columns [datafusion]

Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on issue #10752:
URL: https://github.com/apache/datafusion/issues/10752#issuecomment-2144357666

   ...while looking into this I noticed, that there are no statistics written for an `Interval`, which is also described [here](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#interval).
   
   @alamb I guess we can't extract any statistics here? And writing any tests that check we have no statistics written, does not seem to be very helpful?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Extract parquet statistics from `Interval` columns [datafusion]

Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on issue #10752:
URL: https://github.com/apache/datafusion/issues/10752#issuecomment-2150024521

   I did some digging in order to find out why / or where the writing of those statistics is not supported (yet).
   Since I'm not familiar with the parquet impl, here are my findings, which might be useful in a follow-up ticket in arrow-rs.
   
   1. When trying to `fn write_slice()` the min, max values are never updated due to a [filter-condition;](https://github.com/apache/arrow-rs/blob/master/parquet/src/column/writer/encoder.rs#L137-L146) that checks if the type is INTERVAL
   2. In order to support updating the min max values, we need to handle the comparison of INTERVAL [here](https://github.com/apache/arrow-rs/blob/master/parquet/src/column/writer/mod.rs#L1122-L1167)
   
   I think this should be possible, or put differently, I don't see the reason yet, why this is not supported?
   Somethin similar (comparing FixedLenByteArrays) is already done for DECIMAL [here](https://github.com/apache/arrow-rs/blob/master/parquet/src/column/writer/mod.rs#L1200)?
   
   Perhaps, you have some more information on this @alamb - otherwise this might be enough information to file a ticket in arrow-rs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Extract parquet statistics from `Interval` columns [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10752:
URL: https://github.com/apache/datafusion/issues/10752#issuecomment-2147857707

   
   
   > @alamb I guess we can't extract any statistics here? And writing any tests that check we have no statistics written, does not seem to be very helpful?
   
   I actually think these would be helpful then as soon as there are statistics we can hook them up to the tests. If you had time to write the tests that would be great. We can then perhaps file a ticket in parquet-rs for supporting writing statistics to interval types. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org