You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Jefffrey (via GitHub)" <gi...@apache.org> on 2023/11/29 12:06:19 UTC

[I] Parquet: Interval columns shouldn't write min/max stats [arrow-rs]

Jefffrey opened a new issue, #5145:
URL: https://github.com/apache/arrow-rs/issues/5145

   **Describe the bug**
   <!--
   A clear and concise description of what the bug is.
   -->
   
   See parquet spec:
   
   > The sort order used for `INTERVAL` is undefined. When writing data, no min/max statistics should be saved for this type and if such non-compliant statistics are found during reading, they must be ignored.
   
   - https://github.com/apache/parquet-format/blob/066f9817332da32bdc6dc6dea833b6ee9c269934/LogicalTypes.md?plain=1#L537-L539
   
   **To Reproduce**
   <!--
   Steps to reproduce the behavior:
   -->
   
   Test in https://github.com/apache/arrow-rs/blob/ef6932f31e243d8545e097569653c8d3f1365b4d/parquet/src/column/writer/mod.rs:
   
   ```rust
       #[test]
       fn test_interval_should_not_have_min_max() {
           let input = [
               vec![0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
               vec![0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
               vec![0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2],
           ]
           .into_iter()
           .map(|s| ByteArray::from(s).into())
           .collect::<Vec<_>>();
   
           let page_writer = get_test_page_writer();
           let mut writer = get_test_interval_column_writer(page_writer);
           writer.write_batch(&input, None, None).unwrap();
   
           let metadata = writer.close().unwrap().metadata;
           let stats = if let Some(Statistics::FixedLenByteArray(stats)) = metadata.statistics() {
               stats.clone()
           } else {
               panic!("metadata missing statistics");
           };
           assert!(!stats.has_min_max_set());
       }
   
       fn get_test_interval_column_writer(
           page_writer: Box<dyn PageWriter>,
       ) -> ColumnWriterImpl<'static, FixedLenByteArrayType> {
           let descr = Arc::new(get_test_interval_column_descr());
           let column_writer = get_column_writer(descr, Default::default(), page_writer);
           get_typed_column_writer::<FixedLenByteArrayType>(column_writer)
       }
   
       fn get_test_interval_column_descr() -> ColumnDescriptor {
           let path = ColumnPath::from("col");
           let tpe =
               SchemaType::primitive_type_builder("col", FixedLenByteArrayType::get_physical_type())
                   .with_length(12)
                   .with_converted_type(ConvertedType::INTERVAL)
                   .build()
                   .unwrap();
           ColumnDescriptor::new(Arc::new(tpe), 0, 0, path)
       }
   ```
   
   **Expected behavior**
   <!--
   A clear and concise description of what you expected to happen.
   -->
   
   Should pass test. Currently fails as seems min/max is being set.
   
   **Additional context**
   <!--
   Add any other context about the problem here.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Parquet: Interval columns shouldn't write min/max stats [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5145: Parquet: Interval columns shouldn't write min/max stats
URL: https://github.com/apache/arrow-rs/issues/5145


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Parquet: Interval columns shouldn't write min/max stats [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5145:
URL: https://github.com/apache/arrow-rs/issues/5145#issuecomment-1878523290

   `label_issue.py` automatically added labels {'parquet'} from #5147


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org