[I] coerce_int96 leaves INT96 in two-level and legacy Parquet lists as nanoseconds [datafusion]

["ranflarion (via GitHub)" <gi...@apache.org>]

ranflarion opened a new issue, #26463:
URL: https://github.com/apache/datafusion/issues/26463

   ### Describe the bug
   
   With `datafusion.execution.parquet.coerce_int96` set, INT96 timestamps inside lists that do not use the three-level LIST layout stay `Timestamp(Nanosecond, None)`. That covers Hive's two-level lists (`repeated int96 bag`, how Hive writes `array<timestamp>`), bare `repeated int96` fields, and parquet-mr Avro's legacy lists whose repeated group is named `array`. Values outside 1677-09-21 to 2262-04-11 then overflow on decode, which is what `coerce_int96` is meant to prevent: `0001-01-01T01:02:03.123456` reads back in 1754.
   
   ### To Reproduce
   
   ```rust
   use std::sync::Arc;
   use arrow::datatypes::TimeUnit;
   use datafusion_datasource_parquet::Int96Coercer;
   use parquet::arrow::parquet_to_arrow_schema;
   use parquet::schema::{parser::parse_message_type, types::SchemaDescriptor};
   
   let descr = SchemaDescriptor::new(Arc::new(
       parse_message_type(
           "message m {
             optional group hive (LIST) { repeated int96 bag; }
             optional group three (LIST) { repeated group list { optional int96 element; } }
           }",
       )
       .unwrap(),
   ));
   let arrow_schema = parquet_to_arrow_schema(&descr, None).unwrap();
   let coerced = Int96Coercer::new(&descr, &arrow_schema, &TimeUnit::Microsecond)
       .coerce()
       .unwrap();
   println!("{:?}", coerced.field_with_name("hive").unwrap().data_type());
   println!("{:?}", coerced.field_with_name("three").unwrap().data_type());
   ```
   
   `hive` keeps `Timestamp(Nanosecond, None)` while `three` becomes `Timestamp(Microsecond, None)`.
   
   ### Expected behavior
   
   Every INT96 leaf coerces to the requested unit, whatever the list encoding.
   
   ### Additional context
   
   `coerce_int96_to_resolution_impl` in `datasource-parquet/src/schema_coercion.rs` finds INT96 leaves by rebuilding each Arrow leaf's Parquet path, and assumes a list's path is `<list>.list.<element>`. For `hive` it looks up `hive.list.bag`, which no column has, since the path is `hive.bag`. It also never descends `LargeList` or `FixedSizeList`.
   
   ### Willingness to contribute
   
   I plan to contribute a fix for this bug independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org