[I] Native Parquet scan cannot read a file whose schema has a nested encoding arrow-rs rejects, even in a column the query does not read [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #7035:
URL: https://github.com/apache/datafusion-comet/issues/7035

   ### Describe the bug
   
   The native Parquet scan fails to read a file whose schema contains a nested encoding that arrow-rs cannot convert, even when the query does not read that column. Spark clips the file schema to the requested columns before converting it, so it reads the other columns. arrow-rs converts the whole file schema when it opens the file.
   
   Two shapes that make `SELECT id` fail:
   
   - A LIST-annotated group that is itself repeated, `repeated group rl (LIST) { repeated group list { optional int32 element; } }`. Spark infers `rl` as `array<array<int>>`. Comet fails with `Parquet error: Arrow: List type cannot be repeated`.
   - A MAP whose `key_value` group has a third field. Spark cannot infer that column's type, but reads the other columns. Comet fails with `Parquet error: Arrow: Child of map field must have two children, found 3`.
   
   A related shape fails when it is read: `optional group a (LIST) { repeated group element { repeated int32 v; } }`, a list whose single element field is itself repeated. Spark infers `array<int>`, and Comet fails with `column: [a, list, v], physicalType: UNKNOWN, logicalType: int`. Spark's own answer there is lossy (a record holding `v = [1, 2]` reads as `[1]`), so this one matters less.
   
   ### Steps to reproduce
   
   Write each schema with parquet-mr's `RecordConsumer`, with one record that sets only `id = 1`, then read it with `spark.read.schema("id int").parquet(path).collect()`.
   
   On `main` at 66c63bd307, the same on Spark 3.5.9 and 4.1.3:
   
   | File column | Spark | Comet native scan |
   | --- | --- | --- |
   | `repeated group rl (LIST) { repeated group list { optional int32 element; } }` | `[1]` | `Parquet error: Arrow: List type cannot be repeated` |
   | `optional group m3 (MAP) { repeated group key_value { required binary key (UTF8); optional int32 value; optional int32 extra; } }` | `[1]` | `Parquet error: Arrow: Child of map field must have two children, found 3` |
   
   ### Expected behavior
   
   A column the query does not read does not stop the native scan from reading the others. Failing that, the native scan could fall back for such files, but the file schema is only known when a task opens the file.
   
   ### Additional context
   
   Found in a nested-type audit of the native scans. These encodings are non-standard, and no common writer is known to produce them, so this is low priority.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org