[I] [Bug] Case-insensitive read of an ARRAY column throws ClassCastException [paimon]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang opened a new issue, #9567:
URL: https://github.com/apache/paimon/issues/9567

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   ### Paimon version
   
   master, `2788fe596` (2.1-SNAPSHOT).
   
   ### Compute Engine
   
   Flink and Spark, with catalog `case-sensitive = false` (the case-insensitive Parquet reads added in #8337), on a Parquet file whose column names are not lower case. Spark keeps the DDL spelling in the file footer while Hive Metastore lower-cases column names, so a format table over Spark-written files hits this, as does a table produced by `migrate_table`.
   
   ### Minimal reproduce step
   
   A Parquet file with a three-level LIST column spelled `Tags`:
   
   ```
   message root {
     optional group Tags (LIST) {
       repeated group list {
         optional binary element (UTF8);
       }
     }
   }
   ```
   
   read as `tags ARRAY<STRING>` with `case-sensitive = false`:
   
   ```
   java.lang.ClassCastException: org.apache.parquet.io.PrimitiveColumnIO cannot be cast to org.apache.parquet.io.GroupColumnIO
       at org.apache.paimon.format.parquet.reader.ParquetReaderUtil.buildFieldsList(ParquetReaderUtil.java:173)
   ```
   
   The ARRAY branch of `buildFieldsList` descends through the wrapper groups by name:
   
   ```java
   while (!Objects.equals(groupColumnIO.getName(), fieldName)) {
       groupColumnIO = (GroupColumnIO) groupColumnIO.getChild(0);
   }
   ```
   
   `Tags` never equals `tags`, so it walks `Tags` to `list` to `element`, and `element` is a `PrimitiveColumnIO`.
   
   ### What doesn't meet your expectations?
   
   The other two name lookups in the same file resolve case-insensitively on purpose. `lookupColumnByName`'s javadoc says "Parquet's column names are case insensitive. So when we look up columns we first check for exact match, and if that can not find we look for a case-insensitive match", and `getTypeIgnoreCase` does the same. `lookupColumnByName` has ignored case since #5905 and `getTypeIgnoreCase` was added by #8337, which is where case-insensitive reads came from. This loop was left comparing case-sensitively, so the feature works for scalar and nested-row columns and crashes on an array column; #8337's tests cover the first two shapes only.
   
   ### Anything else?
   
   MAP and MULTISET are not affected: `getMapKeyValueColumn` descends structurally on `getChildrenCount() == 1` and never compares a name. The ROW branch already goes through the two case-insensitive helpers.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Case-insensitive read of an ARRAY column throws ClassCastException [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed issue #9567: [Bug] Case-insensitive read of an ARRAY column throws ClassCastException
URL: https://github.com/apache/paimon/issues/9567


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org