[GitHub] [iceberg] kbendick commented on pull request #3723: Demonstrate issue where Iceberg's parquet reader silently returns nulls

[GitBox <gi...@apache.org>]

kbendick commented on pull request #3723:
URL: https://github.com/apache/iceberg/pull/3723#issuecomment-992082883


   > "bag" is added by hive's parquet serde, ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java. https://github.com/apache/hive/blob/release-1.2.1/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java#L53. We use this serde from hive's 1.2.1 version. However, even latest versions still use "bag", https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java.
   > 
   > Another thing to note here is that even if we use `list` as name for repeated group of a list, I think Iceberg returns just empty list. I am working on a fix for this, along with backward compatibility support. Would love to get your reviews on that as well.
   
   I’d be happy to take a look when you have a fix. We applied a fix for Iceberg 0.12.1 where we stopped relying on the name provided in the parquet-schema for map fields.
   
   I think something like that would be the best route forward so that we aren’t imposing any restrictions on the parquet provided names as we’ve observed these change across systems and parquet versions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org