[GitHub] [incubator-iceberg] edgarRd opened a new issue #213: Incomplete Iceberg to ORC column mapping

[GitBox <gi...@apache.org>]

edgarRd opened a new issue #213: Incomplete Iceberg to ORC column mapping
URL: https://github.com/apache/incubator-iceberg/issues/213
 
 
   Current implementation of ORC does not support mapping of Iceberg schemas fully to ORC schema. ORC assigns indices to [columns based on the a pre-order traversal of the type tree](https://orc.apache.org/specification/ORCv1/), while in Iceberg IDs are more flexible. 
   Specifically, the roundtrip conversion of Iceberg and ORC schemas is not correct, the following test case fails:
   ```
     @Test
     public void testRoundtripConversion() {
       ColumnIdMap mapping = new ColumnIdMap();
       Schema schema = new Schema(
           optional(1, "intCol", Types.IntegerType.get()),
           optional(3, "longCol", Types.LongType.get()),
           optional(6, "intCol2", Types.IntegerType.get()),
           required(9, "doubleCol", Types.DoubleType.get())
       );
       TypeDescription orcSchema = TypeConversion.toOrc(schema, mapping);
       assertEquals(schema.asStruct(),
           TypeConversion.fromOrc(orcSchema, mapping).asStruct());
     }
   ```
   
   This issue makes it harder to provide a correct mapping for creating ORC metrics. The original ORC code [stores the column mapping between Iceberg and ORC in the file metadata section](https://github.com/apache/incubator-iceberg/blob/master/orc/src/main/java/org/apache/iceberg/orc/OrcFileAppender.java#L166), but fails to deserialize on read. Hence, we need to read the Iceberg schema if available in the metadata since there's a need to have an Iceberg Schema when reading an ORC file to provide the correct mapping, otherwise the best we can do is to infer it based on the original ORC ids.
   
   Additionally, the `ColumnIdMap` structure does not include whether the field is required or optional and since all columns in ORC can have nulls, the assumption in this code is that the conversion re-creates all columns as optional, failing to create an accurate Iceberg schema which could be different from the one stored.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org