[GitHub] [iceberg] RussellSpitzer edited a comment on pull request #2877: Spark: Fix nested struct pruning

[GitBox <gi...@apache.org>]

RussellSpitzer edited a comment on pull request #2877:
URL: https://github.com/apache/iceberg/pull/2877#issuecomment-887882956


   I added another test and found an issue with my fix. We create a StructInternalRow with a "type" and it's data "struct"
   When we actually prune the projected record's "type" does not match the struct layout, this is an issue for Map, Array, Struct types.
   
   For example, if we make a BaseFile with a projection we end up with a mapping of ordinal to physical entry in the row, but the StructInternal row does *not* do the same mapping when looking up types. So although my fix works for all non parameterized lookups, it fails on parameterized types since the parameterized types are looked up based on the original layout and not the pruned one.
   
   Ie
   StructInternalRow(TableLayout) stores
   ```
   Types = ( int, string, int, list)
   data = ([1,2], "hello") with Map ( 0 -> 3, 2 -> 1)
   ```
   Since the type lookup doesn't know about the projection it is incorrect ... trying to figure out a fix without breaking everything
   
   This isn't an issue for setters or getters which know their type, since they never touch the "Types" struct.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org