You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aihuaxu (via GitHub)" <gi...@apache.org> on 2026/05/21 22:10:46 UTC

[I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

aihuaxu opened a new issue, #16519:
URL: https://github.com/apache/iceberg/issues/16519

   ### Apache Iceberg version
   
   1.11.0 (latest release)
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
    ### Description
     
   SerializableTable.sortOrders() throws a ValidationException when a table has a historical sort order that references a column that has since been dropped from the schema.
   
   This was introduced in #15150, which added serialization of all sort orders (not just the default) to SerializableTable. During deserialization the new code binds every sort order against the current schema using strict validation, but historical sort orders may legitimately reference fields that no longer exist.
     
   ### Root cause
   
   
   Steps to reproduce
   
   ```
     @Test
     public void testSerializableTableSortOrdersWithDroppedColumn() {
       // add an extra column and establish sort order 1 on id + ts
       table.updateSchema().addColumn("ts", Types.LongType.get()).commit();
       table.replaceSortOrder().asc("id").asc("ts").commit();
       table.newAppend().appendFile(FILE_A).commit();
   
       // switch to sort order 2 using only "id", then drop "ts"
       // historical sort order 1 still references the dropped "ts" field
       table.replaceSortOrder().asc("id").commit();
       table.updateSchema().deleteColumn("ts").commit();
   
       // SerializableTable.copyOf captures all sort orders at construction time
       Table serialized = SerializableTable.copyOf(table);
   
       // sortOrders() must return all historical orders without throwing a ValidationException,
       // even though sort order 1 references the now-dropped "ts" field
       assertThat(serialized.sortOrders()).hasSize(3); // unsorted(0), order-1(id+ts), order-2(id)
       assertThat(serialized.sortOrder().fields()).hasSize(1); // current default is order-2
     }
   ```
   
   ### Expected behavior
   
   sortOrders() should return all historical sort orders without throwing, binding non-default sort orders unchecked (as TableMetadataParser and PartitionSpec already do for the equivalent case).
   
   
   
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on issue #16519:
URL: https://github.com/apache/iceberg/issues/16519#issuecomment-4626564828

   I would like to work on this. 
   Will repro the issue and create a PR with the fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on issue #16519:
URL: https://github.com/apache/iceberg/issues/16519#issuecomment-4643421199

   @aihuaxu Please close the issue as fixed. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on issue #16519:
URL: https://github.com/apache/iceberg/issues/16519#issuecomment-4643420601

   @pallevam This has already been fixed by PR #16521 . 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck closed issue #16519: SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column
URL: https://github.com/apache/iceberg/issues/16519


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on issue #16519:
URL: https://github.com/apache/iceberg/issues/16519#issuecomment-4536097243

   I'd like to work on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on issue #16519:
URL: https://github.com/apache/iceberg/issues/16519#issuecomment-4514357633

   I feel storing the sort order IDs in the manifest should be thought through a bit more, and for now should be reverted in the next 1.11 patch release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

Posted by "pallevam (via GitHub)" <gi...@apache.org>.
pallevam commented on issue #16519:
URL: https://github.com/apache/iceberg/issues/16519#issuecomment-4643311120

   I would like to work on this bug. Please assign to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column [iceberg]

Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on issue #16519:
URL: https://github.com/apache/iceberg/issues/16519#issuecomment-4513536870

   Yes, hit the same issue today and here is a reproducible:
   ```sh
   CREATE TABLE polaris.db.sort_orphan_repro (
       id      bigint,
       time    timestamp,
       source  string,
       payload string
   )
   USING iceberg
   PARTITIONED BY (days(time))
   TBLPROPERTIES ('format-version' = '2');
   
   ALTER TABLE polaris.db.sort_orphan_repro WRITE ORDERED BY time DESC NULLS LAST, source ASC NULLS LAST;
   
   INSERT INTO polaris.db.sort_orphan_repro VALUES (1, timestamp '2026-05-20 00:00:00', 'a', 'p1');
   
   ALTER TABLE polaris.db.sort_orphan_repro DROP PARTITION FIELD days(time);
   ALTER TABLE polaris.db.sort_orphan_repro WRITE UNORDERED;
   ALTER TABLE polaris.db.sort_orphan_repro DROP COLUMN time;
   ALTER TABLE polaris.db.sort_orphan_repro ADD COLUMN time timestamp;
   ALTER TABLE polaris.db.sort_orphan_repro ADD PARTITION FIELD days(time);
   ALTER TABLE polaris.db.sort_orphan_repro WRITE ORDERED BY time DESC NULLS LAST, source ASC NULLS LAST;
   
   # this will fail
   INSERT INTO polaris.db.sort_orphan_repro (id, time, source, payload) VALUES (2, timestamp '2026-05-21 00:00:00', 'b', 'p2');
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org