You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "xiaoxuandev (via GitHub)" <gi...@apache.org> on 2026/08/25 20:06:36 UTC
[PR] Spark 4.1: Fix rewrite failure on tables created with a sort order [iceberg]
xiaoxuandev opened a new pull request, #17814:
URL: https://github.com/apache/iceberg/pull/17814
## Problem
`rewriteDataFiles` fails on a table that declared a sort order at creation when the output is unsorted relative to that order — always for z-order, and for sort whenever the requested order matches none the table declares
```
java.lang.IllegalArgumentException: Cannot use output sort order id 0 because the table does not contain a sort order with that id
```
## Root cause
`SparkShufflingFileRewriteRunner` sets `output-sort-order-id` from `SortOrderUtil.findTableSortOrder`, which returns `SortOrder.unsorted()` (id 0) when no table sort order matches. `SparkWriteConf.outputSortOrderId` then
rejected that id unless the table contained it:
```java
Preconditions.checkArgument(table.sortOrders().containsKey(explicitId), ...);
```
Id 0 is reserved for unsorted order and is valid for any table, but `sortOrders()` contains it only when the table was created unsorted. A table that declares a sort order at creation (only possible through the Iceberg API, REST, or another engine — `SparkCatalog` never sets one) gets `INITIAL_SORT_ORDER_ID = 1` and never lists 0, so the check rejected an id the rewrite legitimately produced. The implicit branch of the same method already returns that id without consulting the table.
## Fix
Accept the reserved unsorted order id without requiring the table to list it. Unknown ids are still rejected. Downstream, `table.sortOrders().get(0)` yields `null`, `DataFiles.Builder.withSortOrder` ignores it, and the file is written with `sort_order_id = 0`, which is the correct value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.1: Fix rewrite failure on tables created with a sort order [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho merged PR #17814:
URL: https://github.com/apache/iceberg/pull/17814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.1: Fix rewrite failure on tables created with a sort order [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17814:
URL: https://github.com/apache/iceberg/pull/17814#issuecomment-5841494014
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.1: Fix rewrite failure on tables created with a sort order [iceberg]
Posted by "rshdhere (via GitHub)" <gi...@apache.org>.
rshdhere commented on PR #17814:
URL: https://github.com/apache/iceberg/pull/17814#issuecomment-6158236388
Hi @xiaoxuandev, thanks for this fix! Spark 4.2 was copied from 4.1 before it merged, so 4.2 still had the bug. I opened https://github.com/apache/iceberg/pull/18493 to apply the same change there. Are you still planning the 3.5 and 4.0 backport? If you're busy, I'd be happy to take it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.1: Fix rewrite failure on tables created with a sort order [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #17814:
URL: https://github.com/apache/iceberg/pull/17814#issuecomment-5841818092
Merged, thanks @xiaoxuandev and @uros-b for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org