You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "laskoviymishka (via GitHub)" <gi...@apache.org> on 2026/04/03 12:34:47 UTC

[I] feat: enforce sort order on write [iceberg-go]

laskoviymishka opened a new issue, #833:
URL: https://github.com/apache/iceberg-go/issues/833

   ### Feature Request / Improvement
   Parent: #829 (v2 spec completion)
   Sort orders are fully modeled in metadata — `SortOrder`, `SortField`, direction, null order, transforms — and tracked through schema/spec evolution. However the write path never consults the table's sort order. Data is written in arrival order regardless of the `default-sort-order` setting.
   
   ### Current state
   - `SortOrder` / `SortField` types with JSON marshaling, validation, compatibility checks — `table/sorting.go`
   - `MetadataBuilder.AddSortOrder`, `SetDefaultSortOrder` — fully wired
   - `WriteTask.SortOrderID` field exists (`table/writer.go`) but is never populated or used
   - `writerFactory` in `table/rolling_data_writer.go` has access to metadata but doesn't read the sort order
   - Arrow Go compute has no sort kernel, but `compute.Take` exists (used for positional deletes in #825, #762) — can reorder batches by permutation index
   
   ### Context
   Sort order enforcement improves query performance for engines that can exploit sorted data (range scans, merge joins). It's also needed for sort-aware compaction (#832) — when compacting, the output should respect the table's sort order.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] feat: enforce sort order on write [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #833:
URL: https://github.com/apache/iceberg-go/issues/833#issuecomment-4185082807

   Definitely open the PR and we can have the discussion there! Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] feat: enforce sort order on write [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #833: feat: enforce sort order on write
URL: https://github.com/apache/iceberg-go/issues/833


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] feat: enforce sort order on write [iceberg-go]

Posted by "hamilton-earthscope (via GitHub)" <gi...@apache.org>.
hamilton-earthscope commented on issue #833:
URL: https://github.com/apache/iceberg-go/issues/833#issuecomment-4184310839

   I have a wip sort implementation for arrow-go but could use some feedback on https://github.com/apache/arrow-go/issues/66. Also happy to just open a pr and have the discussion there. What do you think @zeroshade 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] feat: enforce sort order on write [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #833:
URL: https://github.com/apache/iceberg-go/issues/833#issuecomment-4185084350

   In the meantime, i'll reply to that message (I completely lost track of it when you had posted it so my bad there.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] feat: enforce sort order on write [iceberg-go]

Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on issue #833:
URL: https://github.com/apache/iceberg-go/issues/833#issuecomment-4348117921

   @zeroshade  @laskoviymishka I have a draft implementation for this. 
   
   The write path now resolves the table's default sort order and sorts each record batch before writing, respecting field priority, asc/desc direction, null ordering, and transforms. It uses compute.Take with a stable permutation index to reorder batches, same pattern already used for positional deletes and partition fanout. All write paths are covered (Append, Overwrite, Delete, WriteRecords, and compaction). Current limitation is per-batch sorting rather than global per-file sort, which matches PyIceberg's approach. 
   
   Happy to open a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org