[I] Clustered (fanout disabled) writes fail on sorted input because batches are split in HashMap order [datafusion-iceberg]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #39:
URL: https://github.com/apache/datafusion-iceberg/issues/39

   ### Describe the bug
   
   With fanout disabled, `TaskWriter` splits each batch with iceberg-rust's `RecordBatchPartitionSplitter::split` and hands the pieces to `ClusteredWriter` ([task_writer.rs#L129-L135](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/task_writer.rs#L129-L135), [task_writer.rs#L217-L237](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/task_writer.rs#L217-L237)).
   
   The splitter groups rows in a `HashMap`, so the partitions within a batch come out in arbitrary order. `ClusteredWriter` closes the current partition when it sees a different one, and rejects any partition it has already closed. Input that is correctly sorted by partition therefore fails whenever a partition spans a batch boundary. For example, batch N holds [A, B] but is emitted as B then A, and batch N+1 starts with B.
   
   ### To Reproduce
   
   Execute the plan returned by `IcebergTableProvider::insert_into` directly, so the `SortExec` is still present (#38 explains why the optimized plan loses it). Use a table partitioned by `identity(cat)` with fanout disabled, 200 rows with `cat = id % 10`, `batch_size = 8` and `target_partitions = 1`.
   
   9 of 10 runs fail with:
   
   ```
   The input is not sorted! Cannot write to partition that was previously closed
   ```
   
   ### Expected behavior
   
   Input sorted by partition is written successfully.
   
   ### Additional context
   
   Fixing #38 alone won't make fanout-disabled writes work. The splitter lives in iceberg-rust (`crates/iceberg/src/arrow/record_batch_partition_splitter.rs`). Either it emits partitions in first-occurrence order, or `TaskWriter` orders each batch's pieces by row position before passing them to `ClusteredWriter`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org