[GitHub] [iceberg] RussellSpitzer commented on issue #7657: Unable to append to table after cancelled optmization procedure call

["RussellSpitzer (via GitHub)" <gi...@apache.org>]

RussellSpitzer commented on issue #7657:
URL: https://github.com/apache/iceberg/issues/7657#issuecomment-1554603272

   The `rewrite-datafiles` is almost certainly a red herring. It shouldn't have any effect on the table if it failed since it is transactional like all Iceberg operations. It may be that this is coincidental or perhaps the system has just run out of some resource like disk space.
   
   > Spark will attempt to load and sort the entire table before inserting the new batch even though we are partitioning by loaded_at which should never need more than one partition.
   
   This is quite confusing to me. An "append" should never require reading anything from the target table. The source though will have a shuffle applied to it based on the `write.distribution-mode` of the table. We recently did change the default to hash which would request that the incoming data be shuffled by the partition of the destination table. If you are confident it does not need to be shuffled you can always set the mode to `none`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org