[GitHub] [iceberg] aokolnychyi edited a comment on issue #4031: TestCopyOnWriteUpdate::testUpdateWithoutCondition is flaky

[GitBox <gi...@apache.org>]

aokolnychyi edited a comment on issue #4031:
URL: https://github.com/apache/iceberg/issues/4031#issuecomment-1032851191


   @szehon-ho @rdblue, I think the problem is that we request a hash distribution by `_file` and sometimes records for multiple files end up in a single output task. That leads to an inconsistent number of output files. It used to happen frequently when we had only 4 shuffle partitions but then I changed it to 200. Looks like this is still not enough and we should come up with a more reliable fix.
   
   ```
   // set the num of shuffle partitions to 200 instead of default 4 to reduce the chance of hashing
   // records for multiple source files to one writing task (needed for a predictable num of output files)
   withSQLConf(ImmutableMap.of(SQLConf.SHUFFLE_PARTITIONS().key(), "200"), () -> {
     sql("UPDATE %s SET id = -1", tableName);
   });
   ```
   
   Let me see the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org