[GitHub] [iceberg] jackye1995 commented on a change in pull request #4052: API: Add deleteFiles to FileIO. S3FileIO implementation will perform a batch deletion using RemoveObjects API

[GitBox <gi...@apache.org>]

jackye1995 commented on a change in pull request #4052:
URL: https://github.com/apache/iceberg/pull/4052#discussion_r810577628



##########
File path: aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java
##########
@@ -171,6 +171,25 @@
   public static final String S3_CHECKSUM_ENABLED = "s3.checksum-enabled";
   public static final boolean S3_CHECKSUM_ENABLED_DEFAULT = false;
 
+  /**
+   * Configure the batch size used when deleting multiple files from a given S3 bucket
+   */
+  public static final String S3FILEIO_DELETE_BATCH_SIZE = "s3.delete.batch-size";
+
+  /**
+   * Default batch size used when deleting files.
+   * <p>
+   * Refer to https://github.com/apache/hadoop/commit/56dee667707926f3796c7757be1a133a362f05c9
+   * for more details on why this value was chosen.
+   */
+  public static final int S3FILEIO_DELETE_BATCH_SIZE_DEFAULT = 250;
+
+  /**
+   * Max possible batch size for deletion. Currently, a max of 1000 keys can be deleted in one batch.
+   * https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
+   */
+  public static final int S3FILEIO_DELETE_MAX_BATCH_SIZE = 1000;

Review comment:
       nit `S3FILEIO_DELETE_BATCH_SIZE_MAX`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org