[GitHub] [pinot] klsince commented on a diff in pull request #10263: upload large s3 object by multi parts to overcome the 5GB limit of PutObject

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10263:
URL: https://github.com/apache/pinot/pull/10263#discussion_r1103352879


##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3Config.java:
##########
@@ -49,6 +50,13 @@ public class S3Config {
   public static final String EXTERNAL_ID = "externalId";
   public static final String SESSION_DURATION_SECONDS = "sessionDurationSeconds";
   public static final String ASYNC_SESSION_UPDATED_ENABLED = "asyncSessionUpdateEnabled";
+  public static final String MIN_OBJECT_SIZE_FOR_MULTI_PART_UPLOAD = "minObjectSizeForMultiPartUpload";
+  public static final String MULTI_PART_UPLOAD_PART_SIZE = "multiPartUploadPartSize";
+  public static final String MULTI_PART_UPLOAD_MAX_PART_NUM_ALLOWED = "multiPartUploadMaxPartNumAllowed";

Review Comment:
   I was thinking this config may be helpful when S3 side updates this limit. But a constant works anyway, as it'd be rare for that to change. I'll add a constant for the 5MB part size limit too for the config validation said below. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org