Re: [I] S3Filesystem always initiates multipart uploads, regardless of input size [arrow]

["OliLay (via GitHub)" <gi...@apache.org>]

OliLay commented on issue #40557:
URL: https://github.com/apache/arrow/issues/40557#issuecomment-2071772600

   We observe similar issues using the C++ implementation.
   This behavior adds quite a large constant overhead for writing to S3. You will at least always have 3x RTT to S3, instead of possibly just 1x RTT to S3 using a direct `PutObject` request. 
   I think the best possible solution would be to not directly omit a `CreateMultipartUpload`, but wait until some writes on the OutputStream have accumulated (e.g. >1MB?); if the OutputStream is closed with having written below 1MB, just directly use a `PutObject` request.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org