You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by "behcetm (via GitHub)" <gi...@apache.org> on 2024/03/14 08:12:42 UTC

Re: [I] [Python/C++] Make upload size per part configurable when uploading to S3 [arrow]

behcetm commented on issue #40420:
URL: https://github.com/apache/arrow/issues/40420#issuecomment-1996794094

   Hi @phofl,
   
   Thanks for bringing up this topic. This limitation of the fixed size parts also prevents uploading objects larger than 100GB which I think is quite low compared to the maximum object size(5 TB) supported by AWS S3. I would be very interested for a solution to this issue.
   
   My preferred solution would be having the part size configurable per output stream. I guess that could still work for Cloudflare R2.
   
   Behcet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org