You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/10/12 10:28:37 UTC

[GitHub] [arrow-rs] alamb opened a new issue, #2863: object_store: Support multi-part uploads to S3

alamb opened a new issue, #2863:
URL: https://github.com/apache/arrow-rs/issues/2863

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   We hit got an error from S3 when uploading a 5GB file, even though S3 supports objects up to 5TB. The reason we think is that there is a 5GB limit per "part" of a multi-part upload.
   
   Relevant AWS docs:
   
   [A single S3 PUT cannot exceed 5GB](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html)
   [Multipart uploads can be 5GB per part, up to 5TB total](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html)
   
   
   **Describe the solution you'd like**
   object_store uploads data in multiple parts so the size limit is higher. Perhaps it will also be faster
   
   **Describe alternatives you've considered**
   
   **Additional context**
   Hit in the context of https://github.com/influxdata/influxdb_iox/pull/5834


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold commented on issue #2863: object_store: Add Multipart Upload Doc Examples

Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #2863:
URL: https://github.com/apache/arrow-rs/issues/2863#issuecomment-1291454550

   Updating this issue to cover improving the documentation, in particular adding write examples to the crate docs - https://docs.rs/object_store/latest/object_store/ which already cover listing and fetching objects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold closed issue #2863: object_store: Add Put and Multipart Upload Doc Examples

Posted by GitBox <gi...@apache.org>.
tustvold closed issue #2863: object_store: Add Put and Multipart Upload Doc Examples
URL: https://github.com/apache/arrow-rs/issues/2863


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] alamb commented on issue #2863: object_store: Support multi-part uploads to S3 for larger than 5GB object sizes

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #2863:
URL: https://github.com/apache/arrow-rs/issues/2863#issuecomment-1277927660

   Maybe some improved documentation examples would help


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold commented on issue #2863: object_store: Support multi-part uploads to S3 for larger than 5GB object sizes

Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #2863:
URL: https://github.com/apache/arrow-rs/issues/2863#issuecomment-1277923994

   I think this is already supported by the put_multipart API?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] alamb commented on issue #2863: object_store: Support multi-part uploads to S3 for larger than 5GB object sizes

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #2863:
URL: https://github.com/apache/arrow-rs/issues/2863#issuecomment-1277926972

   https://docs.rs/object_store/0.5.1/object_store/trait.ObjectStore.html#tymethod.put_multipart
   
   definitely looks promising. cc @NGA-TRAN 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org