You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by tc...@tutanota.com on 2020/10/21 23:57:18 UTC

S3 Multi-Part Uploads

We've been attempting to get writes to an S3 store working with the AWS2 SDK, but it  appears that the S3 file system was removed in the process of upgrading from AWS1.  Have not yet found a concrete example of how to work with S3, so we have a few questions:

* Is HadoopFileSystem the preferred means of writing to S3?
* Are multi-part uploads supported via s3a:// urls?
* Is it possible to window writes to an S3 file by amount of data as well as a maximum time limit?  

Appreciate the info.