[jira] [Commented] (JCLOUDS-769) Upload blob from stream

["Andrew Gaul (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/JCLOUDS-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377758#comment-14377758 ] 

Andrew Gaul commented on JCLOUDS-769:
-------------------------------------

This is tricky to implement since jclouds needs to satisfy Amazon's requirement for a Content-length per part.  Depending on the chosen part size, this can require 5 MB to 5 GB of memory per part and per simultaneous upload or spooling the stream to a temporary file on-disk, as s3distcp does.  Alternatively, we could expose the uploadPart method to the users who can do their own buffering and size computation.

> Upload blob from stream
> -----------------------
>
>                 Key: JCLOUDS-769
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-769
>             Project: jclouds
>          Issue Type: New Feature
>          Components: jclouds-blobstore
>    Affects Versions: 1.8.1
>            Reporter: Akos Hajnal
>              Labels: multipart, s3
>
> Dear Developers,
> It was not easy, but using S3 API, it was possible to upload a large blob from stream - without knowing its size in advance (and storing all the data locally). I found solutions using jclouds' aws-s3 specific API (some async interface), but I really miss this feature from jclouds' general API.
> My dream is to have a method like:
> blob.getOutputStream() into which I can write as many data as I want, 
> which pushes data to the storage simultaneously until I close the stream.
> (When I used S3, I created a wrapper class extending OutputStream, which initiates multipart upload, buffers data written to the output stream, writes a part when the buffer is full, and finalizes multipart upload on stream close.) 
> I don't know it is possible for all providers, but I really miss it...
> Thank you,
> Akos Hajnal



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)