You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Artur <aj...@ravenpack.com> on 2017/03/29 16:38:17 UTC
S3 MultiPart upload for InputStream
Hello,
i am trying to use the AWS-S3 component to multipart upload to S3.
The content is coming via network and I was trying to pass an InputStream to
the component for uploading.
This fails with this message:
aws-s3: MultiPart upload requires a File input. (Service: null; Status Code:
0; Error Code: null; Request ID: null)]
I looked at the source code and it seems that when using the component
indeed only File can be passed to it.
The question is.. why.
Imagine there's a very very very large file that your app can get over
network and you want to upload it to s3. It makes sense to do mlutipart not
to load the whole file in memory, so the component could read the bytes in
chunks and pass it to s3 multipart api. You don't have an option of storing
that file to disk first (which doesn't make sense in that case anyway).
Is this scenario not supported for any particular reason?
Thanks Artur
--
View this message in context: http://camel.465427.n5.nabble.com/S3-MultiPart-upload-for-InputStream-tp5796702.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: S3 MultiPart upload for InputStream
Posted by Artur <aj...@ravenpack.com>.
Hey,
Thanks for the answer.
I see. I understand that the file will not be loaded to memory, but there's
assumption that the file is present on the filesystem local to the process
where Camel runs which is not the case in my scenario.
I will happily contribute the patch. Thank you for all the info and links.
Super helpful.
Best,
Artur
On 29 Mar 2017 6:57 p.m., "Zoran Regvart-2 [via Camel]" <
ml-node+s465427n5796703h63@n5.nabble.com> wrote:
> Hi Artur,
> I don't think that the implementation will load the whole file in
> memory, looking at CAMEL-7991[1] that introduced multipart support the
> reasoning behind it was to reduce the memory usage.
> That being said if you have the need for passing InputStream instead,
> could you consider contributing support for that, see Contributing at
> https://camel.apache.org/contributing.html
>
> thanks :)
>
> zoran
>
>
> [1] https://issues.apache.org/jira/browse/CAMEL-7991
>
> On Wed, Mar 29, 2017 at 6:38 PM, Artur <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5796703&i=0>> wrote:
>
> > Hello,
> >
> > i am trying to use the AWS-S3 component to multipart upload to S3.
> >
> > The content is coming via network and I was trying to pass an
> InputStream to
> > the component for uploading.
> >
> > This fails with this message:
> >
> > aws-s3: MultiPart upload requires a File input. (Service: null; Status
> Code:
> > 0; Error Code: null; Request ID: null)]
> >
> > I looked at the source code and it seems that when using the component
> > indeed only File can be passed to it.
> > The question is.. why.
> >
> > Imagine there's a very very very large file that your app can get over
> > network and you want to upload it to s3. It makes sense to do mlutipart
> not
> > to load the whole file in memory, so the component could read the bytes
> in
> > chunks and pass it to s3 multipart api. You don't have an option of
> storing
> > that file to disk first (which doesn't make sense in that case anyway).
> >
> > Is this scenario not supported for any particular reason?
> >
> > Thanks Artur
> >
> >
> >
> > --
> > View this message in context: http://camel.465427.n5.nabble.
> com/S3-MultiPart-upload-for-InputStream-tp5796702.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Zoran Regvart
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/S3-MultiPart-upload-for-
> InputStream-tp5796702p5796703.html
> To unsubscribe from S3 MultiPart upload for InputStream, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5796702&code=YWphYmxvbnNraUByYXZlbnBhY2suY29tfDU3OTY3MDJ8LTYzMjA3NDY3Mg==>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/S3-MultiPart-upload-for-InputStream-tp5796702p5796709.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: S3 MultiPart upload for InputStream
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Artur,
I don't think that the implementation will load the whole file in
memory, looking at CAMEL-7991[1] that introduced multipart support the
reasoning behind it was to reduce the memory usage.
That being said if you have the need for passing InputStream instead,
could you consider contributing support for that, see Contributing at
https://camel.apache.org/contributing.html
thanks :)
zoran
[1] https://issues.apache.org/jira/browse/CAMEL-7991
On Wed, Mar 29, 2017 at 6:38 PM, Artur <aj...@ravenpack.com> wrote:
> Hello,
>
> i am trying to use the AWS-S3 component to multipart upload to S3.
>
> The content is coming via network and I was trying to pass an InputStream to
> the component for uploading.
>
> This fails with this message:
>
> aws-s3: MultiPart upload requires a File input. (Service: null; Status Code:
> 0; Error Code: null; Request ID: null)]
>
> I looked at the source code and it seems that when using the component
> indeed only File can be passed to it.
> The question is.. why.
>
> Imagine there's a very very very large file that your app can get over
> network and you want to upload it to s3. It makes sense to do mlutipart not
> to load the whole file in memory, so the component could read the bytes in
> chunks and pass it to s3 multipart api. You don't have an option of storing
> that file to disk first (which doesn't make sense in that case anyway).
>
> Is this scenario not supported for any particular reason?
>
> Thanks Artur
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/S3-MultiPart-upload-for-InputStream-tp5796702.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Zoran Regvart