You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Anderson Vaz (Jira)" <ji...@apache.org> on 2020/04/18 17:57:00 UTC
[jira] [Comment Edited] (CAMEL-14929) camel-aws2-s3 - Doesn't
support stream download of large files.
[ https://issues.apache.org/jira/browse/CAMEL-14929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086561#comment-17086561 ]
Anderson Vaz edited comment on CAMEL-14929 at 4/18/20, 5:56 PM:
----------------------------------------------------------------
Yes, this is exactly what I would expect from such component. I made a monkey patch on my environment and it worked well when I had to stream get the object from S3 and save it to a local file. Essentially is only 2 lines that needs to be changed. Lines 177 and 201.
was (Author: mranderson):
Yes, this is exactly what I would expect from such component. I made a monkey patch on my environment and it worked well when I had to stream get the object from S3 and save it to a local file.
> camel-aws2-s3 - Doesn't support stream download of large files.
> ----------------------------------------------------------------
>
> Key: CAMEL-14929
> URL: https://issues.apache.org/jira/browse/CAMEL-14929
> Project: Camel
> Issue Type: Improvement
> Components: camel-aws2
> Affects Versions: 3.2.0
> Reporter: Anderson Vaz
> Assignee: Andrea Cosentino
> Priority: Minor
>
> Hi,
> The component `*camel-aws2-s3*` should be able to support streaming consume/download to allow the copy/download of large files from S3. The current implementation or 'saves' the contents of input stream into the memory or completely disregard it not giving a change for the next components to manipulate the stream. This seems to be a no ideal implementation.
> The issue essentially is on class `*org.apache.camel.component.aws2.s3.AWS2S3Endpoint*` in between lines *169* to *178* and lines *201* to *212*.
> The logic on lines 169 to 178 there is:
> * if the parameter `*includeBody*` is *true* it will consume the S3 stream into the memory which is no ideal for large files.
> * if the parameter `*includeBody*` is *false* it won't consume the S3 stream however the S3 stream will be lost, I couldn't find any other way to access it therefore the S3 is open for nothing on this case. This doesn't seem reasonable as well. I think the S3 stream should be put in the `*body*` raw so the next component in the pipeline can consume it.
> The logic on lines 201 to 212 is:
> * if the parameter `*includeBody*` is *false* it surprisingly close the S3 input stream confirming that there will be no way to consume it afterwards.
> * if the parameter `*includeBody*` is *true* the S3 input stream will be left open however there is way to access it as it is created on line 77 of `*org.apache.camel.component.aws2.s3.AWS2S3Consumer*` and afterwards if not included in the body it get lost.
> The ideal behaviour I think would be:
> * if `*includedBody*` is *true* then consume S3 input stream into the memory, save it in the body and close it.
> * if `*includeBody*` is *false* then put the raw S3 input stream in the body and don't close it.
> * if `*autoCloseBody*` is *true* then schedule the S3 input stream closing for when exchange is finished.
> * if `*autoCloseBody*` is *false* then leave to caller to close it which I'm not sure how this can be done in the current implementation.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)