[jira] [Comment Edited] (CAMEL-13433) S3: Exchange body stream is loaded into memory to calculate content length which is already set via headers

["Andrea Cosentino (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821012#comment-16821012 ] 

Andrea Cosentino edited comment on CAMEL-13433 at 4/18/19 12:04 PM:
--------------------------------------------------------------------

The problem is the overidding of the lenght or the input stream read? 

 

Because if it's the overidding we can do

 
{code:java}
is = exchange.getIn().getMandatoryBody(InputStream.class);
baos = determineLengthInputStream(is);
if (!(objectMetadata.getContentLength() > 0)) {
objectMetadata.setContentLength(baos.size());
is = new ByteArrayInputStream(baos.toByteArray());
} else {
is = new ByteArrayInputStream(baos.toByteArray());
}
{code}


was (Author: ancosen):
The problem is the overidding of the lenght or the input stream read? 

 

Because if it's the overidding we can do

 
{code:java}
is = exchange.getIn().getMandatoryBody(InputStream.class);
if (!(objectMetadata.getContentLength() > 0)) {
baos = determineLengthInputStream(is);
objectMetadata.setContentLength(baos.size());
is = new ByteArrayInputStream(baos.toByteArray());
} else {
baos = determineLengthInputStream(is);
is = new ByteArrayInputStream(baos.toByteArray());
}
{code}

> S3: Exchange body stream is loaded into memory to calculate content length which is already set via headers
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-13433
>                 URL: https://issues.apache.org/jira/browse/CAMEL-13433
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-aws
>    Affects Versions: 2.22.1, 2.23.2, 3.0.0-M1
>            Reporter: MykhailoVlakh
>            Assignee: Andrea Cosentino
>            Priority: Major
>
> My code uploads large files to S3 storage (500 MB and bigger). I create an exchange with properly set meta information headers including the _CamelAwsS3ContentLength_, as a body I provide a stream. Everything worked well in Camel 2.14.3 but when I upgraded to Camel ti version 2.22.1 I suddenly began getting OOM issues from time to time. I did some debugging and found that the issue is in *S3Producer* that loads content of the body stream into memory.
> {code:java}
> public void processSingleOp(final Exchange exchange) throws Exception {
> ObjectMetadata objectMetadata = determineMetadata(exchange);  <== here we already set length
> File filePayload = null;
> InputStream is = null;
> ByteArrayOutputStream baos = null;
> Object obj = exchange.getIn().getMandatoryBody();
> PutObjectRequest putObjectRequest = null;
> // Need to check if the message body is WrappedFile
> if (obj instanceof WrappedFile) {
>    obj = ((WrappedFile<?>)obj).getFile();
> }
> if (obj instanceof File) {
>    filePayload = (File)obj;
>    is = new FileInputStream(filePayload);
> } else {
>    is = exchange.getIn().getMandatoryBody(InputStream.class);
>    baos = determineLengthInputStream(is);        <====== the issue is here
>    objectMetadata.setContentLength(baos.size());
>    is = new ByteArrayInputStream(baos.toByteArray());
> }
> ...
> {code}
> This is a defect since, as I mentioned, I already provide content length via headers and in the first line of the method populates that information into object metadata.
> I think there should be some if check that skips content length size calculation if it is already known. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)