You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Lu...@spar-ics.com on 2021/05/12 09:34:09 UTC
streaming with azure-storage-blob component
Hi,
We are transferring lots of data to the azure-storage with the azure-storage-blob component (version 3.7.0)
The Route itself is only working with streams to keep the memory overhead low, streamcaching is enabled.
But looking at this stacktrace and the corresponding sourcecode it’s obvious that the whole stream is read to memory to check the total payload size (seems necessary for the azure client)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.commons.io.output.AbstractByteArrayOutputStream.toByteArrayImpl(AbstractByteArrayOutputStream.java:366)
at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:163)
at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2241)
at org.apache.camel.component.azure.storage.blob.BlobUtils.getInputStreamLength(BlobUtils.java:37)
at org.apache.camel.component.azure.storage.blob.BlobStreamAndLength.createBlobStreamAndLengthFromExchangeBody(BlobStreamAndLength.java:50)
at org.apache.camel.component.azure.storage.blob.operations.BlobOperations.uploadBlockBlob(BlobOperations.java:181)
at org.apache.camel.component.azure.storage.blob.BlobProducer.process(BlobProducer.java:86)
at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
at org.apache.camel.processor.SendDynamicProcessor.lambda$process$0(SendDynamicProcessor.java:195)
I was wondering if there is a better way to do this. Maybe a shortcut for the cached stream that just checks the size of the cache?
regards,
Lukas
[cid:image33b7ae.PNG@730fbf7f.428aa6b0]<https://www.spar-ics.com/>
Dipl. Ing. (FH) Lukas Angerer
Senior Software Engineer
Business Automation & Integration
<http://>
Phone: +43 662 4470 84937
Mobile: +43 664 8159143
E-Mail: Lukas.Angerer@spar-ics.com<ma...@spar-ics.com>
SPAR Business Services GmbH
Information & Communication Services
Europastrasse 3, 5015 Salzburg, Austria
<http://>
[cid:imageefe13d.PNG@d6aeff9c.4dac722f]
Sollten Sie diese E-Mail unbeabsichtigt bzw. irrtümlich erhalten haben, so weisen wir Sie darauf hin, dass gemäß § 93 Abs 4 TKG der Inhalt sowie die Tatsache des Empfangs dieser E-Mail weder aufgezeichnet noch verwertet oder Unbefugten mitgeteilt werden dürfen. Wir ersuchen Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen.
If you have received this email accidentally or in error, we point out that, in accordance with § 93 para. 4 TKG (Telecommunications Act), the contents of this email and the fact of its receipt must not be recorded, exploited or communicated to unauthorized persons. We ask you to delete the message from your system and to contact us.
Re: streaming with azure-storage-blob component
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
What are the sources of those streams?
I wonder if we could enrich from the message some sort of total size header
into the camel blob producer, so it can tell the blob client the expected
length, so it does not read the stream itself to find out.
Also if you have the opportunity you are welcome to test with latest Camel
3.9.0 release, if its still a problem.
And you are welcome to create a JIRA as it would be great to have streaming
work well with azure, especially for blob as its supposed to be also big
blobs of data ;)
On Wed, May 12, 2021 at 11:34 AM <Lu...@spar-ics.com> wrote:
> Hi,
>
>
>
> We are transferring lots of data to the azure-storage with the
> azure-storage-blob component (version 3.7.0)
>
> The Route itself is only working with streams to keep the memory overhead
> low, streamcaching is enabled.
>
>
>
> But looking at this stacktrace and the corresponding sourcecode it’s
> obvious that the whole stream is read to memory to check the total payload
> size (seems necessary for the azure client)
>
>
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
>
> at
> org.apache.commons.io.output.AbstractByteArrayOutputStream.toByteArrayImpl(AbstractByteArrayOutputStream.java:366)
>
> at
> org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:163)
>
> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2241)
>
> at
> org.apache.camel.component.azure.storage.blob.BlobUtils.getInputStreamLength(BlobUtils.java:37)
>
> at
> org.apache.camel.component.azure.storage.blob.BlobStreamAndLength.createBlobStreamAndLengthFromExchangeBody(BlobStreamAndLength.java:50)
>
> at
> org.apache.camel.component.azure.storage.blob.operations.BlobOperations.uploadBlockBlob(BlobOperations.java:181)
>
> at
> org.apache.camel.component.azure.storage.blob.BlobProducer.process(BlobProducer.java:86)
>
> at
> org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
>
> at
> org.apache.camel.processor.SendDynamicProcessor.lambda$process$0(SendDynamicProcessor.java:195)
>
>
>
>
>
> I was wondering if there is a better way to do this. Maybe a shortcut for
> the cached stream that just checks the size of the cache?
>
>
>
> regards,
>
> Lukas
> <https://www.spar-ics.com/>
>
> Dipl. Ing. (FH) Lukas Angerer
> Senior Software Engineer
> Business Automation & Integration
> *Phone:* +43 662 4470 84937
> *Mobile:* +43 664 8159143
> *E-Mail:* Lukas.Angerer@spar-ics.com
>
> *SPAR Business Services GmbH *Information & Communication Services
> Europastrasse 3, 5015 Salzburg, Austria
>
>
> Sollten Sie diese E-Mail unbeabsichtigt bzw. irrtümlich erhalten haben, so
> weisen wir Sie darauf hin, dass gemäß § 93 Abs 4 TKG der Inhalt sowie die
> Tatsache des Empfangs dieser E-Mail weder aufgezeichnet noch verwertet oder
> Unbefugten mitgeteilt werden dürfen. Wir ersuchen Sie, die Nachricht von
> Ihrem System zu löschen und sich mit uns in Verbindung zu setzen.
>
> If you have received this email accidentally or in error, we point out
> that, in accordance with § 93 para. 4 TKG (Telecommunications Act), the
> contents of this email and the fact of its receipt must not be recorded,
> exploited or communicated to unauthorized persons. We ask you to delete the
> message from your system and to contact us.
>
>
>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2