You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Richa <rs...@gmail.com> on 2013/08/01 08:27:24 UTC
Camel Unmarshal throwing heap space exception
Hi,
I have a camel route where I am dealing with huge CSV files which comes in
gzip format.
The route looks as follows:
from(inputLocation)
.unmarshal().gzip()
.unmarshal().csv()
.process(someProcessor)
.to(outputLocation)
This route works fine with small files but when I use huge files, it throws
heap space exception. Can you please tell me a way to avoid this?
I cannot chunk my file before unmarshalling as it will affect my processing
in the processor.
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Unmarshal-throwing-heap-space-exception-tp5736621.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Unmarshal throwing heap space exception
Posted by Christian Müller <ch...@gmail.com>.
Which memory limits do you use? Can you increase it?
Occurs the exception in the gzip or the csv unmarshaller?
Best,
Christian
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Thu, Aug 1, 2013 at 8:27 AM, Richa <rs...@gmail.com> wrote:
> Hi,
> I have a camel route where I am dealing with huge CSV files which comes in
> gzip format.
> The route looks as follows:
>
> from(inputLocation)
> .unmarshal().gzip()
> .unmarshal().csv()
> .process(someProcessor)
> .to(outputLocation)
>
> This route works fine with small files but when I use huge files, it throws
> heap space exception. Can you please tell me a way to avoid this?
> I cannot chunk my file before unmarshalling as it will affect my processing
> in the processor.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Camel-Unmarshal-throwing-heap-space-exception-tp5736621.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Camel Unmarshal throwing heap space exception
Posted by ywang9009 <yw...@gmail.com>.
Claus, I am using gzip data format:
<dataFormats>
<gzip id="gzip"/>
</dataFormats>
I just want to unzip the file and another process would read the file with
streaming. I wasn't sure how "ZipSplitter" could help in my case. Would you
be able to give me some example, ideally in blueprint DSL? I ended up
writing some java code that spawns another process that runs some system
command to unzip the file.
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Unmarshal-throwing-heap-space-exception-tp5736621p5771640.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Unmarshal throwing heap space exception
Posted by Claus Ibsen <cl...@gmail.com>.
Maybe try the zip-file data format when working with files
http://camel.apache.org/zip-file-dataformat.html
On Thu, Sep 17, 2015 at 5:30 PM, ywang9009 <yw...@gmail.com> wrote:
> Claus, could you elaborate on the splitter/streaming approach? I am running
> into the same issue - gzip unmarshaling is done in memory and I am getting
> this error when processing a large file:
>
> java.lang.OutOfMemoryError: Java heap space
>
> at java.util.Arrays.copyOf(Arrays.java:2271)[:1.7.0_45]
>
> at
> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)[:1.7.0_45]
>
> at
> org.apache.camel.impl.GzipDataFormat.unmarshal(GzipDataFormat.java:52)[142:org.apache.camel.camel-core:2.12.0.redhat-610379]
>
> Here is my blueprint:
>
>
> <camel:from
> uri="file://{{cmi.home}}/{{cmi.myworld.data.file.path}}?doneFileName=${file:name}.done"
> />
>
>
> <camel:choice>
>
> <camel:when>
> <camel:simple>${file:name} regex ".*.gz$" </camel:simple>
>
> <camel:unmarshal ref="gzip"/>
>
> <camel:to
>
> uri="file://{{cmi.home}}/{{cmi.myworld.file.path}}?fileName=${file:name.noext}&doneFileName=${file:name}.done"
> />
>
> <camel:log message="********processGzip:gzip file(${file:name})********" />
>
> </camel:when>
> <camel:otherwise>
>
> <camel:log message="********processGzip:regular file(${file:name})********"
> />
>
> <camel:to
>
> uri="file://{{cmi.home}}/{{cmi.myworld.file.path}}?doneFileName=${file:name}.done"
> />
>
> </camel:otherwise>
>
> </camel:choice>
>
>
> </camel:route>
>
> I am using jboss-fuse-6.1.0.redhat-379
>
>
>
> Thank you
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-Unmarshal-throwing-heap-space-exception-tp5736621p5771602.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2nd edition:
https://www.manning.com/books/camel-in-action-second-edition
Re: Camel Unmarshal throwing heap space exception
Posted by ywang9009 <yw...@gmail.com>.
Claus, could you elaborate on the splitter/streaming approach? I am running
into the same issue - gzip unmarshaling is done in memory and I am getting
this error when processing a large file:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)[:1.7.0_45]
at
java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)[:1.7.0_45]
at
org.apache.camel.impl.GzipDataFormat.unmarshal(GzipDataFormat.java:52)[142:org.apache.camel.camel-core:2.12.0.redhat-610379]
Here is my blueprint:
<camel:from
uri="file://{{cmi.home}}/{{cmi.myworld.data.file.path}}?doneFileName=${file:name}.done"
/>
<camel:choice>
<camel:when>
<camel:simple>${file:name} regex ".*.gz$" </camel:simple>
<camel:unmarshal ref="gzip"/>
<camel:to
uri="file://{{cmi.home}}/{{cmi.myworld.file.path}}?fileName=${file:name.noext}&doneFileName=${file:name}.done"
/>
<camel:log message="********processGzip:gzip file(${file:name})********" />
</camel:when>
<camel:otherwise>
<camel:log message="********processGzip:regular file(${file:name})********"
/>
<camel:to
uri="file://{{cmi.home}}/{{cmi.myworld.file.path}}?doneFileName=${file:name}.done"
/>
</camel:otherwise>
</camel:choice>
</camel:route>
I am using jboss-fuse-6.1.0.redhat-379
Thank you
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Unmarshal-throwing-heap-space-exception-tp5736621p5771602.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Unmarshal throwing heap space exception
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Take a look at the splitter EIP pattern and use the streaming mode.
Also I think gzip is in memory based, you may want some way to unzip
to another file directly.
On Thu, Aug 1, 2013 at 8:27 AM, Richa <rs...@gmail.com> wrote:
> Hi,
> I have a camel route where I am dealing with huge CSV files which comes in
> gzip format.
> The route looks as follows:
>
> from(inputLocation)
> .unmarshal().gzip()
> .unmarshal().csv()
> .process(someProcessor)
> .to(outputLocation)
>
> This route works fine with small files but when I use huge files, it throws
> heap space exception. Can you please tell me a way to avoid this?
> I cannot chunk my file before unmarshalling as it will affect my processing
> in the processor.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-Unmarshal-throwing-heap-space-exception-tp5736621.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen