You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Achim Nierbeck <bc...@googlemail.com> on 2012/10/25 16:51:10 UTC
FTP download of bigger files gives me OOM
Hi,
using Camel 2.10.2
I have a ftp endpoint that downloads a couple of files from a remote
FTP server.
There are different files located on this server, some of them are
about 700 MB size.
Now every time I start the route for downloading it gives me a OOM
when trying to download this file.
The route is configured to use the *localWorkDirectory*.
Roughly the route does something like the following:
from("ftp://server?noop=true&localWorkDirectory=/myTemp&sendEmptyMessageWhenIdle=true")
.onException()
... ommited the exception part ....
.onCompletion()
... omitted the onCompletion part ....
.choice()
.when(body().isNull())
.... stop the whole thing here ...
.otherwise()
.to("file:/dropItHere")
.setBody(constant(""))
.to("seda:doSomeLogging");
I tried to increase the memory up to 3GB for the Heap size with no luck.
Following Stacktrace is logged:
Camel (rootContext) thread #28 - ftp://usrXXX@ServerXXXX/
at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)
at java.util.Arrays.copyOf([BI)[B (Arrays.java:2786)
at java.io.ByteArrayOutputStream.write([BII)V (ByteArrayOutputStream.java:94)
at org.apache.commons.net.io.Util.copyStream(Ljava/io/InputStream;Ljava/io/OutputStream;IJLorg/apache/commons/net/io/CopyStreamListener;Z)J
(Util.java:123)
at org.apache.commons.net.ftp.FTPClient._retrieveFile(Ljava/lang/String;Ljava/lang/String;Ljava/io/OutputStream;)Z
(FTPClient.java:1695)
at org.apache.commons.net.ftp.FTPClient.retrieveFile(Ljava/lang/String;Ljava/io/OutputStream;)Z
(FTPClient.java:1669)
at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
(FtpOperations.java:328)
at org.apache.camel.component.file.remote.FtpOperations.retrieveFile(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
(FtpOperations.java:297)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
(GenericFileConsumer.java:317)
at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
(RemoteFileConsumer.java:94)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(Ljava/util/Queue;)I
(GenericFileConsumer.java:189)
at org.apache.camel.component.file.GenericFileConsumer.poll()I
(GenericFileConsumer.java:155)
at org.apache.camel.impl.ScheduledPollConsumer.doRun()V
(ScheduledPollConsumer.java:139)
at org.apache.camel.impl.ScheduledPollConsumer.run()V
(ScheduledPollConsumer.java:91)
The attached OOM-Camel.PNG shows what the Memory Analysis tells me.
Am I missing something critical here? Or do I have to tune up the
memory for those kind
of cases?
Thanks and regards, Achim
--
Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
Committer & Project Lead
OPS4J Pax for Vaadin
<http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
Lead
blog <http://notizblog.nierbeck.de/>
Re: FTP download of bigger files gives me OOM
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
A bit odd as the output stream passed in to the ftp client is a
FileOutputStream.
I suggest if you can check the commons-net code and see what it does.
There was a bug though if you afterwards route to a file endpoint.
Then Camel was optimized to try first to rename the file from
localWorkDir, than doing a file copy.
This bug has been fixed and will be in next releases.
But this is not your problem, as its when you download the file, it
goes into memory. Which it ought to stream
to the file straight. Maybe with a little in memory buffer (but that
would be commons-net specific).
On Thu, Oct 25, 2012 at 4:51 PM, Achim Nierbeck <bc...@googlemail.com> wrote:
> Hi,
>
> using Camel 2.10.2
> I have a ftp endpoint that downloads a couple of files from a remote
> FTP server.
> There are different files located on this server, some of them are
> about 700 MB size.
> Now every time I start the route for downloading it gives me a OOM
> when trying to download this file.
> The route is configured to use the *localWorkDirectory*.
> Roughly the route does something like the following:
>
> from("ftp://server?noop=true&localWorkDirectory=/myTemp&sendEmptyMessageWhenIdle=true")
> .onException()
> ... ommited the exception part ....
> .onCompletion()
> ... omitted the onCompletion part ....
> .choice()
> .when(body().isNull())
> .... stop the whole thing here ...
> .otherwise()
> .to("file:/dropItHere")
> .setBody(constant(""))
> .to("seda:doSomeLogging");
>
> I tried to increase the memory up to 3GB for the Heap size with no luck.
>
> Following Stacktrace is logged:
> Camel (rootContext) thread #28 - ftp://usrXXX@ServerXXXX/
> at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)
> at java.util.Arrays.copyOf([BI)[B (Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write([BII)V (ByteArrayOutputStream.java:94)
> at org.apache.commons.net.io.Util.copyStream(Ljava/io/InputStream;Ljava/io/OutputStream;IJLorg/apache/commons/net/io/CopyStreamListener;Z)J
> (Util.java:123)
> at org.apache.commons.net.ftp.FTPClient._retrieveFile(Ljava/lang/String;Ljava/lang/String;Ljava/io/OutputStream;)Z
> (FTPClient.java:1695)
> at org.apache.commons.net.ftp.FTPClient.retrieveFile(Ljava/lang/String;Ljava/io/OutputStream;)Z
> (FTPClient.java:1669)
> at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
> (FtpOperations.java:328)
> at org.apache.camel.component.file.remote.FtpOperations.retrieveFile(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
> (FtpOperations.java:297)
> at org.apache.camel.component.file.GenericFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
> (GenericFileConsumer.java:317)
> at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
> (RemoteFileConsumer.java:94)
> at org.apache.camel.component.file.GenericFileConsumer.processBatch(Ljava/util/Queue;)I
> (GenericFileConsumer.java:189)
> at org.apache.camel.component.file.GenericFileConsumer.poll()I
> (GenericFileConsumer.java:155)
> at org.apache.camel.impl.ScheduledPollConsumer.doRun()V
> (ScheduledPollConsumer.java:139)
> at org.apache.camel.impl.ScheduledPollConsumer.run()V
> (ScheduledPollConsumer.java:91)
>
>
> The attached OOM-Camel.PNG shows what the Memory Analysis tells me.
>
> Am I missing something critical here? Or do I have to tune up the
> memory for those kind
> of cases?
>
>
> Thanks and regards, Achim
> --
>
> Apache Karaf <http://karaf.apache.org/> Committer & PMC
> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
> Committer & Project Lead
> OPS4J Pax for Vaadin
> <http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
> Lead
> blog <http://notizblog.nierbeck.de/>
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: FTP download of bigger files gives me OOM
Posted by Christian Müller <ch...@gmail.com>.
Thanks for sharing this Achim. I was also not care about this behavior...
;-)
Best,
Christian
Sent from a mobile device
Am 26.10.2012 14:00 schrieb "Achim Nierbeck" <bc...@googlemail.com>:
> Hi,
>
> after taking a closer look on how a bundle update in Karaf does work
> and how Camel handles the camel contexts I found the reason
> for why my update of the bundle containing the route didn't work :-)
>
> Some details on this "issue"
> at first my route didn't contain the needed localWorkDirectory attribute.
> So I added this and updated my bundle in Karaf.
> As I thought the update of the bundle does a clean installation of it.
> But since the camel-context is held (better referenced) by the camel-core
> bundle this update just didn't help any on my impression of updating the
> camel-context/route.
> So instead of trashing the data folder and doing a restart a bundle:refresh
> on camel-core would have worked already ...
>
> It's very logic that it behaves this way but it's not intuitive :)
> Maybe this can be done differently in the future (have to think on how
> thought)
>
> regards, Achim
>
>
> 2012/10/25 Achim Nierbeck <bc...@googlemail.com>:
> > Hmm, interesting ...
> > need to check the reason for this later ...
> >
> > I kept updating my bundle containing this route, which failed, so now
> > after your suggestion
> > I did a "clean" run (should have done that earlier, shame on me :) )
> > and now it just consumes about 1.5 GB Heap :)
> >
> > wonder why it survived the update <bundle-id>
> >
> > thanks for giving pointing me that direction :)
> > I was so sure the update should/would work I didn't think of cleaning
> > the bundle cache directory.
> >
> > regards, Achim
> >
> >
> > 2012/10/25 Claus Ibsen <cl...@gmail.com>:
> >> Digged in the code.
> >>
> >> Somehow this returns false according to your stacktrace. So it goes
> >> for the retrieveFileToStreamInBody instead of the 1st method.
> >> Can you double check your endpoint is configured correctly.
> >>
> >> if (ObjectHelper.isNotEmpty(endpoint.getLocalWorkDirectory())) {
> >> // local work directory is configured so we should store
> >> file content as files in this local directory
> >> return retrieveFileToFileInLocalWorkDirectory(name,
> exchange);
> >> } else {
> >> // store file content directory as stream on the body
> >> return retrieveFileToStreamInBody(name, exchange);
> >> }
> >>
> >>
> >>
> >> On Thu, Oct 25, 2012 at 4:51 PM, Achim Nierbeck <
> bcanhome@googlemail.com> wrote:
> >>> Hi,
> >>>
> >>> using Camel 2.10.2
> >>> I have a ftp endpoint that downloads a couple of files from a remote
> >>> FTP server.
> >>> There are different files located on this server, some of them are
> >>> about 700 MB size.
> >>> Now every time I start the route for downloading it gives me a OOM
> >>> when trying to download this file.
> >>> The route is configured to use the *localWorkDirectory*.
> >>> Roughly the route does something like the following:
> >>>
> >>> from("
> ftp://server?noop=true&localWorkDirectory=/myTemp&sendEmptyMessageWhenIdle=true
> ")
> >>> .onException()
> >>> ... ommited the exception part ....
> >>> .onCompletion()
> >>> ... omitted the onCompletion part ....
> >>> .choice()
> >>> .when(body().isNull())
> >>> .... stop the whole thing here ...
> >>> .otherwise()
> >>> .to("file:/dropItHere")
> >>> .setBody(constant(""))
> >>> .to("seda:doSomeLogging");
> >>>
> >>> I tried to increase the memory up to 3GB for the Heap size with no
> luck.
> >>>
> >>> Following Stacktrace is logged:
> >>> Camel (rootContext) thread #28 - ftp://usrXXX@ServerXXXX/
> >>> at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)
> >>> at java.util.Arrays.copyOf([BI)[B (Arrays.java:2786)
> >>> at java.io.ByteArrayOutputStream.write([BII)V
> (ByteArrayOutputStream.java:94)
> >>> at
> org.apache.commons.net.io.Util.copyStream(Ljava/io/InputStream;Ljava/io/OutputStream;IJLorg/apache/commons/net/io/CopyStreamListener;Z)J
> >>> (Util.java:123)
> >>> at
> org.apache.commons.net.ftp.FTPClient._retrieveFile(Ljava/lang/String;Ljava/lang/String;Ljava/io/OutputStream;)Z
> >>> (FTPClient.java:1695)
> >>> at
> org.apache.commons.net.ftp.FTPClient.retrieveFile(Ljava/lang/String;Ljava/io/OutputStream;)Z
> >>> (FTPClient.java:1669)
> >>> at
> org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
> >>> (FtpOperations.java:328)
> >>> at
> org.apache.camel.component.file.remote.FtpOperations.retrieveFile(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
> >>> (FtpOperations.java:297)
> >>> at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
> >>> (GenericFileConsumer.java:317)
> >>> at
> org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
> >>> (RemoteFileConsumer.java:94)
> >>> at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(Ljava/util/Queue;)I
> >>> (GenericFileConsumer.java:189)
> >>> at org.apache.camel.component.file.GenericFileConsumer.poll()I
> >>> (GenericFileConsumer.java:155)
> >>> at org.apache.camel.impl.ScheduledPollConsumer.doRun()V
> >>> (ScheduledPollConsumer.java:139)
> >>> at org.apache.camel.impl.ScheduledPollConsumer.run()V
> >>> (ScheduledPollConsumer.java:91)
> >>>
> >>>
> >>> The attached OOM-Camel.PNG shows what the Memory Analysis tells me.
> >>>
> >>> Am I missing something critical here? Or do I have to tune up the
> >>> memory for those kind
> >>> of cases?
> >>>
> >>>
> >>> Thanks and regards, Achim
> >>> --
> >>>
> >>> Apache Karaf <http://karaf.apache.org/> Committer & PMC
> >>> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
> >>> Committer & Project Lead
> >>> OPS4J Pax for Vaadin
> >>> <http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
> >>> Lead
> >>> blog <http://notizblog.nierbeck.de/>
> >>
> >>
> >>
> >> --
> >> Claus Ibsen
> >> -----------------
> >> Red Hat, Inc.
> >> FuseSource is now part of Red Hat
> >> Email: cibsen@redhat.com
> >> Web: http://fusesource.com
> >> Twitter: davsclaus
> >> Blog: http://davsclaus.com
> >> Author of Camel in Action: http://www.manning.com/ibsen
> >
> >
> >
> > --
> >
> > Apache Karaf <http://karaf.apache.org/> Committer & PMC
> > OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
> > Committer & Project Lead
> > OPS4J Pax for Vaadin
> > <http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
> > Lead
> > blog <http://notizblog.nierbeck.de/>
>
>
>
> --
>
> Apache Karaf <http://karaf.apache.org/> Committer & PMC
> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
> Committer & Project Lead
> OPS4J Pax for Vaadin
> <http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
> Lead
> blog <http://notizblog.nierbeck.de/>
>
Re: FTP download of bigger files gives me OOM
Posted by Achim Nierbeck <bc...@googlemail.com>.
Hi,
after taking a closer look on how a bundle update in Karaf does work
and how Camel handles the camel contexts I found the reason
for why my update of the bundle containing the route didn't work :-)
Some details on this "issue"
at first my route didn't contain the needed localWorkDirectory attribute.
So I added this and updated my bundle in Karaf.
As I thought the update of the bundle does a clean installation of it.
But since the camel-context is held (better referenced) by the camel-core
bundle this update just didn't help any on my impression of updating the
camel-context/route.
So instead of trashing the data folder and doing a restart a bundle:refresh
on camel-core would have worked already ...
It's very logic that it behaves this way but it's not intuitive :)
Maybe this can be done differently in the future (have to think on how thought)
regards, Achim
2012/10/25 Achim Nierbeck <bc...@googlemail.com>:
> Hmm, interesting ...
> need to check the reason for this later ...
>
> I kept updating my bundle containing this route, which failed, so now
> after your suggestion
> I did a "clean" run (should have done that earlier, shame on me :) )
> and now it just consumes about 1.5 GB Heap :)
>
> wonder why it survived the update <bundle-id>
>
> thanks for giving pointing me that direction :)
> I was so sure the update should/would work I didn't think of cleaning
> the bundle cache directory.
>
> regards, Achim
>
>
> 2012/10/25 Claus Ibsen <cl...@gmail.com>:
>> Digged in the code.
>>
>> Somehow this returns false according to your stacktrace. So it goes
>> for the retrieveFileToStreamInBody instead of the 1st method.
>> Can you double check your endpoint is configured correctly.
>>
>> if (ObjectHelper.isNotEmpty(endpoint.getLocalWorkDirectory())) {
>> // local work directory is configured so we should store
>> file content as files in this local directory
>> return retrieveFileToFileInLocalWorkDirectory(name, exchange);
>> } else {
>> // store file content directory as stream on the body
>> return retrieveFileToStreamInBody(name, exchange);
>> }
>>
>>
>>
>> On Thu, Oct 25, 2012 at 4:51 PM, Achim Nierbeck <bc...@googlemail.com> wrote:
>>> Hi,
>>>
>>> using Camel 2.10.2
>>> I have a ftp endpoint that downloads a couple of files from a remote
>>> FTP server.
>>> There are different files located on this server, some of them are
>>> about 700 MB size.
>>> Now every time I start the route for downloading it gives me a OOM
>>> when trying to download this file.
>>> The route is configured to use the *localWorkDirectory*.
>>> Roughly the route does something like the following:
>>>
>>> from("ftp://server?noop=true&localWorkDirectory=/myTemp&sendEmptyMessageWhenIdle=true")
>>> .onException()
>>> ... ommited the exception part ....
>>> .onCompletion()
>>> ... omitted the onCompletion part ....
>>> .choice()
>>> .when(body().isNull())
>>> .... stop the whole thing here ...
>>> .otherwise()
>>> .to("file:/dropItHere")
>>> .setBody(constant(""))
>>> .to("seda:doSomeLogging");
>>>
>>> I tried to increase the memory up to 3GB for the Heap size with no luck.
>>>
>>> Following Stacktrace is logged:
>>> Camel (rootContext) thread #28 - ftp://usrXXX@ServerXXXX/
>>> at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)
>>> at java.util.Arrays.copyOf([BI)[B (Arrays.java:2786)
>>> at java.io.ByteArrayOutputStream.write([BII)V (ByteArrayOutputStream.java:94)
>>> at org.apache.commons.net.io.Util.copyStream(Ljava/io/InputStream;Ljava/io/OutputStream;IJLorg/apache/commons/net/io/CopyStreamListener;Z)J
>>> (Util.java:123)
>>> at org.apache.commons.net.ftp.FTPClient._retrieveFile(Ljava/lang/String;Ljava/lang/String;Ljava/io/OutputStream;)Z
>>> (FTPClient.java:1695)
>>> at org.apache.commons.net.ftp.FTPClient.retrieveFile(Ljava/lang/String;Ljava/io/OutputStream;)Z
>>> (FTPClient.java:1669)
>>> at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
>>> (FtpOperations.java:328)
>>> at org.apache.camel.component.file.remote.FtpOperations.retrieveFile(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
>>> (FtpOperations.java:297)
>>> at org.apache.camel.component.file.GenericFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
>>> (GenericFileConsumer.java:317)
>>> at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
>>> (RemoteFileConsumer.java:94)
>>> at org.apache.camel.component.file.GenericFileConsumer.processBatch(Ljava/util/Queue;)I
>>> (GenericFileConsumer.java:189)
>>> at org.apache.camel.component.file.GenericFileConsumer.poll()I
>>> (GenericFileConsumer.java:155)
>>> at org.apache.camel.impl.ScheduledPollConsumer.doRun()V
>>> (ScheduledPollConsumer.java:139)
>>> at org.apache.camel.impl.ScheduledPollConsumer.run()V
>>> (ScheduledPollConsumer.java:91)
>>>
>>>
>>> The attached OOM-Camel.PNG shows what the Memory Analysis tells me.
>>>
>>> Am I missing something critical here? Or do I have to tune up the
>>> memory for those kind
>>> of cases?
>>>
>>>
>>> Thanks and regards, Achim
>>> --
>>>
>>> Apache Karaf <http://karaf.apache.org/> Committer & PMC
>>> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
>>> Committer & Project Lead
>>> OPS4J Pax for Vaadin
>>> <http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
>>> Lead
>>> blog <http://notizblog.nierbeck.de/>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> FuseSource is now part of Red Hat
>> Email: cibsen@redhat.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>
>
>
> --
>
> Apache Karaf <http://karaf.apache.org/> Committer & PMC
> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
> Committer & Project Lead
> OPS4J Pax for Vaadin
> <http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
> Lead
> blog <http://notizblog.nierbeck.de/>
--
Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
Committer & Project Lead
OPS4J Pax for Vaadin
<http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
Lead
blog <http://notizblog.nierbeck.de/>
Re: FTP download of bigger files gives me OOM
Posted by Achim Nierbeck <bc...@googlemail.com>.
Hmm, interesting ...
need to check the reason for this later ...
I kept updating my bundle containing this route, which failed, so now
after your suggestion
I did a "clean" run (should have done that earlier, shame on me :) )
and now it just consumes about 1.5 GB Heap :)
wonder why it survived the update <bundle-id>
thanks for giving pointing me that direction :)
I was so sure the update should/would work I didn't think of cleaning
the bundle cache directory.
regards, Achim
2012/10/25 Claus Ibsen <cl...@gmail.com>:
> Digged in the code.
>
> Somehow this returns false according to your stacktrace. So it goes
> for the retrieveFileToStreamInBody instead of the 1st method.
> Can you double check your endpoint is configured correctly.
>
> if (ObjectHelper.isNotEmpty(endpoint.getLocalWorkDirectory())) {
> // local work directory is configured so we should store
> file content as files in this local directory
> return retrieveFileToFileInLocalWorkDirectory(name, exchange);
> } else {
> // store file content directory as stream on the body
> return retrieveFileToStreamInBody(name, exchange);
> }
>
>
>
> On Thu, Oct 25, 2012 at 4:51 PM, Achim Nierbeck <bc...@googlemail.com> wrote:
>> Hi,
>>
>> using Camel 2.10.2
>> I have a ftp endpoint that downloads a couple of files from a remote
>> FTP server.
>> There are different files located on this server, some of them are
>> about 700 MB size.
>> Now every time I start the route for downloading it gives me a OOM
>> when trying to download this file.
>> The route is configured to use the *localWorkDirectory*.
>> Roughly the route does something like the following:
>>
>> from("ftp://server?noop=true&localWorkDirectory=/myTemp&sendEmptyMessageWhenIdle=true")
>> .onException()
>> ... ommited the exception part ....
>> .onCompletion()
>> ... omitted the onCompletion part ....
>> .choice()
>> .when(body().isNull())
>> .... stop the whole thing here ...
>> .otherwise()
>> .to("file:/dropItHere")
>> .setBody(constant(""))
>> .to("seda:doSomeLogging");
>>
>> I tried to increase the memory up to 3GB for the Heap size with no luck.
>>
>> Following Stacktrace is logged:
>> Camel (rootContext) thread #28 - ftp://usrXXX@ServerXXXX/
>> at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)
>> at java.util.Arrays.copyOf([BI)[B (Arrays.java:2786)
>> at java.io.ByteArrayOutputStream.write([BII)V (ByteArrayOutputStream.java:94)
>> at org.apache.commons.net.io.Util.copyStream(Ljava/io/InputStream;Ljava/io/OutputStream;IJLorg/apache/commons/net/io/CopyStreamListener;Z)J
>> (Util.java:123)
>> at org.apache.commons.net.ftp.FTPClient._retrieveFile(Ljava/lang/String;Ljava/lang/String;Ljava/io/OutputStream;)Z
>> (FTPClient.java:1695)
>> at org.apache.commons.net.ftp.FTPClient.retrieveFile(Ljava/lang/String;Ljava/io/OutputStream;)Z
>> (FTPClient.java:1669)
>> at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
>> (FtpOperations.java:328)
>> at org.apache.camel.component.file.remote.FtpOperations.retrieveFile(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
>> (FtpOperations.java:297)
>> at org.apache.camel.component.file.GenericFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
>> (GenericFileConsumer.java:317)
>> at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
>> (RemoteFileConsumer.java:94)
>> at org.apache.camel.component.file.GenericFileConsumer.processBatch(Ljava/util/Queue;)I
>> (GenericFileConsumer.java:189)
>> at org.apache.camel.component.file.GenericFileConsumer.poll()I
>> (GenericFileConsumer.java:155)
>> at org.apache.camel.impl.ScheduledPollConsumer.doRun()V
>> (ScheduledPollConsumer.java:139)
>> at org.apache.camel.impl.ScheduledPollConsumer.run()V
>> (ScheduledPollConsumer.java:91)
>>
>>
>> The attached OOM-Camel.PNG shows what the Memory Analysis tells me.
>>
>> Am I missing something critical here? Or do I have to tune up the
>> memory for those kind
>> of cases?
>>
>>
>> Thanks and regards, Achim
>> --
>>
>> Apache Karaf <http://karaf.apache.org/> Committer & PMC
>> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
>> Committer & Project Lead
>> OPS4J Pax for Vaadin
>> <http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
>> Lead
>> blog <http://notizblog.nierbeck.de/>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: cibsen@redhat.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
--
Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
Committer & Project Lead
OPS4J Pax for Vaadin
<http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
Lead
blog <http://notizblog.nierbeck.de/>
Re: FTP download of bigger files gives me OOM
Posted by Claus Ibsen <cl...@gmail.com>.
Digged in the code.
Somehow this returns false according to your stacktrace. So it goes
for the retrieveFileToStreamInBody instead of the 1st method.
Can you double check your endpoint is configured correctly.
if (ObjectHelper.isNotEmpty(endpoint.getLocalWorkDirectory())) {
// local work directory is configured so we should store
file content as files in this local directory
return retrieveFileToFileInLocalWorkDirectory(name, exchange);
} else {
// store file content directory as stream on the body
return retrieveFileToStreamInBody(name, exchange);
}
On Thu, Oct 25, 2012 at 4:51 PM, Achim Nierbeck <bc...@googlemail.com> wrote:
> Hi,
>
> using Camel 2.10.2
> I have a ftp endpoint that downloads a couple of files from a remote
> FTP server.
> There are different files located on this server, some of them are
> about 700 MB size.
> Now every time I start the route for downloading it gives me a OOM
> when trying to download this file.
> The route is configured to use the *localWorkDirectory*.
> Roughly the route does something like the following:
>
> from("ftp://server?noop=true&localWorkDirectory=/myTemp&sendEmptyMessageWhenIdle=true")
> .onException()
> ... ommited the exception part ....
> .onCompletion()
> ... omitted the onCompletion part ....
> .choice()
> .when(body().isNull())
> .... stop the whole thing here ...
> .otherwise()
> .to("file:/dropItHere")
> .setBody(constant(""))
> .to("seda:doSomeLogging");
>
> I tried to increase the memory up to 3GB for the Heap size with no luck.
>
> Following Stacktrace is logged:
> Camel (rootContext) thread #28 - ftp://usrXXX@ServerXXXX/
> at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)
> at java.util.Arrays.copyOf([BI)[B (Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write([BII)V (ByteArrayOutputStream.java:94)
> at org.apache.commons.net.io.Util.copyStream(Ljava/io/InputStream;Ljava/io/OutputStream;IJLorg/apache/commons/net/io/CopyStreamListener;Z)J
> (Util.java:123)
> at org.apache.commons.net.ftp.FTPClient._retrieveFile(Ljava/lang/String;Ljava/lang/String;Ljava/io/OutputStream;)Z
> (FTPClient.java:1695)
> at org.apache.commons.net.ftp.FTPClient.retrieveFile(Ljava/lang/String;Ljava/io/OutputStream;)Z
> (FTPClient.java:1669)
> at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
> (FtpOperations.java:328)
> at org.apache.camel.component.file.remote.FtpOperations.retrieveFile(Ljava/lang/String;Lorg/apache/camel/Exchange;)Z
> (FtpOperations.java:297)
> at org.apache.camel.component.file.GenericFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
> (GenericFileConsumer.java:317)
> at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(Lorg/apache/camel/Exchange;)V
> (RemoteFileConsumer.java:94)
> at org.apache.camel.component.file.GenericFileConsumer.processBatch(Ljava/util/Queue;)I
> (GenericFileConsumer.java:189)
> at org.apache.camel.component.file.GenericFileConsumer.poll()I
> (GenericFileConsumer.java:155)
> at org.apache.camel.impl.ScheduledPollConsumer.doRun()V
> (ScheduledPollConsumer.java:139)
> at org.apache.camel.impl.ScheduledPollConsumer.run()V
> (ScheduledPollConsumer.java:91)
>
>
> The attached OOM-Camel.PNG shows what the Memory Analysis tells me.
>
> Am I missing something critical here? Or do I have to tune up the
> memory for those kind
> of cases?
>
>
> Thanks and regards, Achim
> --
>
> Apache Karaf <http://karaf.apache.org/> Committer & PMC
> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
> Committer & Project Lead
> OPS4J Pax for Vaadin
> <http://team.ops4j.org/wiki/display/PAXVAADIN/Home> Commiter & Project
> Lead
> blog <http://notizblog.nierbeck.de/>
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen