You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by skasish <as...@hotmail.com> on 2014/02/21 09:13:39 UTC

out of heap space if remote FTP site has too many files to pick up

out of heap space if remote FTP site has too many files to pick up.

Is there configuration problem.

My simple code is given below

import org.slf4j.*;
import org.apache.camel.*;
import org.apache.camel.builder.*;
import org.apache.camel.main.Main;
import org.apache.camel.component.dataset.*;

public class FileDemoCamel extends Main {
    static Logger LOG = LoggerFactory.getLogger(FileDemoCamel.class);
    public static void main(String[] args) throws Exception {
        FileDemoCamel main = new FileDemoCamel();
        main.enableHangupSupport();
        main.addRouteBuilder(createRouteBuilder());
        main.bind("sampleGenerator", createDataSet());
        main.run(args);
    }
    static RouteBuilder createRouteBuilder() {
        return new RouteBuilder() {
            public void configure() {
               
from("ftp://admin@localhost:2121/MF?noop=true&password=admin&passiveMode=true&maxMessagesPerPoll=10000&doneFileName=${file:name.noext}.TRG")
                .to("file://D:/temp");
            }
        };
    }
    static DataSet createDataSet() {
        return new SimpleDataSet();
    }
}



--
View this message in context: http://camel.465427.n5.nabble.com/out-of-heap-space-if-remote-FTP-site-has-too-many-files-to-pick-up-tp5747716.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: out of heap space if remote FTP site has too many files to pick up

Posted by Claus Ibsen <cl...@gmail.com>.
You can limit the number of files to pickup in the poll,
http://camel.apache.org/file2

See the maxMessagesPerPoll option

On Fri, Feb 21, 2014 at 9:13 AM, skasish <as...@hotmail.com> wrote:
> out of heap space if remote FTP site has too many files to pick up.
>
> Is there configuration problem.
>
> My simple code is given below
>
> import org.slf4j.*;
> import org.apache.camel.*;
> import org.apache.camel.builder.*;
> import org.apache.camel.main.Main;
> import org.apache.camel.component.dataset.*;
>
> public class FileDemoCamel extends Main {
>     static Logger LOG = LoggerFactory.getLogger(FileDemoCamel.class);
>     public static void main(String[] args) throws Exception {
>         FileDemoCamel main = new FileDemoCamel();
>         main.enableHangupSupport();
>         main.addRouteBuilder(createRouteBuilder());
>         main.bind("sampleGenerator", createDataSet());
>         main.run(args);
>     }
>     static RouteBuilder createRouteBuilder() {
>         return new RouteBuilder() {
>             public void configure() {
>
> from("ftp://admin@localhost:2121/MF?noop=true&password=admin&passiveMode=true&maxMessagesPerPoll=10000&doneFileName=${file:name.noext}.TRG")
>                 .to("file://D:/temp");
>             }
>         };
>     }
>     static DataSet createDataSet() {
>         return new SimpleDataSet();
>     }
> }
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/out-of-heap-space-if-remote-FTP-site-has-too-many-files-to-pick-up-tp5747716.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io

Re: out of heap space if remote FTP site has too many files to pick up

Posted by skasish <as...@hotmail.com>.
Following error has been occurred

13:39:40.609 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF]
DEBUG o.a.c.c.file.GenericFileProducer - Wrote
[D:\temp\GT_aplist_ghiiwb02_apache_20140111.lst] to
[Endpoint[file://D:/temp]]
13:39:40.609 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF]
DEBUG o.a.c.c.file.GenericFileOnCompletion - Done processing file:
RemoteFile[GT_aplist_ghiiwb02_apache_20140111.lst] using exchange:
Exchange[GT_aplist_ghiiwb02_apache_20140111.lst]
13:39:40.781 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF]
WARN  o.a.c.c.file.remote.FtpConsumer - Consumer
FtpConsumer[ftp://admin@localhost:2121/MF?doneFileName=%24%7Bfile%3Aname.noext%7D.TRG&maxMessagesPerPoll=10000&noop=true&passiveMode=true&password=xxxxxx]
failed polling endpoint:
Endpoint[ftp://admin@localhost:2121/MF?doneFileName=%24%7Bfile%3Aname.noext%7D.TRG&maxMessagesPerPoll=10000&noop=true&passiveMode=true&password=xxxxxx].
Will try again at next poll. Caused by: [java.lang.OutOfMemoryError - Java
heap space]
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2786) ~[na:1.6.0_13]
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
~[na:1.6.0_13]
	at org.apache.commons.net.io.Util.copyStream(Util.java:119)
~[commons-net-2.0.jar:2.0]
	at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1286)
~[commons-net-2.0.jar:2.0]
	at
org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(FtpOperations.java:353)
~[camel-ftp-2.12.1.jar:2.12.1]
	at
org.apache.camel.component.file.remote.FtpOperations.retrieveFile(FtpOperations.java:301)
~[camel-ftp-2.12.1.jar:2.12.1]
	at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:362)
~[camel-core-2.12.1.jar:2.12.1]
	at
org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:99)
~[camel-ftp-2.12.1.jar:2.12.1]
	at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:201)
~[camel-core-2.12.1.jar:2.12.1]
	at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:165)
~[camel-core-2.12.1.jar:2.12.1]
	at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
[camel-core-2.12.1.jar:2.12.1]
	at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
[camel-core-2.12.1.jar:2.12.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[na:1.6.0_13]
	at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
[na:1.6.0_13]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
[na:1.6.0_13]
	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
[na:1.6.0_13]
	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
[na:1.6.0_13]
	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
[na:1.6.0_13]
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[na:1.6.0_13]
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[na:1.6.0_13]
	at java.lang.Thread.run(Thread.java:619) [na:1.6.0_13]
13:39:41.296 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF]
DEBUG o.a.c.c.file.remote.FtpConsumer - Took 0.000 seconds to poll: MF
13:39:41.811 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF]
DEBUG o.a.c.c.file.remote.FtpConsumer - Took 0.000 seconds to poll: MF
13:39:42.326 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF]
DEBUG o.a.c.c.file.remote.FtpConsumer - Took 0.000 seconds to poll: MF



--
View this message in context: http://camel.465427.n5.nabble.com/out-of-heap-space-if-remote-FTP-site-has-too-many-files-to-pick-up-tp5747716p5747717.html
Sent from the Camel - Users mailing list archive at Nabble.com.