You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Niels <ni...@kabisa.nl> on 2012/01/16 09:25:38 UTC
Exception FTP Component
Hey everybody,
I'm using to ftp component to consume from a ftp server.
The little application seems to do the job but when investigating the log files this exception shows up:
Trying to recover by disconnecting from remote server forcing a re-connect at next poll: ftp://userl@server:21
Error occurred while disconnecting from ftp://user@server:21 due: File operation failed: 421 Terminating connection.
FTP response 421 received. Server closed connection.. Code: 421. This exception will be ignored.
Consumer FtpConsumer[ftp://server?delay=900000&delete=true&password=******&username=user] could not poll endpoint: Endpoint[ftp://server?de$
FTP response 421 received. Server closed connection.. Code: 421
org.apache.camel.component.file.GenericFileOperationFailedException: File operation failed: 421 Timeout (120 seconds): closing control connection.
FTP response 421 received. Server closed connection.. Code: 421
at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:569)
at org.apache.camel.component.file.remote.FtpConsumer.pollDirectory(FtpConsumer.java:45)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:100)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:140)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.commons.net.ftp.FTPConnectionClosedException: FTP response 421 received. Server closed connection.
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:347)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:495)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
at org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:567)
... 13 more
This is the route I'm using to consume the service:
from("ftp://" + props.getProperty(FTP_URL) + "?username=" + props.getProperty(FTP_USER) + "&password=" + props.getProperty(FTP_PWD) + "&delete=true&delay=" + props.getProperty(FTP_DELAY))
.filter(xmlPredicate)
.to("jms:xmlQueue").end()
.filter(imgPredicate)
.to("file://" + props.getProperty(ROOT_DIR) + "/img").end()
.filter(pdfPredicate)
.to("file://" + props.getProperty(ROOT_DIR) + "/pdf").end()
.to("file://" + props.getProperty(ROOT_DIR) + "/junk");
Now I'm wondering if this could be an issue with camel? In my opinion I think it has something to do with the FTP server itself and not so much with camel.
Therefore I'm asking some experts opinion on my issue.
And please if you have any remarks on my code, I happy to here them I'm just a novice when it comes to camel.
Thanks!
Regards,
Niels
Re: Exception FTP Component
Posted by Niels <ni...@kabisa.nl>.
I changed my routing and added a catch for an exception.
I thought everything was running fine but apparently it doesn't.
I still get this exception:
Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot retrieve file: GenericFile[DZ_20120104_DZ_004.PDF] from:
Endpoint[ftp://server?delay=300000&delete=true&disconnect=true&password=******&username=user]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
retrieve file: GenericFile[DZ_20120104_DZ_004.PDF] from:
Endpoint[ftp://server?delay=300000&delete=true&disconnect=true&password=******&username=user]
at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:338)
at
org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:94)
at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:175)
at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:136)
at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:140)
at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
I changed my route to this:
//XML Predicate
//only allows names without spaces
Predicate xmlPredicate =
header(CAMEL_FILE_NAME).regex("([\\S]+(\\.(?i)(xml))$)");
//Images Predicate
//only allows names without spaces
Predicate imgPredicate =
header(CAMEL_FILE_NAME).regex("([\\S]+(\\.(?i)(jpg|png|gif))$)");
//PDF Predicate
//only allows names without spaces
Predicate pdfPredicate =
header(CAMEL_FILE_NAME).regex("([\\S]+(\\.(?i)(pdf))$)");
onException(SchemaValidationException.class)
.to("file://" + props.getProperty(ROOT_DIR) + "/invalid");
from("ftp://" + props.getProperty(FTP_URL) + "?username=" +
props.getProperty(FTP_USER) + "&password=" + props.getProperty(FTP_PWD) +
"&disconnect=true&delete=true&delay=" + props.getProperty(FTP_DELAY))
.choice()
.when(xmlPredicate)
.to("jms:xmlQueue")
.when(imgPredicate)
.to("file://" + props.getProperty(ROOT_DIR) + "/img")
.when(pdfPredicate)
.to("file://" + props.getProperty(ROOT_DIR) + "/pdf")
.otherwise()
.to("file://" + props.getProperty(ROOT_DIR) + "/junk");
from("jms:xmlQueue")
.to("validator:FtpXmlValidator.xsd")
.to("xslt://XmlToRssConverter.xsl")
.to("file://" + props.getProperty(ROOT_DIR) + "/rss");
It was already running for a week now. This error occured when I dropped
like 42 pdf's (size between 5- 10 mb) on the ftp server.
But the days before it processed 435 files, sporadicly added to the server,
like 5 at a time or less.
It seems like the ftp has problems when it has to fetch a large numbers of
files. When I first run camel, it started to process the ftp server, at the
moment there were 1200 files stored on it. All xml files or small images, so
is it running out of memory or what is causing this exception.
Do I need to add more memory to the JVM? Or is it something else?
Like I mentioned before I'm using :
Debian 5.0
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)
Hopefully you guys have some advise
regards
Camel: 2.9.0
--
View this message in context: http://camel.465427.n5.nabble.com/Exception-FTP-Component-tp5148122p5160347.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Exception FTP Component
Posted by Niels <ni...@kabisa.nl>.
Thanks for the fast reply.
I already changed to content-based routing.
I'm using the latest version of camel 2.9.0.
It's better to use the disconnect=true option like you mentioned, because there is a 5 min interval between the polling.
Will update a soon as I can redeploy my application en re-encounter the same problem in the logs.
Regards,
Niels
On Jan 16, 2012, at 11:46 AM, Claus Ibsen wrote:
> Hi
>
> Just a note about your Java DSL route.
>
> What you do looks much more like the Content Based Router (eg if ...
> else if ... else if ... else).
> http://camel.apache.org/content-based-router
>
> So you may want to look into that instead of using the filter EIP if
> you think the code is more readable.
>
>
> And in terms of the FTP error. What version of Camel are you using? We
> have fixed some pieces in recent times about detecting
> broken connection and re-connection.
>
> If you do not want to keep the connection alive, you can use the
> disconnect=true option. Then you force Camel to re-login on each poll.
> This could make sense if there is a long delay between polls.
>
>
>
> On Mon, Jan 16, 2012 at 9:25 AM, Niels <ni...@kabisa.nl> wrote:
>> Hey everybody,
>>
>> I'm using to ftp component to consume from a ftp server.
>> The little application seems to do the job but when investigating the log files this exception shows up:
>>
>> Trying to recover by disconnecting from remote server forcing a re-connect at next poll: ftp://userl@server:21
>> Error occurred while disconnecting from ftp://user@server:21 due: File operation failed: 421 Terminating connection.
>> FTP response 421 received. Server closed connection.. Code: 421. This exception will be ignored.
>> Consumer FtpConsumer[ftp://server?delay=900000&delete=true&password=******&username=user] could not poll endpoint: Endpoint[ftp://server?de$
>> FTP response 421 received. Server closed connection.. Code: 421
>> org.apache.camel.component.file.GenericFileOperationFailedException: File operation failed: 421 Timeout (120 seconds): closing control connection.
>> FTP response 421 received. Server closed connection.. Code: 421
>> at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:569)
>> at org.apache.camel.component.file.remote.FtpConsumer.pollDirectory(FtpConsumer.java:45)
>> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:100)
>> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:140)
>> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: org.apache.commons.net.ftp.FTPConnectionClosedException: FTP response 421 received. Server closed connection.
>> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:347)
>> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:495)
>> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
>> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
>> at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
>> at org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
>> at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:567)
>> ... 13 more
>>
>> This is the route I'm using to consume the service:
>>
>> from("ftp://" + props.getProperty(FTP_URL) + "?username=" + props.getProperty(FTP_USER) + "&password=" + props.getProperty(FTP_PWD) + "&delete=true&delay=" + props.getProperty(FTP_DELAY))
>> .filter(xmlPredicate)
>> .to("jms:xmlQueue").end()
>> .filter(imgPredicate)
>> .to("file://" + props.getProperty(ROOT_DIR) + "/img").end()
>> .filter(pdfPredicate)
>> .to("file://" + props.getProperty(ROOT_DIR) + "/pdf").end()
>> .to("file://" + props.getProperty(ROOT_DIR) + "/junk");
>>
>> Now I'm wondering if this could be an issue with camel? In my opinion I think it has something to do with the FTP server itself and not so much with camel.
>> Therefore I'm asking some experts opinion on my issue.
>>
>> And please if you have any remarks on my code, I happy to here them I'm just a novice when it comes to camel.
>>
>> Thanks!
>>
>> Regards,
>>
>> Niels
>>
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
Re: Exception FTP Component
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Just a note about your Java DSL route.
What you do looks much more like the Content Based Router (eg if ...
else if ... else if ... else).
http://camel.apache.org/content-based-router
So you may want to look into that instead of using the filter EIP if
you think the code is more readable.
And in terms of the FTP error. What version of Camel are you using? We
have fixed some pieces in recent times about detecting
broken connection and re-connection.
If you do not want to keep the connection alive, you can use the
disconnect=true option. Then you force Camel to re-login on each poll.
This could make sense if there is a long delay between polls.
On Mon, Jan 16, 2012 at 9:25 AM, Niels <ni...@kabisa.nl> wrote:
> Hey everybody,
>
> I'm using to ftp component to consume from a ftp server.
> The little application seems to do the job but when investigating the log files this exception shows up:
>
> Trying to recover by disconnecting from remote server forcing a re-connect at next poll: ftp://userl@server:21
> Error occurred while disconnecting from ftp://user@server:21 due: File operation failed: 421 Terminating connection.
> FTP response 421 received. Server closed connection.. Code: 421. This exception will be ignored.
> Consumer FtpConsumer[ftp://server?delay=900000&delete=true&password=******&username=user] could not poll endpoint: Endpoint[ftp://server?de$
> FTP response 421 received. Server closed connection.. Code: 421
> org.apache.camel.component.file.GenericFileOperationFailedException: File operation failed: 421 Timeout (120 seconds): closing control connection.
> FTP response 421 received. Server closed connection.. Code: 421
> at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:569)
> at org.apache.camel.component.file.remote.FtpConsumer.pollDirectory(FtpConsumer.java:45)
> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:100)
> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:140)
> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.commons.net.ftp.FTPConnectionClosedException: FTP response 421 received. Server closed connection.
> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:347)
> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:495)
> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
> at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
> at org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
> at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:567)
> ... 13 more
>
> This is the route I'm using to consume the service:
>
> from("ftp://" + props.getProperty(FTP_URL) + "?username=" + props.getProperty(FTP_USER) + "&password=" + props.getProperty(FTP_PWD) + "&delete=true&delay=" + props.getProperty(FTP_DELAY))
> .filter(xmlPredicate)
> .to("jms:xmlQueue").end()
> .filter(imgPredicate)
> .to("file://" + props.getProperty(ROOT_DIR) + "/img").end()
> .filter(pdfPredicate)
> .to("file://" + props.getProperty(ROOT_DIR) + "/pdf").end()
> .to("file://" + props.getProperty(ROOT_DIR) + "/junk");
>
> Now I'm wondering if this could be an issue with camel? In my opinion I think it has something to do with the FTP server itself and not so much with camel.
> Therefore I'm asking some experts opinion on my issue.
>
> And please if you have any remarks on my code, I happy to here them I'm just a novice when it comes to camel.
>
> Thanks!
>
> Regards,
>
> Niels
>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/