You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@camel.apache.org by Davide Rossi <da...@gmail.com> on 2019/01/13 12:17:39 UTC

Connection problem when moving last ftp file and stopping route

Hi everyone!
I have a connection problem when stopping a ftp route. My route is
something like this:













*<routes xmlns="http://camel.apache.org/schema/spring
<http://camel.apache.org/schema/spring>">    <route id="jobIdFtp_1">
<from
uri="ftp://admin@localhost:2121/testFolder?move=.done&amp;moveFailed=.error&amp;delay=3000000&amp;recursive=true&amp;antInclude=**&amp;password=admin&amp;download=true&amp;binary=true&amp;passiveMode=true&amp;sendEmptyMessageWhenIdle=true&amp;maxMessagesPerPoll=1000&amp;readLockDeleteOrphanLockFiles=false&amp;autoCreate=false&amp;localWorkDirectory=C:\Users\Test\Desktop\testAcquisition&amp;throwExceptionOnConnectFailed=true&amp;consumer.bridgeErrorHandler=true"/>
    <to uri="bean:myProcessor"/>        <choice>
<when>
<simple>${exchangeProperty.CamelBatchComplete}</simple>                <to
uri="bean:stopRouteProcessor"/>            </when>        </choice>
</route></routes>*
where *stopRouteProcessor *is a prcessor that stops the route just as
reported in the documentation
public void process(final Exchange exchange) throws Exception {
        Thread stop = new Thread() {
            @Override
            public void run() {
                String routeId = exchange.getFromRouteId();
                try {
                    SpringCamelContext exchangeContext =
(SpringCamelContext) exchange.getContext();
                    RouteDefinition routeDefinition =
exchangeContext.getRouteDefinition(routeId);

exchangeContext.stopRoute(routeId);

exchangeContext.removeRoute(routeId);

exchangeContext.removeRouteDefinition(routeDefinition);

                } catch (Exception e) {
                    ....
                }
            }
        };
        // start the thread that stops this route
        stop.start();
    }

The route is correctly stopped, but I receive a connection problem when
Camel tries to move on the server the last acquired file, because the
connection is already closed, just because of the route stop (in fact, if I
remove the stopRouteProcessor everything works fine).
2019-01-11 16:53:24,303 | lMediaProcessorQueue | WARN  |
GenericFileOnCompletion                  | Error during commit.
Exchange[ID-User-NBK-50955-1547221992014-0-6][Message: [Body is file based:
C:\Users\Test\Desktop\testAcquisition\mioFile.txt]]. Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException - File
operation failed: null Connection is not open. Code: 221]
org.apache.camel.component.file.GenericFileOperationFailedException: File
operation failed: null Connection is not open. Code: 221
    at
org.apache.camel.component.file.remote.FtpOperations.buildDirectory(FtpOperations.java:304)
    at
org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:106)
    at
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
    at
org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
    at
org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at
org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
    at
org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
    at
org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
    at
org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
    at
org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:653)
    at
org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:608)
    at
org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:239)
    at
org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:250)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:491)
    at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
    at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
    at
org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:298)
    at
org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:207)
    at
org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:154)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Connection is not open
    at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:472)
    at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
    at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
    at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
    at
org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
    at
org.apache.camel.component.file.remote.FtpOperations.buildDirectory(FtpOperations.java:281)
    ... 21 common frames omitted

Now, I know that onCompletion is executed in a separated thread, but is
there a way to know if/when onCompletion is executed for the last exchange,
so I can stop the route only after that? If this is not possible, is there
a different approach/solution to be sure that even the last file will be
correctly moved?
Thank you
Regards
Davide

Re: Connection problem when moving last ftp file and stopping route

Posted by Davide Rossi <da...@gmail.com>.

Hi Claus,
thanks for your reply.
So, yes, I need to stop every route after their processing because in my
system I have routes as well as other "crawling" tasks that I need to start
and stop according to specific policies.
Of course, I can add a delay before starting the thread but I was looking
for something "surer", just to avoid any problem in case of delay.
Probably, the best approach is onCompletion on the last exchange. I already
evaluated that, but hoped that changes could be made only in the
stopRouteProcessor, just to avoid updates in the different routes I have
and, consequently, system updates (in my case not so simple, unfortunately).
Anyway, thank you again for your support!
Best regards
Davide

Il giorno lun 14 gen 2019 alle ore 16:54 Claus Ibsen <cl...@gmail.com>
ha scritto:

> Hi
>
> You can also add that stop route stuff as an on-completion to the
> current exchange.
>
> There is API on Exchange for that.
> And you can implemented Ordered and get it an ordering number to be
> executed LAST.
>
> And you can also try to add a delay in the code that stops the route
> to give the on completion time to complete first.
>
> And btw do you want to stop the route always after it has processed some
> files?
> Why dont you want to keep the route running? Or are you looking for
> running this only once or something?
>
> On Sun, Jan 13, 2019 at 1:17 PM Davide Rossi <da...@gmail.com>
> wrote:
> >
> > Hi everyone!
> > I have a connection problem when stopping a ftp route. My route is
> > something like this:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *<routes xmlns="http://camel.apache.org/schema/spring
> > <http://camel.apache.org/schema/spring>">    <route id="jobIdFtp_1">
> > <from
> > uri="ftp://admin@localhost
> :2121/testFolder?move=.done&amp;moveFailed=.error&amp;delay=3000000&amp;recursive=true&amp;antInclude=**&amp;password=admin&amp;download=true&amp;binary=true&amp;passiveMode=true&amp;sendEmptyMessageWhenIdle=true&amp;maxMessagesPerPoll=1000&amp;readLockDeleteOrphanLockFiles=false&amp;autoCreate=false&amp;localWorkDirectory=C:\Users\Test\Desktop\testAcquisition&amp;throwExceptionOnConnectFailed=true&amp;consumer.bridgeErrorHandler=true"/>
> >     <to uri="bean:myProcessor"/>        <choice>
> > <when>
> > <simple>${exchangeProperty.CamelBatchComplete}</simple>
> <to
> > uri="bean:stopRouteProcessor"/>            </when>        </choice>
> > </route></routes>*
> > where *stopRouteProcessor *is a prcessor that stops the route just as
> > reported in the documentation
> > public void process(final Exchange exchange) throws Exception {
> >         Thread stop = new Thread() {
> >             @Override
> >             public void run() {
> >                 String routeId = exchange.getFromRouteId();
> >                 try {
> >                     SpringCamelContext exchangeContext =
> > (SpringCamelContext) exchange.getContext();
> >                     RouteDefinition routeDefinition =
> > exchangeContext.getRouteDefinition(routeId);
> >
> > exchangeContext.stopRoute(routeId);
> >
> > exchangeContext.removeRoute(routeId);
> >
> > exchangeContext.removeRouteDefinition(routeDefinition);
> >
> >                 } catch (Exception e) {
> >                     ....
> >                 }
> >             }
> >         };
> >         // start the thread that stops this route
> >         stop.start();
> >     }
> >
> > The route is correctly stopped, but I receive a connection problem when
> > Camel tries to move on the server the last acquired file, because the
> > connection is already closed, just because of the route stop (in fact,
> if I
> > remove the stopRouteProcessor everything works fine).
> > 2019-01-11 16:53:24,303 | lMediaProcessorQueue | WARN  |
> > GenericFileOnCompletion                  | Error during commit.
> > Exchange[ID-User-NBK-50955-1547221992014-0-6][Message: [Body is file
> based:
> > C:\Users\Test\Desktop\testAcquisition\mioFile.txt]]. Caused by:
> > [org.apache.camel.component.file.GenericFileOperationFailedException -
> File
> > operation failed: null Connection is not open. Code: 221]
> > org.apache.camel.component.file.GenericFileOperationFailedException: File
> > operation failed: null Connection is not open. Code: 221
> >     at
> >
> org.apache.camel.component.file.remote.FtpOperations.buildDirectory(FtpOperations.java:304)
> >     at
> >
> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:106)
> >     at
> >
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> >     at
> >
> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
> >     at
> >
> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
> >     at
> >
> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
> >     at
> >
> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
> >     at
> > org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
> >     at
> > org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
> >     at
> >
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:653)
> >     at
> >
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:608)
> >     at
> >
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:239)
> >     at
> >
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:250)
> >     at
> >
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:491)
> >     at
> >
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> >     at
> >
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> >     at
> >
> org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:298)
> >     at
> > org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:207)
> >     at
> > org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:154)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >     at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.io.IOException: Connection is not open
> >     at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:472)
> >     at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
> >     at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
> >     at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
> >     at
> >
> org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
> >     at
> >
> org.apache.camel.component.file.remote.FtpOperations.buildDirectory(FtpOperations.java:281)
> >     ... 21 common frames omitted
> >
> > Now, I know that onCompletion is executed in a separated thread, but is
> > there a way to know if/when onCompletion is executed for the last
> exchange,
> > so I can stop the route only after that? If this is not possible, is
> there
> > a different approach/solution to be sure that even the last file will be
> > correctly moved?
> > Thank you
> > Regards
> > Davide
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>

Re: Connection problem when moving last ftp file and stopping route

Posted by Claus Ibsen <cl...@gmail.com>.

Hi

You can also add that stop route stuff as an on-completion to the
current exchange.

There is API on Exchange for that.
And you can implemented Ordered and get it an ordering number to be
executed LAST.

And you can also try to add a delay in the code that stops the route
to give the on completion time to complete first.

And btw do you want to stop the route always after it has processed some files?
Why dont you want to keep the route running? Or are you looking for
running this only once or something?

On Sun, Jan 13, 2019 at 1:17 PM Davide Rossi <da...@gmail.com> wrote:
>
> Hi everyone!
> I have a connection problem when stopping a ftp route. My route is
> something like this:
>
>
>
>
>
>
>
>
>
>
>
>
>
> *<routes xmlns="http://camel.apache.org/schema/spring
> <http://camel.apache.org/schema/spring>">    <route id="jobIdFtp_1">
> <from
> uri="ftp://admin@localhost:2121/testFolder?move=.done&amp;moveFailed=.error&amp;delay=3000000&amp;recursive=true&amp;antInclude=**&amp;password=admin&amp;download=true&amp;binary=true&amp;passiveMode=true&amp;sendEmptyMessageWhenIdle=true&amp;maxMessagesPerPoll=1000&amp;readLockDeleteOrphanLockFiles=false&amp;autoCreate=false&amp;localWorkDirectory=C:\Users\Test\Desktop\testAcquisition&amp;throwExceptionOnConnectFailed=true&amp;consumer.bridgeErrorHandler=true"/>
>     <to uri="bean:myProcessor"/>        <choice>
> <when>
> <simple>${exchangeProperty.CamelBatchComplete}</simple>                <to
> uri="bean:stopRouteProcessor"/>            </when>        </choice>
> </route></routes>*
> where *stopRouteProcessor *is a prcessor that stops the route just as
> reported in the documentation
> public void process(final Exchange exchange) throws Exception {
>         Thread stop = new Thread() {
>             @Override
>             public void run() {
>                 String routeId = exchange.getFromRouteId();
>                 try {
>                     SpringCamelContext exchangeContext =
> (SpringCamelContext) exchange.getContext();
>                     RouteDefinition routeDefinition =
> exchangeContext.getRouteDefinition(routeId);
>
> exchangeContext.stopRoute(routeId);
>
> exchangeContext.removeRoute(routeId);
>
> exchangeContext.removeRouteDefinition(routeDefinition);
>
>                 } catch (Exception e) {
>                     ....
>                 }
>             }
>         };
>         // start the thread that stops this route
>         stop.start();
>     }
>
> The route is correctly stopped, but I receive a connection problem when
> Camel tries to move on the server the last acquired file, because the
> connection is already closed, just because of the route stop (in fact, if I
> remove the stopRouteProcessor everything works fine).
> 2019-01-11 16:53:24,303 | lMediaProcessorQueue | WARN  |
> GenericFileOnCompletion                  | Error during commit.
> Exchange[ID-User-NBK-50955-1547221992014-0-6][Message: [Body is file based:
> C:\Users\Test\Desktop\testAcquisition\mioFile.txt]]. Caused by:
> [org.apache.camel.component.file.GenericFileOperationFailedException - File
> operation failed: null Connection is not open. Code: 221]
> org.apache.camel.component.file.GenericFileOperationFailedException: File
> operation failed: null Connection is not open. Code: 221
>     at
> org.apache.camel.component.file.remote.FtpOperations.buildDirectory(FtpOperations.java:304)
>     at
> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:106)
>     at
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>     at
> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
>     at
> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
>     at
> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
>     at
> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
>     at
> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
>     at
> org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
>     at
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:653)
>     at
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:608)
>     at
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:239)
>     at
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:250)
>     at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:491)
>     at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>     at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>     at
> org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:298)
>     at
> org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:207)
>     at
> org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:154)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Connection is not open
>     at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:472)
>     at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
>     at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
>     at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
>     at
> org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
>     at
> org.apache.camel.component.file.remote.FtpOperations.buildDirectory(FtpOperations.java:281)
>     ... 21 common frames omitted
>
> Now, I know that onCompletion is executed in a separated thread, but is
> there a way to know if/when onCompletion is executed for the last exchange,
> so I can stop the route only after that? If this is not possible, is there
> a different approach/solution to be sure that even the last file will be
> correctly moved?
> Thank you
> Regards
> Davide



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2