You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Hilde <hi...@yahoo.de> on 2012/11/23 13:25:48 UTC
ftp-consumer: recovering a previous established connection
Hi there!
We have a polling ftp-consumer route configured as follows:
ftp://00001@upload-dev.mecdoc.local/?binary=true&consumer.bridgeErrorHandler=true&filter=#dlDataFileFilter
&localWorkDirectory=/EXPORTDATAD/tmp&maximumReconnectAttempts=3&noop=true
&pollStrategy=#dn1PollingConsumerPollStrategy&reconnectDelay=10000&sendEmptyMessageWhenIdle=true
&startingDirectoryMustExist=true&throwExceptionOnConnectFailed=true
The problem we have encounterd is the connection can be disrupted once it
was established first.
At this moment we are not trying to establish the conneciton in order to
enter the route but instead we are already
inside the polling ftp consumer route and trust on a valid connection.
Call me naive but I was conviced that the option /maximumReconnectAttept/
was intended to catch and remedy such a situation
namely tries to repair the using connection inside the route.
I have debbuged camel code an have figured out that only before establishing
the polling ftp consumer route the
option /maximumReconnectAttept/ is taking into account. The method
/prePollCheck()/ in class
/org.apache.camel.component.file.remote.RemoteFileConsumer<T>/ that uses
the option /maximumReconnectAttept/ is only called
before but not after the polling consumer route is established.
For Telling you the whole story I have an exception policy inside the route.
Could that be the reason to ignore option
/maximumReconnectAttept/?
*We are using Apache Camel 2.10.2*
Any ideas would be appreciated!
Cheers
Hilde
--
View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: ftp-consumer: recovering a previous established connection
Posted by Hildegard Schedthelm <hi...@yahoo.de>.
Hello Claus!
Now we solved it. We have used a customized poll strategy and therefore the option maximumReconnectAttempts
and reconnectDelay respectively were without effect. We could make the reconnect work by restart the consumer
in the poll strategy implementation's rollback() method:
public class DN1PollingConsumerPollStrategy implements PollingConsumerPollStrategy {
...
@Override
public boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause)
throws Exception {
LOG.error("Polling consumer failed: " + cause.getMessage(), cause);
if (cause.getMessage().contains("User cannot log")) {
throw new
DN1TransferException(
"The user cannot log in!");
}
LOG.info("Consumer is going to restart!");
consumer.stop();
consumer.start();
return false;
}
}
The code snippet above stops the route if the ftp login failed but otherwise always tries to reconnect
when a established connection was disruppted.
Because the ftp polling consumer route is a dynamic one we also check if the server is
reachable before the dynamic route is added to the camel context:
private boolean checkConnection(String host, Integer port) {
Socket socket = new Socket();
InetSocketAddress endPoint = new InetSocketAddress(host, port);
boolean isConnectionValid = false;
if (endPoint.isUnresolved()) {
return isConnectionValid;
} else {
try {
socket.connect(endPoint, 2000);
isConnectionValid = true;
} catch (IOException ioe) {
return isConnectionValid;
} finally {
if (socket != null) {
try {
socket.close();
}
catch (IOException ioe) {
LOG.error("Error occured by closing the ftp connection", ioe);
}
}
}
}
return isConnectionValid;
}
By this we have the chance to react accordingly when the server is out of range or the
ftp login failed, but more importantly otherwise the reconnect stays intact.
Cheers
Hilde
________________________________
Von: Claus Ibsen <cl...@gmail.com>
An: users@camel.apache.org; Hildegard Schedthelm <hi...@yahoo.de>
Gesendet: 11:34 Donnerstag, 29.November 2012
Betreff: Re: ftp-consumer: recovering a previous established connection
On Thu, Nov 29, 2012 at 10:12 AM, Hildegard Schedthelm
<hi...@yahoo.de> wrote:
> Hello Claus!
>
> Thanks a lot for
helping.
>
>
> I cannot conceive how a different poll interval can help to start polling at all.
>
> The ftp component derived from the file component that has the batch consumer
>
> implemented to poll multiple exchanges. When I start downloading e.g. about 10 files from
>
> a ftp server I'd infer that polling occurs just once. That might be the reason the re-connect
>
> doesn't trigger as a consequence no polling occurs any more. Are theses notions correct?
>
If you download 4 files, and have problem with the 5th, then the
consumer will fail.
And on next scheduled poll, it will re-connect and start again, but as
the previous 4 files has been downloaded,
then it will start with the 5th file.
Re: ftp-consumer: recovering a previous established connection
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Nov 29, 2012 at 10:12 AM, Hildegard Schedthelm
<hi...@yahoo.de> wrote:
> Hello Claus!
>
> Thanks a lot for helping.
>
>
> I cannot conceive how a different poll interval can help to start polling at all.
>
> The ftp component derived from the file component that has the batch consumer
>
> implemented to poll multiple exchanges. When I start downloading e.g. about 10 files from
>
> a ftp server I'd infer that polling occurs just once. That might be the reason the re-connect
>
> doesn't trigger as a consequence no polling occurs any more. Are theses notions correct?
>
If you download 4 files, and have problem with the 5th, then the
consumer will fail.
And on next scheduled poll, it will re-connect and start again, but as
the previous 4 files has been downloaded,
then it will start with the 5th file.
>
> Cheers
> Hilde
>
>
>
> ________________________________
> Von: Claus Ibsen <cl...@gmail.com>
> An: users@camel.apache.org
> Gesendet: 19:01 Mittwoch, 28.November 2012
> Betreff: Re: ftp-consumer: recovering a previous established connection
>
> On Tue, Nov 27, 2012 at 8:57 AM, Hildegard Schedthelm
> <hi...@yahoo.de> wrote:
>> Hello All!
>>
>> Has someone any ideas how to trigger a reconnect? A new poll probably not
>> occur.
>>
>
> The ftp consumer is scheduled. There is options to configure how
> frequent it should poll.
> By default that's very fast, I think its half a second interval
> between last pool ends.
>
>>
>> Hilde
>>
>>
>>
>> ________________________________
>> Von: Hilde <hi...@yahoo.de>
>> An: users@camel.apache.org
>> Gesendet: 12:55 Montag, 26.November 2012
>> Betreff: Re: ftp-consumer: recovering a previous established connection
>>
>> When does the next poll starts? My debugging session under the hood of camel
>> couldn't discover any activities to call for a reconnect although
>> downloading
>> files was still in progress. Is a new poll suspended because the ftp
>> component
>> implements the Batch Consumer? How can I trigger a new poll explicitly that
>> causes camel with an attempt to reconnect?
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161p5723231.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: cibsen@redhat.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: ftp-consumer: recovering a previous established connection
Posted by Hildegard Schedthelm <hi...@yahoo.de>.
Hello Claus!
Thanks a lot for helping.
I cannot conceive how a different poll interval can help to start polling at all.
The ftp component derived from the file component that has the batch consumer
implemented to poll multiple exchanges. When I start downloading e.g. about 10 files from
a ftp server I'd infer that polling occurs just once. That might be the reason the re-connect
doesn't trigger as a consequence no polling occurs any more. Are theses notions correct?
Cheers
Hilde
________________________________
Von: Claus Ibsen <cl...@gmail.com>
An: users@camel.apache.org
Gesendet: 19:01 Mittwoch, 28.November 2012
Betreff: Re: ftp-consumer: recovering a previous established connection
On Tue, Nov 27, 2012 at 8:57 AM, Hildegard Schedthelm
<hi...@yahoo.de> wrote:
> Hello All!
>
> Has someone any ideas how to trigger a reconnect? A new poll probably not
> occur.
>
The ftp consumer is scheduled. There is options to configure how
frequent it should poll.
By default that's very fast, I think its half a second interval
between last pool ends.
>
> Hilde
>
>
>
> ________________________________
> Von: Hilde <hi...@yahoo.de>
> An: users@camel.apache.org
> Gesendet: 12:55 Montag, 26.November 2012
> Betreff: Re: ftp-consumer: recovering a previous established connection
>
> When does the next poll starts? My debugging session under the hood of camel
> couldn't discover any activities to call for a reconnect although
> downloading
> files was still in progress. Is a new poll suspended because the ftp
> component
> implements the Batch Consumer? How can I trigger a new poll explicitly that
> causes camel with an attempt to reconnect?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161p5723231.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: ftp-consumer: recovering a previous established connection
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Nov 27, 2012 at 8:57 AM, Hildegard Schedthelm
<hi...@yahoo.de> wrote:
> Hello All!
>
> Has someone any ideas how to trigger a reconnect? A new poll probably not
> occur.
>
The ftp consumer is scheduled. There is options to configure how
frequent it should poll.
By default that's very fast, I think its half a second interval
between last pool ends.
>
> Hilde
>
>
>
> ________________________________
> Von: Hilde <hi...@yahoo.de>
> An: users@camel.apache.org
> Gesendet: 12:55 Montag, 26.November 2012
> Betreff: Re: ftp-consumer: recovering a previous established connection
>
> When does the next poll starts? My debugging session under the hood of camel
> couldn't discover any activities to call for a reconnect although
> downloading
> files was still in progress. Is a new poll suspended because the ftp
> component
> implements the Batch Consumer? How can I trigger a new poll explicitly that
> causes camel with an attempt to reconnect?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161p5723231.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: ftp-consumer: recovering a previous established connection
Posted by Hildegard Schedthelm <hi...@yahoo.de>.
Hello All!
Has someone any ideas how to trigger a reconnect? A new poll probably not
occur.
Hilde
________________________________
Von: Hilde <hi...@yahoo.de>
An: users@camel.apache.org
Gesendet: 12:55 Montag, 26.November 2012
Betreff: Re: ftp-consumer: recovering a previous established connection
When does the next poll starts? My debugging session under the hood of camel
couldn't discover any activities to call for a reconnect although
downloading
files was still in progress. Is a new poll suspended because the ftp
component
implements the Batch Consumer? How can I trigger a new poll explicitly that
causes camel with an attempt to reconnect?
--
View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161p5723231.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: ftp-consumer: recovering a previous established connection
Posted by Hilde <hi...@yahoo.de>.
When does the next poll starts? My debugging session under the hood of camel
couldn't discover any activities to call for a reconnect although
downloading
files was still in progress. Is a new poll suspended because the ftp
component
implements the Batch Consumer? How can I trigger a new poll explicitly that
causes camel with an attempt to reconnect?
--
View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161p5723231.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: ftp-consumer: recovering a previous established connection
Posted by Claus Ibsen <cl...@gmail.com>.
Ah a typo of mine, I mean next poll. eg when the ftp consumer will run
again. Then it will re-connect and pickup the files which wasn't
transfered last time.
On Mon, Nov 26, 2012 at 10:52 AM, Hilde <hi...@yahoo.de> wrote:
> Hello Claus!
>
> Thanks for answering! The requirements are to download a whole range of
> files that needs
> some time. The ftp-consumer component should cope with connection breaks and
> continue
> downloading after trying to reconnect.
>
> Could you please explain me what do you mean when talking about a pool? I am
> not aware
> of using a connection pool for the ftp connection. Supposed I use one how
> can I avoid
> using a already corrupt connection from the pool?
>
> Hilde
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161p5723222.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: ftp-consumer: recovering a previous established connection
Posted by Hilde <hi...@yahoo.de>.
Hello Claus!
Thanks for answering! The requirements are to download a whole range of
files that needs
some time. The ftp-consumer component should cope with connection breaks and
continue
downloading after trying to reconnect.
Could you please explain me what do you mean when talking about a pool? I am
not aware
of using a connection pool for the ftp connection. Supposed I use one how
can I avoid
using a already corrupt connection from the pool?
Hilde
--
View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161p5723222.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: ftp-consumer: recovering a previous established connection
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 23, 2012 at 1:25 PM, Hilde <hi...@yahoo.de> wrote:
> Hi there!
>
> We have a polling ftp-consumer route configured as follows:
>
> ftp://00001@upload-dev.mecdoc.local/?binary=true&consumer.bridgeErrorHandler=true&filter=#dlDataFileFilter
> &localWorkDirectory=/EXPORTDATAD/tmp&maximumReconnectAttempts=3&noop=true
> &pollStrategy=#dn1PollingConsumerPollStrategy&reconnectDelay=10000&sendEmptyMessageWhenIdle=true
> &startingDirectoryMustExist=true&throwExceptionOnConnectFailed=true
>
> The problem we have encounterd is the connection can be disrupted once it
> was established first.
> At this moment we are not trying to establish the conneciton in order to
> enter the route but instead we are already
> inside the polling ftp consumer route and trust on a valid connection.
> Call me naive but I was conviced that the option /maximumReconnectAttept/
> was intended to catch and remedy such a situation
> namely tries to repair the using connection inside the route.
> I have debbuged camel code an have figured out that only before establishing
> the polling ftp consumer route the
> option /maximumReconnectAttept/ is taking into account. The method
> /prePollCheck()/ in class
> /org.apache.camel.component.file.remote.RemoteFileConsumer<T>/ that uses
> the option /maximumReconnectAttept/ is only called
> before but not after the polling consumer route is established.
> For Telling you the whole story I have an exception policy inside the route.
> Could that be the reason to ignore option
> /maximumReconnectAttept/?
>
> *We are using Apache Camel 2.10.2*
>
> Any ideas would be appreciated!
>
It will re-connect on next pool. So if there is a problem during a
pooling, then that would fail.
And on next scheduled pool, it would re-connection connection.
> Cheers
> Hilde
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ftp-consumer-recovering-a-previous-established-connection-tp5723161.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen