You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Deepti Bhogle <de...@coverago.com> on 2011/01/12 00:57:04 UTC
how to find out if a ftp file consumer has finished reading all files available?
Hello everyone,
I have a route set up to connect to a remote ftp server to download and
process documents.
I wish to shutdown the route after it finishes processing all the documents
available from the ftp server.
I dont have any way to find out if the given ftp consumer got all files.
Is there any way the ftp component will tell us that it has completed
reading all files,so that I can trigger the onCompletion to shutdown the
route.
I am using Camel 2.4.
Thank you,
--
Deepti Bhogle
Software Engineer
C O V E R A G O
744 Broad Street, 32nd Floor
Newark, NJ 07102
(w) +1 973-200-6820
deepti.bhogle@coverago.com
www.coverago.com
Re: how to find out if a ftp file consumer has finished reading all
files available?
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Jan 12, 2011 at 11:14 PM, David Yang <da...@coverago.com> wrote:
>
> Thanks for the responses. We looked through the materials but they didn't quite solve our problem. What we want to do is use Camel's FtpConsumer for a one time shot:
>
> 1. Check a ftp site
> 2. Process all the files
> 3. Shut down after processing (the entire JVM etc).
>
> I thought one path we could go down is somehow check the pendingExchanges inside the GenericFileConsumer, but we didn't find an obvious way to do that and we weren't sure what to do if we didn't get any new files (no exchanges generated, no Processors get run).
>
The ugly solution would allow the polling consumer to emit an empty
exchange into the route if there was no data to poll.
That causes the route to deal with exchanges which either contains a
file or no data at all. So this is not a desireable approach.
> Is there some way to use the RoutePolicy's to somehow check if there are any exchanges and then control the CamelContext (and the rest of the JVM via that method)?
>
RoutePolicy is tied into when an exchange is being routed. It doesn't
provide knowledge when a consumer is polling something, or when a
consumer is doing something active to intake messages.
In your case its a polling consumer based, which allows us to improve
the API to allow end users to cater for this more easily.
We have the PollingConsumerPollStrategy API which is for polling
consumers. It allows you control on begin / commit / rollback.
What it currently lacks is that it doesnt provide details in the
commit message how many messages it did poll. So all we need to do is
to improve this.
Then you can use a custom poll strategy on the ftp endpoint where you
can shutdown the consumer. And eventually also the entire JVM, but
stopping the CamelContext. Or signal to some flag that it should stop
so the main loop can shutdown camel context and exit and thus
terminate the JVM.
I have created ticket
https://issues.apache.org/jira/browse/CAMEL-3538
> Thanks,
> David
>
>
> On Jan 12, 2011, at 12:53 AM, Claus Ibsen wrote:
>
>> See also chapter 13 in the Camel book
>>
>> And this FAQ
>> http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
>>
>>
>> On Wed, Jan 12, 2011 at 12:57 AM, Deepti Bhogle
>> <de...@coverago.com> wrote:
>>> Hello everyone,
>>> I have a route set up to connect to a remote ftp server to download and
>>> process documents.
>>> I wish to shutdown the route after it finishes processing all the documents
>>> available from the ftp server.
>>> I dont have any way to find out if the given ftp consumer got all files.
>>> Is there any way the ftp component will tell us that it has completed
>>> reading all files,so that I can trigger the onCompletion to shutdown the
>>> route.
>>> I am using Camel 2.4.
>>> Thank you,
>>>
>>> --
>>> Deepti Bhogle
>>> Software Engineer
>>> C O V E R A G O
>>> 744 Broad Street, 32nd Floor
>>> Newark, NJ 07102
>>> (w) +1 973-200-6820
>>> deepti.bhogle@coverago.com
>>> www.coverago.com
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: how to find out if a ftp file consumer has finished reading all files available?
Posted by David Yang <da...@coverago.com>.
Thanks for the responses. We looked through the materials but they didn't quite solve our problem. What we want to do is use Camel's FtpConsumer for a one time shot:
1. Check a ftp site
2. Process all the files
3. Shut down after processing (the entire JVM etc).
I thought one path we could go down is somehow check the pendingExchanges inside the GenericFileConsumer, but we didn't find an obvious way to do that and we weren't sure what to do if we didn't get any new files (no exchanges generated, no Processors get run).
Is there some way to use the RoutePolicy's to somehow check if there are any exchanges and then control the CamelContext (and the rest of the JVM via that method)?
Thanks,
David
On Jan 12, 2011, at 12:53 AM, Claus Ibsen wrote:
> See also chapter 13 in the Camel book
>
> And this FAQ
> http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
>
>
> On Wed, Jan 12, 2011 at 12:57 AM, Deepti Bhogle
> <de...@coverago.com> wrote:
>> Hello everyone,
>> I have a route set up to connect to a remote ftp server to download and
>> process documents.
>> I wish to shutdown the route after it finishes processing all the documents
>> available from the ftp server.
>> I dont have any way to find out if the given ftp consumer got all files.
>> Is there any way the ftp component will tell us that it has completed
>> reading all files,so that I can trigger the onCompletion to shutdown the
>> route.
>> I am using Camel 2.4.
>> Thank you,
>>
>> --
>> Deepti Bhogle
>> Software Engineer
>> C O V E R A G O
>> 744 Broad Street, 32nd Floor
>> Newark, NJ 07102
>> (w) +1 973-200-6820
>> deepti.bhogle@coverago.com
>> www.coverago.com
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
Re: how to find out if a ftp file consumer has finished reading all
files available?
Posted by Claus Ibsen <cl...@gmail.com>.
See also chapter 13 in the Camel book
And this FAQ
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
On Wed, Jan 12, 2011 at 12:57 AM, Deepti Bhogle
<de...@coverago.com> wrote:
> Hello everyone,
> I have a route set up to connect to a remote ftp server to download and
> process documents.
> I wish to shutdown the route after it finishes processing all the documents
> available from the ftp server.
> I dont have any way to find out if the given ftp consumer got all files.
> Is there any way the ftp component will tell us that it has completed
> reading all files,so that I can trigger the onCompletion to shutdown the
> route.
> I am using Camel 2.4.
> Thank you,
>
> --
> Deepti Bhogle
> Software Engineer
> C O V E R A G O
> 744 Broad Street, 32nd Floor
> Newark, NJ 07102
> (w) +1 973-200-6820
> deepti.bhogle@coverago.com
> www.coverago.com
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: how to find out if a ftp file consumer has finished reading all
files available?
Posted by Ashwin Karpe <ak...@fusesource.com>.
Hi,
The camel-ftp component is a polling component. On each pass it will pick up
all the files present in the remote directory. If you want to run this like
a batch job on a schedule please check out
http://camel.apache.org/simplescheduledroutepolicy.html
http://camel.apache.org/simplescheduledroutepolicy.html
http://camel.apache.org/cronscheduledroutepolicy.html
http://camel.apache.org/cronscheduledroutepolicy.html
You can do something like
from("ftp:....")
.routeId("test")
.routePolicy(policy)
.to("...");
Hope this helps.
Cheers,
Ashwin...
-----
---------------------------------------------------------
Ashwin Karpe
Apache Camel Committer & Sr Principal Consultant
FUSESource (a Progress Software Corporation subsidiary)
http://fusesource.com http://fusesource.com
Blog: http://opensourceknowledge.blogspot.com
http://opensourceknowledge.blogspot.com
---------------------------------------------------------
--
View this message in context: http://camel.465427.n5.nabble.com/how-to-find-out-if-a-ftp-file-consumer-has-finished-reading-all-files-available-tp3337532p3337564.html
Sent from the Camel - Users mailing list archive at Nabble.com.