You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by lunchbox <lu...@gmail.com> on 2011/07/11 10:58:18 UTC

Re: Scheduled Polling Consumer on FTP endpoint

Hi all,

I'm having the exact same problem as leNerd explains in the OP, except I'm
enriching from a JMS queue instead of a FTP server. Basically, I want ALL
the queued messages instead of just one.

What's the situation and the recommended solution with Camel now?

If we go with Willem's idea of starting and stopping a Camel route, how long
should we let it run? I think in my use case, we could have a few ten
messages in the queue at the most. How should we schedule the route stopping
-- with another quartz timer, or just sleep in the route starting thread, or
something else?

As background, the JMS queue is a temporary resend queue for messages which
have been processed for their content successfully, but have failed to be
sent over SOAP to another server (network interruptions etc). We store them
in the JMS queue and periodically attempt to resend (all of) them.

Thanks,
Lunchbox

--
View this message in context: http://camel.465427.n5.nabble.com/Scheduled-Polling-Consumer-on-FTP-endpoint-tp472602p4574353.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Scheduled Polling Consumer on FTP endpoint

Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Jul 13, 2011 at 8:57 AM, lunchbox <lu...@gmail.com> wrote:
> Hi Claus,
>
> Thanks for the info. But actually, the route is pulling messages from a JMS
> channel if you check my first post on this thread :)
>
> Also, it will most likely be a co-located or embedded JMS broker, so I think
> that will improve the performance quite a bit compared to a remote FTP
> server. But yeah, I'll do some testing and see how it all goes. Overall
> Camel's performance seems real nice so I'm quite optimistic.
>
> JMS is not listed in the batch consumer list so I guess that's off. I'm
> using context.start/stopRoute, but looking at the other link you posted,
> perhaps suspend/resume would be better?
>

If you are going to do this more frequently then suspend/resume is
more "gentle" as it keeps the connection warm etc.


> And, although the JMS consumer is not batched, is there a way to know when
> there are no more messages waiting in the queue? I'm using the
> ActiveMQComponent if that makes any difference.
>

You can use JMX API on ActiveMQ to query the queue depth on queues etc.

See this FAQ
http://activemq.apache.org/how-do-i-find-the-size-of-a-queue.html

You can also use the statistic plugin and do a request/reply and get a
message back with details about the queue. See this blog:
http://rajdavies.blogspot.com/2009/10/query-statistics-for-apache-activemq.html

And Camel has an inflight repository which you can query how many in
progress messages is from a given endpoint etc
http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/InflightRepository.html




> Thanks a lot,
> Lunchbox
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Scheduled-Polling-Consumer-on-FTP-endpoint-tp472602p4581766.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Scheduled Polling Consumer on FTP endpoint

Posted by lunchbox <lu...@gmail.com>.
Hi Claus,

Thanks for the info. But actually, the route is pulling messages from a JMS
channel if you check my first post on this thread :)

Also, it will most likely be a co-located or embedded JMS broker, so I think
that will improve the performance quite a bit compared to a remote FTP
server. But yeah, I'll do some testing and see how it all goes. Overall
Camel's performance seems real nice so I'm quite optimistic.

JMS is not listed in the batch consumer list so I guess that's off. I'm
using context.start/stopRoute, but looking at the other link you posted,
perhaps suspend/resume would be better?

And, although the JMS consumer is not batched, is there a way to know when
there are no more messages waiting in the queue? I'm using the
ActiveMQComponent if that makes any difference.

Thanks a lot,
Lunchbox

--
View this message in context: http://camel.465427.n5.nabble.com/Scheduled-Polling-Consumer-on-FTP-endpoint-tp472602p4581766.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Scheduled Polling Consumer on FTP endpoint

Posted by Claus Ibsen <cl...@gmail.com>.
Hi

5 sec is not much if its to active a route that causes it to download
files from a FTP server.
You may want to increase that value.

Do some testing and if it works, then it works :)


The ftp consumer is a batch consumer
http://camel.apache.org/batch-consumer.html

So you can actually know when its done, and thus signal this to that
other route which waits.
You can of course also just let the ftp route stop itself when its done.

Camel in Action book chapter 13 shows that. And there is a FAQ as well
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html



On Tue, Jul 12, 2011 at 2:56 PM, lunchbox <lu...@gmail.com> wrote:
> Hi Claus & Mitko,
>
> Thanks for the replies. I actually went forward with the route
> activation/deactivation scheme and my solution looks like follows:
>
> from(quartzScheduler)
>  .to(beanWhichActivatesResendRoute).delay(5000L)
>  .to(beanWhichDeactivatesResendRoute);
>
> from(...).routeId(RESEND) // route is initially not activated
>  .to(...)
>
> Since the messages may fail again, I put any failed resends in a temporary
> dead letter queue. Then, when the resend route is deactivated, the bean also
> activates another route which pumps the failed resends from the temporary
> queue back into the resend queue. This is so there isn't a loop of furiously
> resending when resends fail.
>
> Here I decided that 5 seconds is enough :) The quartz scheduler will likely
> schedule every one hour or something like that in production.
>
> Do you see any problem with this approach?
>
> Thanks a ton for the answers, plus, Camel is awesome! Keep up the great
> work.
>
> BR,
> Lunchbox
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Scheduled-Polling-Consumer-on-FTP-endpoint-tp472602p4578777.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Scheduled Polling Consumer on FTP endpoint

Posted by lunchbox <lu...@gmail.com>.
Hi Claus & Mitko,

Thanks for the replies. I actually went forward with the route
activation/deactivation scheme and my solution looks like follows:

from(quartzScheduler)
  .to(beanWhichActivatesResendRoute).delay(5000L)
  .to(beanWhichDeactivatesResendRoute);

from(...).routeId(RESEND) // route is initially not activated
  .to(...)

Since the messages may fail again, I put any failed resends in a temporary
dead letter queue. Then, when the resend route is deactivated, the bean also
activates another route which pumps the failed resends from the temporary
queue back into the resend queue. This is so there isn't a loop of furiously
resending when resends fail.

Here I decided that 5 seconds is enough :) The quartz scheduler will likely
schedule every one hour or something like that in production.

Do you see any problem with this approach?

Thanks a ton for the answers, plus, Camel is awesome! Keep up the great
work.

BR,
Lunchbox

--
View this message in context: http://camel.465427.n5.nabble.com/Scheduled-Polling-Consumer-on-FTP-endpoint-tp472602p4578777.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Scheduled Polling Consumer on FTP endpoint

Posted by mi...@icw.de.
Hi leNerd, 


I solved this by adding a pollStrategy to the file:

from("file:target/input/?pollStrategy=#pollStrategy&delay=10000")

from("quartz:yourSchedule")
 .bean("pollStrategy")

public class FilePollStrategy implements PollingConsumerPollStrategy {
    @Override
    public boolean begin(Consumer consumer, Endpoint endpoint) {
        return poll.get();
    }
 
    ...

    @Handler
    public void allowPolling(Exchange exchange){
        poll.set(true);
    }

}


Hope this helps, 
Mitko







From:
Claus Ibsen <cl...@gmail.com>
To:
users@camel.apache.org
Date:
11.07.2011 14:51
Subject:
Re: Scheduled Polling Consumer on FTP endpoint



On Mon, Jul 11, 2011 at 10:58 AM, lunchbox <lu...@gmail.com> 
wrote:
> Hi all,
>
> I'm having the exact same problem as leNerd explains in the OP, except 
I'm
> enriching from a JMS queue instead of a FTP server. Basically, I want 
ALL
> the queued messages instead of just one.
>

You can use a java bean / Camel Processor and then do a loop to drain
the message queue.



> What's the situation and the recommended solution with Camel now?
>
> If we go with Willem's idea of starting and stopping a Camel route, how 
long
> should we let it run? I think in my use case, we could have a few ten
> messages in the queue at the most. How should we schedule the route 
stopping
> -- with another quartz timer, or just sleep in the route starting 
thread, or
> something else?
>
> As background, the JMS queue is a temporary resend queue for messages 
which
> have been processed for their content successfully, but have failed to 
be
> sent over SOAP to another server (network interruptions etc). We store 
them
> in the JMS queue and periodically attempt to resend (all of) them.
>
> Thanks,
> Lunchbox
>
> --
> View this message in context: 
http://camel.465427.n5.nabble.com/Scheduled-Polling-Consumer-on-FTP-endpoint-tp472602p4574353.html

> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/





�� 

InterComponentWare AG:  
Vorstand: Peter Kirschbauer (Vors.), J�rg Stadler  
Aufsichtsratsvors.: Prof. Dr. Christof Hettich  
Firmensitz: 69190 Walldorf, Altrottstra�e 31  
AG Mannheim HRB 351761 / USt.-IdNr.: DE 198388516

Re: Scheduled Polling Consumer on FTP endpoint

Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Jul 11, 2011 at 10:58 AM, lunchbox <lu...@gmail.com> wrote:
> Hi all,
>
> I'm having the exact same problem as leNerd explains in the OP, except I'm
> enriching from a JMS queue instead of a FTP server. Basically, I want ALL
> the queued messages instead of just one.
>

You can use a java bean / Camel Processor and then do a loop to drain
the message queue.



> What's the situation and the recommended solution with Camel now?
>
> If we go with Willem's idea of starting and stopping a Camel route, how long
> should we let it run? I think in my use case, we could have a few ten
> messages in the queue at the most. How should we schedule the route stopping
> -- with another quartz timer, or just sleep in the route starting thread, or
> something else?
>
> As background, the JMS queue is a temporary resend queue for messages which
> have been processed for their content successfully, but have failed to be
> sent over SOAP to another server (network interruptions etc). We store them
> in the JMS queue and periodically attempt to resend (all of) them.
>
> Thanks,
> Lunchbox
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Scheduled-Polling-Consumer-on-FTP-endpoint-tp472602p4574353.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/