Good / Adviced way to poll an entire folder at scheduled moment

[Jonathan Schoreels <jo...@gmail.com>]

Hi.

I would like to know, in your experience, what's the best way to poll an
entire folder/remote folder, at an regular interval ?

It could seems quite easy, since the the file consumer can be used as a
batch consumer, that a scheduler can be used with it, but I have some
limitation each time :

1. Batch consumer : How to set a "good maxMessagePerPoll" ? If I hardcode
it, and poll those messages each day, I'm not future proof if more file
comes.
2. poll enrich only poll one at a time. So if I need to poll ALL the folder
each day, it doesn't help me.
3. having to start/stop routes with controle bus seems lot of
tweaks/hacking just to design a simple flow.

Have you any suggestion ?

Jonathan

Re: Good / Adviced way to poll an entire folder at scheduled moment

[Jonathan Schoreels <jo...@gmail.com>]

Hi,

No thoughts about that really trivial use case ? I discovered the "greedy"
option in polled consumre which could help, with a timer defined to once a
day, to poll all the file once a day without limitation of the number.
However, I don't really know if it's possible to couple that to a
pollenrich or whatever the mean to trigger the consumption at a certain
moment, based on another message, for example.

Jonathan

2017-05-08 16:49 GMT+02:00 Jonathan Schoreels <jo...@gmail.com>
:

> Hi.
>
> I would like to know, in your experience, what's the best way to poll an
> entire folder/remote folder, at an regular interval ?
>
> It could seems quite easy, since the the file consumer can be used as a
> batch consumer, that a scheduler can be used with it, but I have some
> limitation each time :
>
> 1. Batch consumer : How to set a "good maxMessagePerPoll" ? If I hardcode
> it, and poll those messages each day, I'm not future proof if more file
> comes.
> 2. poll enrich only poll one at a time. So if I need to poll ALL the
> folder each day, it doesn't help me.
> 3. having to start/stop routes with controle bus seems lot of
> tweaks/hacking just to design a simple flow.
>
> Have you any suggestion ?
>
> Jonathan
>

Re: Good / Adviced way to poll an entire folder at scheduled moment

[Quinn Stevenson <qu...@pronoia-solutions.com>]

I’ve done this with a combination of a ScheduledRoutePolicy (to start the route when I want to start reading the directory) and a call to the control bus when all the files have been processed.  I use the “sendEmptyMessageWhenIdle” option on the file component to detect when it’s done.

HTH

> On May 8, 2017, at 8:49 AM, Jonathan Schoreels <jo...@gmail.com> wrote:
> 
> Hi.
> 
> I would like to know, in your experience, what's the best way to poll an
> entire folder/remote folder, at an regular interval ?
> 
> It could seems quite easy, since the the file consumer can be used as a
> batch consumer, that a scheduler can be used with it, but I have some
> limitation each time :
> 
> 1. Batch consumer : How to set a "good maxMessagePerPoll" ? If I hardcode
> it, and poll those messages each day, I'm not future proof if more file
> comes.
> 2. poll enrich only poll one at a time. So if I need to poll ALL the folder
> each day, it doesn't help me.
> 3. having to start/stop routes with controle bus seems lot of
> tweaks/hacking just to design a simple flow.
> 
> Have you any suggestion ?
> 
> Jonathan

Re: Good / Adviced way to poll an entire folder at scheduled moment

[Claus Ibsen <cl...@gmail.com>]

You can build your own route policy that controls when the route runs
where you can start/stop accordingly to your needs.
You will need some kind of background task to wake up the route again
on the next day.


On Mon, May 8, 2017 at 4:49 PM, Jonathan Schoreels
<jo...@gmail.com> wrote:
> Hi.
>
> I would like to know, in your experience, what's the best way to poll an
> entire folder/remote folder, at an regular interval ?
>
> It could seems quite easy, since the the file consumer can be used as a
> batch consumer, that a scheduler can be used with it, but I have some
> limitation each time :
>
> 1. Batch consumer : How to set a "good maxMessagePerPoll" ? If I hardcode
> it, and poll those messages each day, I'm not future proof if more file
> comes.
> 2. poll enrich only poll one at a time. So if I need to poll ALL the folder
> each day, it doesn't help me.
> 3. having to start/stop routes with controle bus seems lot of
> tweaks/hacking just to design a simple flow.
>
> Have you any suggestion ?
>
> Jonathan



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2