Servicemix File Poller, Limit the number of files

[Hans Bogaards <hb...@gmail.com>]

Hi All,

I have a SA consisting of:
- a file poller SU polling a directory
- a jms SU putting the preprocessed data in an AcitveMQ queue
- a Camel SU that reads from the queue, processes that data further and then
load it into a database.

This works nicely as long as the number of files that file poller gets in
one poll is not too large.
The SA is processing 5 files per minute for live data (traffic data) which
isn't a problem.
But now I am asked to load all historical data for 1 year (which is
2,628,000 files)

Can I somehow limit number of file the file poller picks up in one poll?
-- 
Hans Bogaards
Software Engineer
TA Team
The Netherlands

Re: Servicemix File Poller, Limit the number of files

[Gert Vanthienen <ge...@gmail.com>]

Hans,

There's currently no way of limiting the number of files being polled.
 However, if you are using ServiceMix 3.2.x, the file poller endpoint
uses synchronous MessageExchanges to handle the file being polled, so
you'd never get more MessageExchanges in the NMR than there are
threads available to the component.

For ServiceMix 3.3, Chris Custine recently fixed an issue to handle
this scenario (https://issues.apache.org/activemq/browse/SMXCOMP-52)
so you might want to give that a try.

If you just want a quick, one-off fix, consider writing a little
FileFilter implementation that will only match x exchange per second
or something -- that would also limit the number of files being
processed.

Regards,

Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/



2009/2/26 Hans Bogaards <hb...@gmail.com>:
> Hi All,
>
> I have a SA consisting of:
> - a file poller SU polling a directory
> - a jms SU putting the preprocessed data in an AcitveMQ queue
> - a Camel SU that reads from the queue, processes that data further and then
> load it into a database.
>
> This works nicely as long as the number of files that file poller gets in
> one poll is not too large.
> The SA is processing 5 files per minute for live data (traffic data) which
> isn't a problem.
> But now I am asked to load all historical data for 1 year (which is
> 2,628,000 files)
>
> Can I somehow limit number of file the file poller picks up in one poll?
> --
> Hans Bogaards
> Software Engineer
> TA Team
> The Netherlands
>