ConsumerTemplate for large number of files

[spilled_beverage <gr...@yahoo.co.uk>]

I'm consuming from a directory which can contain a large number of files
(currently trying with 1500).
When using a basic route all is fine and all files are moved from input to
output;



However, I need to use the ConsumerTemplate with something similar to;


The problem is that running with the ConsumerTemplate is not working.  The
majority of the files are moved (around 1150 but different each time) but
not all are.  The ones left in the input folder also have a .camelLock file
but they are not moved.

I've had a look through the documentation but can't find anything that
obviously addresses this issue.
Any help or pointers would be very gratefully received as I can't figure out
what the problem is.




--
View this message in context: http://camel.465427.n5.nabble.com/ConsumerTemplate-for-large-number-of-files-tp5733037.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: ConsumerTemplate for large number of files

[spilled_beverage <gr...@yahoo.co.uk>]

I don't actually just move the files from one folder to another - they are
consumed from file (via ConsumerTemplate) but then go to a direct route and
a whole host of other stuff happens - I just simplified the example in order
to be as clear as possible in my post.

My consumer either reads from a queue (default) or file (back-up mode)
depending on the mode that we are running.  When reading from a queue I need
to poll hence the ConsumerTemplate.  In back-up mode from file which is
where I was having the issue I might be able to change things to use a route
- I was just trying to keep the code common and either read from queue or
file depending on the mode.

Thanks for you help




--
View this message in context: http://camel.465427.n5.nabble.com/ConsumerTemplate-for-large-number-of-files-tp5733037p5733102.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: ConsumerTemplate for large number of files

[Claus Ibsen <cl...@gmail.com>]

Hi

The file component is intende for being used in routes, where it can work
on batches of files.

The polling consumer API you do is for a single message at a time, and
doesnt work as fast when the component is batch based.

So if you really want to move files from one dir to another from java code,
then write the java code using java.io.File API to do that.


On Fri, May 24, 2013 at 10:30 AM, spilled_beverage <
grahamduckett@yahoo.co.uk> wrote:

> Hi Claus,
>
> Thanks for the reply (and congratulations on the Camel in Action book -
> very
> readable and a great intro to Camel).
>
> Setting maxMessagesPerPoll=1 works but slows down the run hugely (from 10
> seconds to around 10 minutes in my quick test).
> Setting it to a higher value (e.g. 500) still works but runs quicker (back
> to seconds).
> However, setting it to a value such as 5000 causes the original problem -
> not all files are moved.
> Is this issue only when it is greater than the number of files?
> Is there a reason why setting it to 0 (so no limit) fails when using the
> ConsumerTemplate?
>
> Thanks for any pointers.
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/ConsumerTemplate-for-large-number-of-files-tp5733037p5733072.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Re: ConsumerTemplate for large number of files

[spilled_beverage <gr...@yahoo.co.uk>]

Hi Claus,

Thanks for the reply (and congratulations on the Camel in Action book - very
readable and a great intro to Camel).

Setting maxMessagesPerPoll=1 works but slows down the run hugely (from 10
seconds to around 10 minutes in my quick test).
Setting it to a higher value (e.g. 500) still works but runs quicker (back
to seconds).
However, setting it to a value such as 5000 causes the original problem -
not all files are moved.
Is this issue only when it is greater than the number of files?
Is there a reason why setting it to 0 (so no limit) fails when using the
ConsumerTemplate?

Thanks for any pointers.




--
View this message in context: http://camel.465427.n5.nabble.com/ConsumerTemplate-for-large-number-of-files-tp5733037p5733072.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: ConsumerTemplate for large number of files

[Claus Ibsen <cl...@gmail.com>]

Hi

Use maxMessagesPerPoll=1 when using the consumer template in the way you do.

On Thu, May 23, 2013 at 6:03 PM, spilled_beverage
<gr...@yahoo.co.uk> wrote:
> I'm consuming from a directory which can contain a large number of files
> (currently trying with 1500).
> When using a basic route all is fine and all files are moved from input to
> output;
>
>
>
> However, I need to use the ConsumerTemplate with something similar to;
>
>
> The problem is that running with the ConsumerTemplate is not working.  The
> majority of the files are moved (around 1150 but different each time) but
> not all are.  The ones left in the input folder also have a .camelLock file
> but they are not moved.
>
> I've had a look through the documentation but can't find anything that
> obviously addresses this issue.
> Any help or pointers would be very gratefully received as I can't figure out
> what the problem is.
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ConsumerTemplate-for-large-number-of-files-tp5733037.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen