file component polling files

[redpower1989 <re...@hotmail.com>]

Hello,

I have a route which consumes from a file component and the message end up
to a activemq queue .
I was testing my application to consume a large amount of files. The
consumption's is slow and while i was trying to solve the problem i use the
follow uri:

file://inbox?move=backup/${date:now:yyyyMMdd}/${file:name}&delay=30000&maxMessagesPerPoll=1"

thinking  that the file component will pick up a single file every 30 secs
but it seems to consume the files like normal. Any idea why is it doing that
or am i doing something wrong? My idea is to use the maxMessagesPerPoll to
make the file component pick up a large amount of files every poll. Is that
going to speed up the consumption?

Thanks






--
View this message in context: http://camel.465427.n5.nabble.com/file-component-polling-files-tp5788570.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: file component polling files

[redpower1989 <re...@hotmail.com>]

My general goal is to consume fast large quantity of files but while i was
experimenting with file component in order to understood how polling works i
tried this case. To poll every 30 secs one file but it seems to pick more
than one file in those 30 secs and i was wondering if am doing something
wrong. Yes i am unmarshaling at a point between file consumer and queue
producer uri's. How is that affect the speed? What do you mean by getting
off the file reader?

Thanks 



--
View this message in context: http://camel.465427.n5.nabble.com/file-component-polling-files-tp5788570p5788846.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: file component polling files

[Brad Johnson <br...@mediadriver.com>]

I'm not quite sure what you are attempting to do here.  You poll every 30
seconds and seem to only want to pick up 1 file but then say you want it to
pick up a "large amount of files every poll".  Are you trying to pick up
just 1 file every 30 seconds or are you trying to read all the files when
the poll fires?  If you want to pick up everything it will do that
normally.  Are you unmarshaling these files at some point? Speeding things
up can be accomplished via thread pools and getting them off the file
reader can be accomplished by something as simple as a SEDA queue with a
thread pool.  But you also have to be aware of how much data you have in
each file and make sure it doesn't exceed your memory settings.  That
usually means tremendously big files because folks usually overestimate how
much data a file represents.

"thinking  that the file component will pick up a single file every 30 secs
but it seems to consume the files like normal. Any idea why is it doing that
or am i doing something wrong? My idea is to use the maxMessagesPerPoll to
make the file component pick up a large amount of files every poll. Is that
going to speed up the consumption?"

On Fri, Oct 14, 2016 at 3:48 AM, redpower1989 <re...@hotmail.com>
wrote:

> I tried maxMessagesPerPoll = 1 and delay 30 secs and it wasn't picking 1
> file
> every 30 secs and i am wondering if i am setting something wrong on the
> consumer.
>
> By the way i found a way to speed up the file processing and it is the use
> of threads. Be careful that you need to find a balance for the poolsize
> value.
> http://camel.apache.org/async.html
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/file-component-polling-files-tp5788570p5788775.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

Re: file component polling files

[redpower1989 <re...@hotmail.com>]

I tried maxMessagesPerPoll = 1 and delay 30 secs and it wasn't picking 1 file
every 30 secs and i am wondering if i am setting something wrong on the
consumer. 

By the way i found a way to speed up the file processing and it is the use
of threads. Be careful that you need to find a balance for the poolsize
value.
http://camel.apache.org/async.html



--
View this message in context: http://camel.465427.n5.nabble.com/file-component-polling-files-tp5788570p5788775.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: file component polling files

[DariusX <da...@gmail.com>]

The delay will control the frequency at which it polls the directory. To
limit the number of files each time, check out maxMessagesPerPoll and
eagermaxmessagesperPoll http://camel.apache.org/file2.html



--
View this message in context: http://camel.465427.n5.nabble.com/file-component-polling-files-tp5788570p5788599.html
Sent from the Camel - Users mailing list archive at Nabble.com.