You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by hefiso <he...@gmail.com> on 2013/04/24 13:29:18 UTC
File2: maxMessagesPerPoll combined with readLock=changed
Hi
Usually the file-endpoint (File2) allows for processing more than one file
per poll by setting the property maxMessagesPerPoll.
However, when also using readLock=changed, then the file-consumer does not
consume more than one file per poll, regardless of the value specified for
maxMessagesPerPoll. When changing readLock back to "markerFile" then the
file endpoint is again able to consume several files per poll.
I am surprised by this behaviour, is it by design or should I log a JIRA?
Best regards
Henrik
--
View this message in context: http://camel.465427.n5.nabble.com/File2-maxMessagesPerPoll-combined-with-readLock-changed-tp5731434.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File2: maxMessagesPerPoll combined with readLock=changed
Posted by hefiso <he...@gmail.com>.
Hi
OK, I see. Of course the fileconsumer may impose an order on consumation.
However, it may very well happen that several files (in the right order)
will become ready for consumation within the same readLockTimeout-period and
these files *could* be processed in one go.
However, it is probably not worth the effort to implement this optimization
when new features in JDK 7 almost solves the problem for us ;-)
/Henrik
--
View this message in context: http://camel.465427.n5.nabble.com/File2-maxMessagesPerPoll-combined-with-readLock-changed-tp5731434p5731720.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File2: maxMessagesPerPoll combined with readLock=changed
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Ah this is by design. The file component read lock strategy is per
file, and don't do a "bulk operation".
Also if you want to process files in order (eg maybe sorted) etc, then
you would need to grab the read lock in that order as well.
When we move to Java7 as minimum JDK version, then we can leverage the
NIO2 file watcher API.
It may have some easier API for watching new files being saved to directories.
And btw you should write files using temporary names, and then rename
to final name when done. Then its easier for the other party to pickup
the files as they can match on file name.
Having to use readLock=change is not an ideal way of exchanging files,
as you really don't know when the other party is done writing the
file.
On Wed, Apr 24, 2013 at 9:18 PM, hefiso
<he...@gmail.com> wrote:
> Hi
>
> The setting maxMessagesPerPoll is actually respected but the files will only
> be processed one by one with a delay matching readLockTimeout. This
> behaviour is happening because the method acquireExclusiveReadLock in
> FileChangedExclusiveReadLockStrategy effectively calls
> Thread.sleep(checkInterval) for a single file. When using a high setting for
> readLockTimeout, say 10 seconds, then the throughput really suffers.
>
> This is also the case with FileRenameExclusiveReadLockStrategy (since it
> extends GenericFileRenameExclusiveReadLockStrategy<File>) because it also
> calls Thread.sleep when processing a single file.
>
> I don't see a quick fix for this since the Thread.sleep must be performed at
> a higher level than on a single file (i.e. after trying to process all the
> files in the collection given by the poll).
>
> I will log a JIRA shortly.
>
> Best regards
> Henrik
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File2-maxMessagesPerPoll-combined-with-readLock-changed-tp5731434p5731476.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: File2: maxMessagesPerPoll combined with readLock=changed
Posted by hefiso <he...@gmail.com>.
Hi
The setting maxMessagesPerPoll is actually respected but the files will only
be processed one by one with a delay matching readLockTimeout. This
behaviour is happening because the method acquireExclusiveReadLock in
FileChangedExclusiveReadLockStrategy effectively calls
Thread.sleep(checkInterval) for a single file. When using a high setting for
readLockTimeout, say 10 seconds, then the throughput really suffers.
This is also the case with FileRenameExclusiveReadLockStrategy (since it
extends GenericFileRenameExclusiveReadLockStrategy<File>) because it also
calls Thread.sleep when processing a single file.
I don't see a quick fix for this since the Thread.sleep must be performed at
a higher level than on a single file (i.e. after trying to process all the
files in the collection given by the poll).
I will log a JIRA shortly.
Best regards
Henrik
--
View this message in context: http://camel.465427.n5.nabble.com/File2-maxMessagesPerPoll-combined-with-readLock-changed-tp5731434p5731476.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File2: maxMessagesPerPoll combined with readLock=changed
Posted by Claus Ibsen <cl...@gmail.com>.
Hi Henrik
If you are able to double check again that this behavior is seen, when
you have, for example, 10 files ready to be picked up, and have
maxMessagesPerPoll=50, and still the consumer only pickup 1 file per
poll. Then yeah for sure it sounds like a bug, and feel free to log a
JIRA.
On Wed, Apr 24, 2013 at 1:29 PM, hefiso
<he...@gmail.com> wrote:
> Hi
>
> Usually the file-endpoint (File2) allows for processing more than one file
> per poll by setting the property maxMessagesPerPoll.
>
> However, when also using readLock=changed, then the file-consumer does not
> consume more than one file per poll, regardless of the value specified for
> maxMessagesPerPoll. When changing readLock back to "markerFile" then the
> file endpoint is again able to consume several files per poll.
>
> I am surprised by this behaviour, is it by design or should I log a JIRA?
>
> Best regards
> Henrik
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File2-maxMessagesPerPoll-combined-with-readLock-changed-tp5731434.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: File2: maxMessagesPerPoll combined with readLock=changed
Posted by Claus Ibsen <cl...@gmail.com>.
It could be, or maybe there is only 1 file that is not changed. The
read lock is only granted if Camel "sees" the file as not being
changed (which is that its file size / timestamp) is not changing over
a period of time.
On Wed, Apr 24, 2013 at 1:29 PM, hefiso
<he...@gmail.com> wrote:
> Hi
>
> Usually the file-endpoint (File2) allows for processing more than one file
> per poll by setting the property maxMessagesPerPoll.
>
> However, when also using readLock=changed, then the file-consumer does not
> consume more than one file per poll, regardless of the value specified for
> maxMessagesPerPoll. When changing readLock back to "markerFile" then the
> file endpoint is again able to consume several files per poll.
>
> I am surprised by this behaviour, is it by design or should I log a JIRA?
>
> Best regards
> Henrik
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File2-maxMessagesPerPoll-combined-with-readLock-changed-tp5731434.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen