You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Vladimir Okhotnikov <vo...@gmail.com> on 2009/10/08 16:57:12 UTC
File component blocked by existing files
Hi
I'm observing quite a strange behavior at the moment. In my app, I have
several routes which all start by reading files in the same directory,
deleting them after processing. Obviously, each rule uses its own include
pattern to process separate subset of files.
Now, the problem is that if the camel context is started when there are some
existing files in the directory, the matching routes do not seem to work. I
cannot see any related activity in the log file (except for the first "1
file to consume message"), and the newly added files are being piled up in
the directory. The other routes, which had no matching files on camel start
do work.
Is it a bug or am I just missing something?
--
View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25803233.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component blocked by existing files
Posted by Vladimir Okhotnikov <vo...@gmail.com>.
Thanks for the changes and suggestions - for my case just setting the timeout
to 5 or 10 seconds should suffice, but at least I got some insight about
async options and the reason file component does work sequentially.
Claus Ibsen-2 wrote:
>
> BTW if you want to speed up the file routing you can use the async
> concurrency. A bit on this blog
> http://davsclaus.blogspot.com/2009/05/on-road-to-camel-20-concurrency-with.html
>
> Note the DSL was renamed from async to threads
>
> And then use a single file consumer to avoid race conditions for
> grabbing new files.
>
>
> On Mon, Oct 12, 2009 at 12:56 PM, Claus Ibsen <cl...@gmail.com>
> wrote:
>> On Mon, Oct 12, 2009 at 12:27 PM, Vladimir Okhotnikov
>> <vo...@gmail.com> wrote:
>>>
>>> Sorry for the poor English - I know about the timeout option now - I saw
>>> it
>>> immediately AFTER I found the reason. I obviously saw the option before,
>>> among others, but did not pay enough attention to realize I should not
>>> use
>>> the default.
>>>
>>
>> You english is good. Just as poor/good as mine :)
>>
>> You have a good point about the default timeout not being good. I have
>> just changed it to 10 seconds and also Camel will now log at WARN
>> level if the lock could not be acquired within that period. Before it
>> was just DEBUG level.
>>
>>> And I did have the camel DEBUG level enabled - was the first thing I
>>> have
>>> done to debug the problem - and it did not help me at all - there were
>>> no
>>> relevant messages at all - and the DEBUG logging did work - I get a log
>>> of
>>> all routes on camel start etc.
>>>
>>
>> You can enable DEBUG for only the file component
>> org.apache.camel.component.file=DEBUG
>>
>>
>>> Claus, I'm not trying to blame or rant or something. I still think the
>>> framework is awesome. I apologize if I somehow made you think I'm
>>> feeling
>>> different. I just truly believe that infinite waits by default should be
>>> ruthlessly eliminated, and doing this will benefit the Camel and its
>>> users
>>> a great deal.
>>>
>>
>> None taken. I may have well started this, sorry as well. Its just that
>> there is alot of free based support which does take much time.
>> But also gives great feedback on the Camel. All together helps improve
>> the community, which is ultimate the best for us and Apache.
>>
>> In you case you can set the timeout to a low value such as 1, 100,
>> 500, 1000 or 2000 etc to let Camel quickly try the next file so you
>> get it to process files quicker.
>>
>>
>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Mon, Oct 12, 2009 at 10:15 AM, Vladimir Okhotnikov
>>>> <vo...@gmail.com> wrote:
>>>>>
>>>>> Yes, I known about the readLockTimeout option. The problem with this
>>>>> option
>>>>> is that by a) default it waits forever and b) while it waits for one
>>>>> file
>>>>> it
>>>>> does not process others. That's what made me spend 2 days wondering
>>>>> about
>>>>> the behavior described in the thread almost without clues, and it will
>>>>> made
>>>>> others.
>>>>
>>>> You did not mention that you knew this option and why have you not
>>>> tried using it?
>>>>
>>>> Camel does TRACE / DEBUG logging which you could have enabled.
>>>>
>>>> Which is also written on the file component wiki page
>>>> http://camel.apache.org/file2.html
>>>>
>>>> Debug logging
>>>> This component has log level TRACE that can be helpful if you have
>>>> problems.
>>>>
>>>> So if you have done that you would NOT have been in the dark for 2
>>>> days.
>>>>
>>>>
>>>>
>>>> The file consumer is NOT concurrent. So yes it will wait.
>>>>
>>>> If you want concurrency then define multiple routes with a file
>>>> consumer. Which btw is not fun as file system is not transactional and
>>>> having multiple consumers racing for the same files is not fun.
>>>>
>>>>
>>>> Good idea about the default timeout which I have created a ticket for
>>>> https://issues.apache.org/activemq/browse/CAMEL-2062
>>>>
>>>>
>>>>
>>>>>
>>>>> As is pretty well explained in the "Release It" book, infinite waits
>>>>> are
>>>>> bad. Infinite waits by default are very bad, especially in integration
>>>>> software - it virtually guarantees that at least one will slip into
>>>>> your
>>>>> live system, then it's just a matter of time until it blocks critical
>>>>> function or critical amount of resources - and you're waken in the
>>>>> middle
>>>>> of
>>>>> the night to deal with it.
>>>>>
>>>>> What do you think?
>>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>>
>>>>>> You can already do this by the readLockTimeout option. See more
>>>>>> details
>>>>>> here:
>>>>>> http://camel.apache.org/file2.html
>>>>>>
>>>>>> For example you can set it to 10000 as 10 seconds and if Camel cannot
>>>>>> acquire the lock within that period it will skip this file and try
>>>>>> the
>>>>>> next one.
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25851978.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25853488.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25854546.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component blocked by existing files
Posted by Claus Ibsen <cl...@gmail.com>.
BTW if you want to speed up the file routing you can use the async
concurrency. A bit on this blog
http://davsclaus.blogspot.com/2009/05/on-road-to-camel-20-concurrency-with.html
Note the DSL was renamed from async to threads
And then use a single file consumer to avoid race conditions for
grabbing new files.
On Mon, Oct 12, 2009 at 12:56 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Mon, Oct 12, 2009 at 12:27 PM, Vladimir Okhotnikov
> <vo...@gmail.com> wrote:
>>
>> Sorry for the poor English - I know about the timeout option now - I saw it
>> immediately AFTER I found the reason. I obviously saw the option before,
>> among others, but did not pay enough attention to realize I should not use
>> the default.
>>
>
> You english is good. Just as poor/good as mine :)
>
> You have a good point about the default timeout not being good. I have
> just changed it to 10 seconds and also Camel will now log at WARN
> level if the lock could not be acquired within that period. Before it
> was just DEBUG level.
>
>> And I did have the camel DEBUG level enabled - was the first thing I have
>> done to debug the problem - and it did not help me at all - there were no
>> relevant messages at all - and the DEBUG logging did work - I get a log of
>> all routes on camel start etc.
>>
>
> You can enable DEBUG for only the file component
> org.apache.camel.component.file=DEBUG
>
>
>> Claus, I'm not trying to blame or rant or something. I still think the
>> framework is awesome. I apologize if I somehow made you think I'm feeling
>> different. I just truly believe that infinite waits by default should be
>> ruthlessly eliminated, and doing this will benefit the Camel and its users
>> a great deal.
>>
>
> None taken. I may have well started this, sorry as well. Its just that
> there is alot of free based support which does take much time.
> But also gives great feedback on the Camel. All together helps improve
> the community, which is ultimate the best for us and Apache.
>
> In you case you can set the timeout to a low value such as 1, 100,
> 500, 1000 or 2000 etc to let Camel quickly try the next file so you
> get it to process files quicker.
>
>
>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Mon, Oct 12, 2009 at 10:15 AM, Vladimir Okhotnikov
>>> <vo...@gmail.com> wrote:
>>>>
>>>> Yes, I known about the readLockTimeout option. The problem with this
>>>> option
>>>> is that by a) default it waits forever and b) while it waits for one file
>>>> it
>>>> does not process others. That's what made me spend 2 days wondering about
>>>> the behavior described in the thread almost without clues, and it will
>>>> made
>>>> others.
>>>
>>> You did not mention that you knew this option and why have you not
>>> tried using it?
>>>
>>> Camel does TRACE / DEBUG logging which you could have enabled.
>>>
>>> Which is also written on the file component wiki page
>>> http://camel.apache.org/file2.html
>>>
>>> Debug logging
>>> This component has log level TRACE that can be helpful if you have
>>> problems.
>>>
>>> So if you have done that you would NOT have been in the dark for 2 days.
>>>
>>>
>>>
>>> The file consumer is NOT concurrent. So yes it will wait.
>>>
>>> If you want concurrency then define multiple routes with a file
>>> consumer. Which btw is not fun as file system is not transactional and
>>> having multiple consumers racing for the same files is not fun.
>>>
>>>
>>> Good idea about the default timeout which I have created a ticket for
>>> https://issues.apache.org/activemq/browse/CAMEL-2062
>>>
>>>
>>>
>>>>
>>>> As is pretty well explained in the "Release It" book, infinite waits are
>>>> bad. Infinite waits by default are very bad, especially in integration
>>>> software - it virtually guarantees that at least one will slip into your
>>>> live system, then it's just a matter of time until it blocks critical
>>>> function or critical amount of resources - and you're waken in the middle
>>>> of
>>>> the night to deal with it.
>>>>
>>>> What do you think?
>>>>
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>>
>>>>> You can already do this by the readLockTimeout option. See more details
>>>>> here:
>>>>> http://camel.apache.org/file2.html
>>>>>
>>>>> For example you can set it to 10000 as 10 seconds and if Camel cannot
>>>>> acquire the lock within that period it will skip this file and try the
>>>>> next one.
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25851978.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25853488.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File component blocked by existing files
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Oct 12, 2009 at 12:27 PM, Vladimir Okhotnikov
<vo...@gmail.com> wrote:
>
> Sorry for the poor English - I know about the timeout option now - I saw it
> immediately AFTER I found the reason. I obviously saw the option before,
> among others, but did not pay enough attention to realize I should not use
> the default.
>
You english is good. Just as poor/good as mine :)
You have a good point about the default timeout not being good. I have
just changed it to 10 seconds and also Camel will now log at WARN
level if the lock could not be acquired within that period. Before it
was just DEBUG level.
> And I did have the camel DEBUG level enabled - was the first thing I have
> done to debug the problem - and it did not help me at all - there were no
> relevant messages at all - and the DEBUG logging did work - I get a log of
> all routes on camel start etc.
>
You can enable DEBUG for only the file component
org.apache.camel.component.file=DEBUG
> Claus, I'm not trying to blame or rant or something. I still think the
> framework is awesome. I apologize if I somehow made you think I'm feeling
> different. I just truly believe that infinite waits by default should be
> ruthlessly eliminated, and doing this will benefit the Camel and its users
> a great deal.
>
None taken. I may have well started this, sorry as well. Its just that
there is alot of free based support which does take much time.
But also gives great feedback on the Camel. All together helps improve
the community, which is ultimate the best for us and Apache.
In you case you can set the timeout to a low value such as 1, 100,
500, 1000 or 2000 etc to let Camel quickly try the next file so you
get it to process files quicker.
>
> Claus Ibsen-2 wrote:
>>
>> On Mon, Oct 12, 2009 at 10:15 AM, Vladimir Okhotnikov
>> <vo...@gmail.com> wrote:
>>>
>>> Yes, I known about the readLockTimeout option. The problem with this
>>> option
>>> is that by a) default it waits forever and b) while it waits for one file
>>> it
>>> does not process others. That's what made me spend 2 days wondering about
>>> the behavior described in the thread almost without clues, and it will
>>> made
>>> others.
>>
>> You did not mention that you knew this option and why have you not
>> tried using it?
>>
>> Camel does TRACE / DEBUG logging which you could have enabled.
>>
>> Which is also written on the file component wiki page
>> http://camel.apache.org/file2.html
>>
>> Debug logging
>> This component has log level TRACE that can be helpful if you have
>> problems.
>>
>> So if you have done that you would NOT have been in the dark for 2 days.
>>
>>
>>
>> The file consumer is NOT concurrent. So yes it will wait.
>>
>> If you want concurrency then define multiple routes with a file
>> consumer. Which btw is not fun as file system is not transactional and
>> having multiple consumers racing for the same files is not fun.
>>
>>
>> Good idea about the default timeout which I have created a ticket for
>> https://issues.apache.org/activemq/browse/CAMEL-2062
>>
>>
>>
>>>
>>> As is pretty well explained in the "Release It" book, infinite waits are
>>> bad. Infinite waits by default are very bad, especially in integration
>>> software - it virtually guarantees that at least one will slip into your
>>> live system, then it's just a matter of time until it blocks critical
>>> function or critical amount of resources - and you're waken in the middle
>>> of
>>> the night to deal with it.
>>>
>>> What do you think?
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>>
>>>> You can already do this by the readLockTimeout option. See more details
>>>> here:
>>>> http://camel.apache.org/file2.html
>>>>
>>>> For example you can set it to 10000 as 10 seconds and if Camel cannot
>>>> acquire the lock within that period it will skip this file and try the
>>>> next one.
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25851978.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25853488.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File component blocked by existing files
Posted by Vladimir Okhotnikov <vo...@gmail.com>.
Sorry for the poor English - I know about the timeout option now - I saw it
immediately AFTER I found the reason. I obviously saw the option before,
among others, but did not pay enough attention to realize I should not use
the default.
And I did have the camel DEBUG level enabled - was the first thing I have
done to debug the problem - and it did not help me at all - there were no
relevant messages at all - and the DEBUG logging did work - I get a log of
all routes on camel start etc.
Claus, I'm not trying to blame or rant or something. I still think the
framework is awesome. I apologize if I somehow made you think I'm feeling
different. I just truly believe that infinite waits by default should be
ruthlessly eliminated, and doing this will benefit the Camel and its users
a great deal.
Claus Ibsen-2 wrote:
>
> On Mon, Oct 12, 2009 at 10:15 AM, Vladimir Okhotnikov
> <vo...@gmail.com> wrote:
>>
>> Yes, I known about the readLockTimeout option. The problem with this
>> option
>> is that by a) default it waits forever and b) while it waits for one file
>> it
>> does not process others. That's what made me spend 2 days wondering about
>> the behavior described in the thread almost without clues, and it will
>> made
>> others.
>
> You did not mention that you knew this option and why have you not
> tried using it?
>
> Camel does TRACE / DEBUG logging which you could have enabled.
>
> Which is also written on the file component wiki page
> http://camel.apache.org/file2.html
>
> Debug logging
> This component has log level TRACE that can be helpful if you have
> problems.
>
> So if you have done that you would NOT have been in the dark for 2 days.
>
>
>
> The file consumer is NOT concurrent. So yes it will wait.
>
> If you want concurrency then define multiple routes with a file
> consumer. Which btw is not fun as file system is not transactional and
> having multiple consumers racing for the same files is not fun.
>
>
> Good idea about the default timeout which I have created a ticket for
> https://issues.apache.org/activemq/browse/CAMEL-2062
>
>
>
>>
>> As is pretty well explained in the "Release It" book, infinite waits are
>> bad. Infinite waits by default are very bad, especially in integration
>> software - it virtually guarantees that at least one will slip into your
>> live system, then it's just a matter of time until it blocks critical
>> function or critical amount of resources - and you're waken in the middle
>> of
>> the night to deal with it.
>>
>> What do you think?
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>>
>>> You can already do this by the readLockTimeout option. See more details
>>> here:
>>> http://camel.apache.org/file2.html
>>>
>>> For example you can set it to 10000 as 10 seconds and if Camel cannot
>>> acquire the lock within that period it will skip this file and try the
>>> next one.
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25851978.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25853488.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component blocked by existing files
Posted by Vladimir Okhotnikov <vo...@gmail.com>.
I closed the JIRA ticket I added initially.
Claus Ibsen-2 wrote:
>
> On Mon, Oct 12, 2009 at 10:15 AM, Vladimir Okhotnikov
> <vo...@gmail.com> wrote:
>>
>> Yes, I known about the readLockTimeout option. The problem with this
>> option
>> is that by a) default it waits forever and b) while it waits for one file
>> it
>> does not process others. That's what made me spend 2 days wondering about
>> the behavior described in the thread almost without clues, and it will
>> made
>> others.
>
> You did not mention that you knew this option and why have you not
> tried using it?
>
> Camel does TRACE / DEBUG logging which you could have enabled.
>
> Which is also written on the file component wiki page
> http://camel.apache.org/file2.html
>
> Debug logging
> This component has log level TRACE that can be helpful if you have
> problems.
>
> So if you have done that you would NOT have been in the dark for 2 days.
>
>
>
> The file consumer is NOT concurrent. So yes it will wait.
>
> If you want concurrency then define multiple routes with a file
> consumer. Which btw is not fun as file system is not transactional and
> having multiple consumers racing for the same files is not fun.
>
>
> Good idea about the default timeout which I have created a ticket for
> https://issues.apache.org/activemq/browse/CAMEL-2062
>
>
>
>>
>> As is pretty well explained in the "Release It" book, infinite waits are
>> bad. Infinite waits by default are very bad, especially in integration
>> software - it virtually guarantees that at least one will slip into your
>> live system, then it's just a matter of time until it blocks critical
>> function or critical amount of resources - and you're waken in the middle
>> of
>> the night to deal with it.
>>
>> What do you think?
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>>
>>> You can already do this by the readLockTimeout option. See more details
>>> here:
>>> http://camel.apache.org/file2.html
>>>
>>> For example you can set it to 10000 as 10 seconds and if Camel cannot
>>> acquire the lock within that period it will skip this file and try the
>>> next one.
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25851978.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25853796.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component blocked by existing files
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Oct 12, 2009 at 10:15 AM, Vladimir Okhotnikov
<vo...@gmail.com> wrote:
>
> Yes, I known about the readLockTimeout option. The problem with this option
> is that by a) default it waits forever and b) while it waits for one file it
> does not process others. That's what made me spend 2 days wondering about
> the behavior described in the thread almost without clues, and it will made
> others.
You did not mention that you knew this option and why have you not
tried using it?
Camel does TRACE / DEBUG logging which you could have enabled.
Which is also written on the file component wiki page
http://camel.apache.org/file2.html
Debug logging
This component has log level TRACE that can be helpful if you have problems.
So if you have done that you would NOT have been in the dark for 2 days.
The file consumer is NOT concurrent. So yes it will wait.
If you want concurrency then define multiple routes with a file
consumer. Which btw is not fun as file system is not transactional and
having multiple consumers racing for the same files is not fun.
Good idea about the default timeout which I have created a ticket for
https://issues.apache.org/activemq/browse/CAMEL-2062
>
> As is pretty well explained in the "Release It" book, infinite waits are
> bad. Infinite waits by default are very bad, especially in integration
> software - it virtually guarantees that at least one will slip into your
> live system, then it's just a matter of time until it blocks critical
> function or critical amount of resources - and you're waken in the middle of
> the night to deal with it.
>
> What do you think?
>
>
>
> Claus Ibsen-2 wrote:
>>
>>
>> You can already do this by the readLockTimeout option. See more details
>> here:
>> http://camel.apache.org/file2.html
>>
>> For example you can set it to 10000 as 10 seconds and if Camel cannot
>> acquire the lock within that period it will skip this file and try the
>> next one.
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25851978.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File component blocked by existing files
Posted by Vladimir Okhotnikov <vo...@gmail.com>.
Yes, I known about the readLockTimeout option. The problem with this option
is that by a) default it waits forever and b) while it waits for one file it
does not process others. That's what made me spend 2 days wondering about
the behavior described in the thread almost without clues, and it will made
others.
As is pretty well explained in the "Release It" book, infinite waits are
bad. Infinite waits by default are very bad, especially in integration
software - it virtually guarantees that at least one will slip into your
live system, then it's just a matter of time until it blocks critical
function or critical amount of resources - and you're waken in the middle of
the night to deal with it.
What do you think?
Claus Ibsen-2 wrote:
>
>
> You can already do this by the readLockTimeout option. See more details
> here:
> http://camel.apache.org/file2.html
>
> For example you can set it to 10000 as 10 seconds and if Camel cannot
> acquire the lock within that period it will skip this file and try the
> next one.
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25851978.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component blocked by existing files
Posted by Claus Ibsen <cl...@gmail.com>.
On Sat, Oct 10, 2009 at 1:05 PM, Vladimir Okhotnikov
<vo...@gmail.com> wrote:
>
> Got it! The bitness (and probably OS flavor) is not relevant - the problem is
> because the service account does not have "Modify" permission on the
> existing file. I strongly suspect that it cannot rename the offending file
> to satisfy readLock=rename and hangs forever. When I manually gave the
> "Modify" permission on one of the offending files, the corresponding
> consumer worked after service start.
>
> Nevertherless, I think it is still a bug - having such a poison file should
> not prevent the consumer from processing other files. At the very least, an
> indication of the problem in the log and skipping the problematic file are
> needed.
>
> What do you think?
>
You can already do this by the readLockTimeout option. See more details here:
http://camel.apache.org/file2.html
For example you can set it to 10000 as 10 seconds and if Camel cannot
acquire the lock within that period it will skip this file and try the
next one.
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> It should work, at least that is the goal.
>>
>> Is it always the same consumer that does not work?
>> Can you try with only 2 and 1 to see if the issue is there as well?
>>
>> And can you create a ticket in JIRA about this? I will get time next
>> week to take a peak, but would appreciate if you had time to track
>> down more details where the problem may lie.
>> BTW Which OS do you use?
>>
>>
>> On Thu, Oct 8, 2009 at 9:41 PM, Vladimir Okhotnikov
>> <vo...@gmail.com> wrote:
>>>
>>> Sorry,
>>>
>>> Camel 2.0 release
>>> 3 routes like:
>>> file:///home/javadev/workspace/camelroutes/testdir/testdata/testimport?delete=true&include=%5E.%2Bmapping_warnings%5C.txt%24&readLock=rename
>>>
>>> the only difference between all 3 is the include pattern.
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> Hi
>>>>
>>>> Always state which version of Camel you are using.
>>>> And post how you route is configured.
>>>>
>>>>
>>>> On Thu, Oct 8, 2009 at 4:57 PM, Vladimir Okhotnikov
>>>> <vo...@gmail.com> wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> I'm observing quite a strange behavior at the moment. In my app, I have
>>>>> several routes which all start by reading files in the same directory,
>>>>> deleting them after processing. Obviously, each rule uses its own
>>>>> include
>>>>> pattern to process separate subset of files.
>>>>>
>>>>> Now, the problem is that if the camel context is started when there are
>>>>> some
>>>>> existing files in the directory, the matching routes do not seem to
>>>>> work.
>>>>> I
>>>>> cannot see any related activity in the log file (except for the first
>>>>> "1
>>>>> file to consume message"), and the newly added files are being piled up
>>>>> in
>>>>> the directory. The other routes, which had no matching files on camel
>>>>> start
>>>>> do work.
>>>>>
>>>>> Is it a bug or am I just missing something?
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25803233.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25806158.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25833119.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File component blocked by existing files
Posted by Vladimir Okhotnikov <vo...@gmail.com>.
Got it! The bitness (and probably OS flavor) is not relevant - the problem is
because the service account does not have "Modify" permission on the
existing file. I strongly suspect that it cannot rename the offending file
to satisfy readLock=rename and hangs forever. When I manually gave the
"Modify" permission on one of the offending files, the corresponding
consumer worked after service start.
Nevertherless, I think it is still a bug - having such a poison file should
not prevent the consumer from processing other files. At the very least, an
indication of the problem in the log and skipping the problematic file are
needed.
What do you think?
Claus Ibsen-2 wrote:
>
> Hi
>
> It should work, at least that is the goal.
>
> Is it always the same consumer that does not work?
> Can you try with only 2 and 1 to see if the issue is there as well?
>
> And can you create a ticket in JIRA about this? I will get time next
> week to take a peak, but would appreciate if you had time to track
> down more details where the problem may lie.
> BTW Which OS do you use?
>
>
> On Thu, Oct 8, 2009 at 9:41 PM, Vladimir Okhotnikov
> <vo...@gmail.com> wrote:
>>
>> Sorry,
>>
>> Camel 2.0 release
>> 3 routes like:
>> file:///home/javadev/workspace/camelroutes/testdir/testdata/testimport?delete=true&include=%5E.%2Bmapping_warnings%5C.txt%24&readLock=rename
>>
>> the only difference between all 3 is the include pattern.
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> Always state which version of Camel you are using.
>>> And post how you route is configured.
>>>
>>>
>>> On Thu, Oct 8, 2009 at 4:57 PM, Vladimir Okhotnikov
>>> <vo...@gmail.com> wrote:
>>>>
>>>> Hi
>>>>
>>>> I'm observing quite a strange behavior at the moment. In my app, I have
>>>> several routes which all start by reading files in the same directory,
>>>> deleting them after processing. Obviously, each rule uses its own
>>>> include
>>>> pattern to process separate subset of files.
>>>>
>>>> Now, the problem is that if the camel context is started when there are
>>>> some
>>>> existing files in the directory, the matching routes do not seem to
>>>> work.
>>>> I
>>>> cannot see any related activity in the log file (except for the first
>>>> "1
>>>> file to consume message"), and the newly added files are being piled up
>>>> in
>>>> the directory. The other routes, which had no matching files on camel
>>>> start
>>>> do work.
>>>>
>>>> Is it a bug or am I just missing something?
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25803233.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25806158.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25833119.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component blocked by existing files
Posted by Vladimir Okhotnikov <vo...@gmail.com>.
Turned out that the OS question is the most important one. The effect happens
at Windows 2003 x64, which is our live environment. So far I was not able to
reproduce it on WinXP 32 VM. Sadly, I think I have no available Win2003x64
machines I could use for java development/debugging.
To complicate things more - the live routes app is deployed as a windows
service, using the 'wrapper' application as packaged by the appassembler
maven plugin. As such, it actually runs the camel context in a 32-bit java
VM there:
Java Executable: C:\WINDOWS\system32\java.exe
Windows version: 5.2.3790
Java Version : 1.6.0_11-b03 Java HotSpot(TM) Client VM
Java VM Vendor : Sun Microsystems Inc.
I think I'll try putting together a minimum sample which illustrates the
error. Judging by the absence of any messages in the log relevant to those
routes, the culprit likely happens in the FileComponent itself.
As for other question: I tried restarting the camel context with different
existing files in the pickup directory, and the rule is just that - if there
is a matching file for a consumer on start - it hangs. If there are no
matching files on start - the consumer works.
I've added https://issues.apache.org/activemq/browse/CAMEL-2057 with "Minor"
priority.
Claus Ibsen-2 wrote:
>
> Hi
>
> It should work, at least that is the goal.
>
> Is it always the same consumer that does not work?
> Can you try with only 2 and 1 to see if the issue is there as well?
>
> And can you create a ticket in JIRA about this? I will get time next
> week to take a peak, but would appreciate if you had time to track
> down more details where the problem may lie.
> BTW Which OS do you use?
>
>
> On Thu, Oct 8, 2009 at 9:41 PM, Vladimir Okhotnikov
> <vo...@gmail.com> wrote:
>>
>> Sorry,
>>
>> Camel 2.0 release
>> 3 routes like:
>> file:///home/javadev/workspace/camelroutes/testdir/testdata/testimport?delete=true&include=%5E.%2Bmapping_warnings%5C.txt%24&readLock=rename
>>
>> the only difference between all 3 is the include pattern.
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> Always state which version of Camel you are using.
>>> And post how you route is configured.
>>>
>>>
>>> On Thu, Oct 8, 2009 at 4:57 PM, Vladimir Okhotnikov
>>> <vo...@gmail.com> wrote:
>>>>
>>>> Hi
>>>>
>>>> I'm observing quite a strange behavior at the moment. In my app, I have
>>>> several routes which all start by reading files in the same directory,
>>>> deleting them after processing. Obviously, each rule uses its own
>>>> include
>>>> pattern to process separate subset of files.
>>>>
>>>> Now, the problem is that if the camel context is started when there are
>>>> some
>>>> existing files in the directory, the matching routes do not seem to
>>>> work.
>>>> I
>>>> cannot see any related activity in the log file (except for the first
>>>> "1
>>>> file to consume message"), and the newly added files are being piled up
>>>> in
>>>> the directory. The other routes, which had no matching files on camel
>>>> start
>>>> do work.
>>>>
>>>> Is it a bug or am I just missing something?
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25803233.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25806158.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25825859.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component blocked by existing files
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
It should work, at least that is the goal.
Is it always the same consumer that does not work?
Can you try with only 2 and 1 to see if the issue is there as well?
And can you create a ticket in JIRA about this? I will get time next
week to take a peak, but would appreciate if you had time to track
down more details where the problem may lie.
BTW Which OS do you use?
On Thu, Oct 8, 2009 at 9:41 PM, Vladimir Okhotnikov
<vo...@gmail.com> wrote:
>
> Sorry,
>
> Camel 2.0 release
> 3 routes like:
> file:///home/javadev/workspace/camelroutes/testdir/testdata/testimport?delete=true&include=%5E.%2Bmapping_warnings%5C.txt%24&readLock=rename
>
> the only difference between all 3 is the include pattern.
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Always state which version of Camel you are using.
>> And post how you route is configured.
>>
>>
>> On Thu, Oct 8, 2009 at 4:57 PM, Vladimir Okhotnikov
>> <vo...@gmail.com> wrote:
>>>
>>> Hi
>>>
>>> I'm observing quite a strange behavior at the moment. In my app, I have
>>> several routes which all start by reading files in the same directory,
>>> deleting them after processing. Obviously, each rule uses its own include
>>> pattern to process separate subset of files.
>>>
>>> Now, the problem is that if the camel context is started when there are
>>> some
>>> existing files in the directory, the matching routes do not seem to work.
>>> I
>>> cannot see any related activity in the log file (except for the first "1
>>> file to consume message"), and the newly added files are being piled up
>>> in
>>> the directory. The other routes, which had no matching files on camel
>>> start
>>> do work.
>>>
>>> Is it a bug or am I just missing something?
>>> --
>>> View this message in context:
>>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25803233.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25806158.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File component blocked by existing files
Posted by Vladimir Okhotnikov <vo...@gmail.com>.
Sorry,
Camel 2.0 release
3 routes like:
file:///home/javadev/workspace/camelroutes/testdir/testdata/testimport?delete=true&include=%5E.%2Bmapping_warnings%5C.txt%24&readLock=rename
the only difference between all 3 is the include pattern.
Claus Ibsen-2 wrote:
>
> Hi
>
> Always state which version of Camel you are using.
> And post how you route is configured.
>
>
> On Thu, Oct 8, 2009 at 4:57 PM, Vladimir Okhotnikov
> <vo...@gmail.com> wrote:
>>
>> Hi
>>
>> I'm observing quite a strange behavior at the moment. In my app, I have
>> several routes which all start by reading files in the same directory,
>> deleting them after processing. Obviously, each rule uses its own include
>> pattern to process separate subset of files.
>>
>> Now, the problem is that if the camel context is started when there are
>> some
>> existing files in the directory, the matching routes do not seem to work.
>> I
>> cannot see any related activity in the log file (except for the first "1
>> file to consume message"), and the newly added files are being piled up
>> in
>> the directory. The other routes, which had no matching files on camel
>> start
>> do work.
>>
>> Is it a bug or am I just missing something?
>> --
>> View this message in context:
>> http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25803233.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25806158.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component blocked by existing files
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Always state which version of Camel you are using.
And post how you route is configured.
On Thu, Oct 8, 2009 at 4:57 PM, Vladimir Okhotnikov
<vo...@gmail.com> wrote:
>
> Hi
>
> I'm observing quite a strange behavior at the moment. In my app, I have
> several routes which all start by reading files in the same directory,
> deleting them after processing. Obviously, each rule uses its own include
> pattern to process separate subset of files.
>
> Now, the problem is that if the camel context is started when there are some
> existing files in the directory, the matching routes do not seem to work. I
> cannot see any related activity in the log file (except for the first "1
> file to consume message"), and the newly added files are being piled up in
> the directory. The other routes, which had no matching files on camel start
> do work.
>
> Is it a bug or am I just missing something?
> --
> View this message in context: http://www.nabble.com/File-component-blocked-by-existing-files-tp25803233p25803233.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus