You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Aida <ai...@gmail.com> on 2013/09/20 13:55:53 UTC
[ConsumerTemplate] CamelLocks deleted before file is completely
processed
Hi,
I'm using the consumer template for reading files, and I'm experiencing
something that makes me think that I'm not using it correctly. What happens
to me is:
1. I have a dir with multiple files
2. The consumer template reads one of those files, creating a camelLock file
for the file that is going to be processed
3. The consumer template reads another file (same route, different thread)
and, since the camelLock of the previous file still exists, deletes it and
creates the camelLock for the file that is going to be processed now (the
previous file haven´t ended its processing and haven´t been "finished" using
the doneUnitOfWork)
4. The same for each file that is read (the camelLocks are removed and a
WARNING appears in the log " Deleting orphaned lock file: .../..
.camelLock")
I think that maybe this is because I'm creating a new consumerTemplate each
time I have to read something (could it be?), because I have seen that it
has a ConsumerCache.
The problem is that I use some "rules" to use it (one of them creating the
CT each time), because of some problems I have had in the past:
- Don´t use the disconnect option for FTPs (Nullpointer, at least in Camel
2.9.0 that leaves threads alive, Camel post in [1])
- Create a new Consumer template each time I read a file (same reasons as
previous sentence) and stop it after use
- I synchronize the method that consumes an endpoint because otherwise, when
reading files in parallel instead of reading two files, the consumer reads
one twice
In case it can helps, I put my code here:
ConsumerTemplate cTemplate = camelContext.createConsumerTemplate();
Exchange dataExchange = null;
if(timeoutInMilis == 0){
dataExchange = cTemplate.receiveNoWait(endpoint2Consume);
}
else{
dataExchange = cTemplate.receive(endpoint2Consume, timeoutInMilis);
}
try {
cTemplate.stop();
} catch (Exception e) {
log ...
}
Thanks in advance
Camel Version: 2.10.4
[1]
http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-td5548338.html#a5723551
--
View this message in context: http://camel.465427.n5.nabble.com/ConsumerTemplate-CamelLocks-deleted-before-file-is-completely-processed-tp5739870.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [ConsumerTemplate] CamelLocks deleted before file is completely
processed
Posted by Aida <ai...@gmail.com>.
Hi,
I have been investigating about this issue, because using the same
ConsumerTemplate doesn´t seem to fix it.
Just in case it can help somebody, I have seen that the
MarkerFileExclusiveReadLockStrategy#deleteLockFiles method is called once
per endpoint. So if two endpoints differ only in the "fileName" parameter
(but both files are in the same dir) this method is called and the
.camelLocks are removed.
If somebody has this problem he or she could try using a GenericFileFilter
implementation to filter by fileName. I haven´t tried it because this
approach has problems consuming files in parallel as I will comment in [1]
in some minutes. But if there is no need of concurrency, give it a try.
BR.
[1]
http://camel.465427.n5.nabble.com/ConsumerTemplate-Concurrency-problems-in-Camel-2-11-2-attached-test-td5740295.html
Claus Ibsen-2 wrote
> Hi
>
> Yes create the template once and reuse it. See this page it also
> applies to consumer template
> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>
> On Fri, Sep 20, 2013 at 1:55 PM, Aida <
> ai.desu@
> > wrote:
>> Hi,
>>
>> I'm using the consumer template for reading files, and I'm experiencing
>> something that makes me think that I'm not using it correctly. What
>> happens
>> to me is:
>>
>> 1. I have a dir with multiple files
>> 2. The consumer template reads one of those files, creating a camelLock
>> file
>> for the file that is going to be processed
>> 3. The consumer template reads another file (same route, different
>> thread)
>> and, since the camelLock of the previous file still exists, deletes it
>> and
>> creates the camelLock for the file that is going to be processed now (the
>> previous file haven´t ended its processing and haven´t been "finished"
>> using
>> the doneUnitOfWork)
>> 4. The same for each file that is read (the camelLocks are removed and a
>> WARNING appears in the log " Deleting orphaned lock file: .../..
>> .camelLock")
>>
>> I think that maybe this is because I'm creating a new consumerTemplate
>> each
>> time I have to read something (could it be?), because I have seen that it
>> has a ConsumerCache.
>>
>> The problem is that I use some "rules" to use it (one of them creating
>> the
>> CT each time), because of some problems I have had in the past:
>>
>> - Don´t use the disconnect option for FTPs (Nullpointer, at least in
>> Camel
>> 2.9.0 that leaves threads alive, Camel post in [1])
>> - Create a new Consumer template each time I read a file (same reasons as
>> previous sentence) and stop it after use
>> - I synchronize the method that consumes an endpoint because otherwise,
>> when
>> reading files in parallel instead of reading two files, the consumer
>> reads
>> one twice
>>
>>
>> In case it can helps, I put my code here:
>>
>> ConsumerTemplate cTemplate =
>> camelContext.createConsumerTemplate();
>> Exchange dataExchange = null;
>> if(timeoutInMilis == 0){
>> dataExchange = cTemplate.receiveNoWait(endpoint2Consume);
>> }
>> else{
>> dataExchange = cTemplate.receive(endpoint2Consume,
>> timeoutInMilis);
>> }
>> try {
>> cTemplate.stop();
>> } catch (Exception e) {
>> log ...
>> }
>>
>>
>> Thanks in advance
>>
>> Camel Version: 2.10.4
>> [1]
>> http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-td5548338.html#a5723551
>>
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/ConsumerTemplate-CamelLocks-deleted-before-file-is-completely-processed-tp5739870.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email:
> cibsen@
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
--
View this message in context: http://camel.465427.n5.nabble.com/ConsumerTemplate-CamelLocks-deleted-before-file-is-completely-processed-tp5739870p5740681.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [ConsumerTemplate] CamelLocks deleted before file is completely
processed
Posted by Aida <ai...@gmail.com>.
Hi Claus,
Once I have read and understood the link you indicated, I have two doubts:
1. If the idea is to create one consumerTemplate at the begginig and reuse
it, and stop it after use ... What happens if two processes (different
processes) use the template at the same time? I mean, if one finishes using
it and stops it, it would close down all the resources (its own resources
and the resources of the other process if it hasn´t finished using the
template I suppose).
2. I have seen in the 2.12.X docs ([1]) that the ConsumerTemplate is
threadsafe, but I'm using Camel 2.10.4 and I don´t see something similar in
2.10 API docs ([2]). If the idea is to reuse it, I guess I could have
problems with this (in the past I found that reading different files in
parallel didn´t work well and I had to synchronize the template usage)
Thanks in advance,
Aida.
[1]
http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/ConsumerTemplate.html
[2]
http://camel.apache.org/maven/camel-2.10.0/camel-core/apidocs/org/apache/camel/ConsumerTemplate.html
Claus Ibsen-2 wrote
> Hi
>
> Yes create the template once and reuse it. See this page it also
> applies to consumer template
> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>
> On Fri, Sep 20, 2013 at 1:55 PM, Aida <
> ai.desu@
> > wrote:
>> Hi,
>>
>> I'm using the consumer template for reading files, and I'm experiencing
>> something that makes me think that I'm not using it correctly. What
>> happens
>> to me is:
>>
>> 1. I have a dir with multiple files
>> 2. The consumer template reads one of those files, creating a camelLock
>> file
>> for the file that is going to be processed
>> 3. The consumer template reads another file (same route, different
>> thread)
>> and, since the camelLock of the previous file still exists, deletes it
>> and
>> creates the camelLock for the file that is going to be processed now (the
>> previous file haven´t ended its processing and haven´t been "finished"
>> using
>> the doneUnitOfWork)
>> 4. The same for each file that is read (the camelLocks are removed and a
>> WARNING appears in the log " Deleting orphaned lock file: .../..
>> .camelLock")
>>
>> I think that maybe this is because I'm creating a new consumerTemplate
>> each
>> time I have to read something (could it be?), because I have seen that it
>> has a ConsumerCache.
>>
>> The problem is that I use some "rules" to use it (one of them creating
>> the
>> CT each time), because of some problems I have had in the past:
>>
>> - Don´t use the disconnect option for FTPs (Nullpointer, at least in
>> Camel
>> 2.9.0 that leaves threads alive, Camel post in [1])
>> - Create a new Consumer template each time I read a file (same reasons as
>> previous sentence) and stop it after use
>> - I synchronize the method that consumes an endpoint because otherwise,
>> when
>> reading files in parallel instead of reading two files, the consumer
>> reads
>> one twice
>>
>>
>> In case it can helps, I put my code here:
>>
>> ConsumerTemplate cTemplate =
>> camelContext.createConsumerTemplate();
>> Exchange dataExchange = null;
>> if(timeoutInMilis == 0){
>> dataExchange = cTemplate.receiveNoWait(endpoint2Consume);
>> }
>> else{
>> dataExchange = cTemplate.receive(endpoint2Consume,
>> timeoutInMilis);
>> }
>> try {
>> cTemplate.stop();
>> } catch (Exception e) {
>> log ...
>> }
>>
>>
>> Thanks in advance
>>
>> Camel Version: 2.10.4
>> [1]
>> http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-td5548338.html#a5723551
>>
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/ConsumerTemplate-CamelLocks-deleted-before-file-is-completely-processed-tp5739870.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email:
> cibsen@
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
--
View this message in context: http://camel.465427.n5.nabble.com/ConsumerTemplate-CamelLocks-deleted-before-file-is-completely-processed-tp5739870p5740048.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [ConsumerTemplate] CamelLocks deleted before file is completely processed
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yes create the template once and reuse it. See this page it also
applies to consumer template
http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
On Fri, Sep 20, 2013 at 1:55 PM, Aida <ai...@gmail.com> wrote:
> Hi,
>
> I'm using the consumer template for reading files, and I'm experiencing
> something that makes me think that I'm not using it correctly. What happens
> to me is:
>
> 1. I have a dir with multiple files
> 2. The consumer template reads one of those files, creating a camelLock file
> for the file that is going to be processed
> 3. The consumer template reads another file (same route, different thread)
> and, since the camelLock of the previous file still exists, deletes it and
> creates the camelLock for the file that is going to be processed now (the
> previous file haven´t ended its processing and haven´t been "finished" using
> the doneUnitOfWork)
> 4. The same for each file that is read (the camelLocks are removed and a
> WARNING appears in the log " Deleting orphaned lock file: .../..
> .camelLock")
>
> I think that maybe this is because I'm creating a new consumerTemplate each
> time I have to read something (could it be?), because I have seen that it
> has a ConsumerCache.
>
> The problem is that I use some "rules" to use it (one of them creating the
> CT each time), because of some problems I have had in the past:
>
> - Don´t use the disconnect option for FTPs (Nullpointer, at least in Camel
> 2.9.0 that leaves threads alive, Camel post in [1])
> - Create a new Consumer template each time I read a file (same reasons as
> previous sentence) and stop it after use
> - I synchronize the method that consumes an endpoint because otherwise, when
> reading files in parallel instead of reading two files, the consumer reads
> one twice
>
>
> In case it can helps, I put my code here:
>
> ConsumerTemplate cTemplate = camelContext.createConsumerTemplate();
> Exchange dataExchange = null;
> if(timeoutInMilis == 0){
> dataExchange = cTemplate.receiveNoWait(endpoint2Consume);
> }
> else{
> dataExchange = cTemplate.receive(endpoint2Consume, timeoutInMilis);
> }
> try {
> cTemplate.stop();
> } catch (Exception e) {
> log ...
> }
>
>
> Thanks in advance
>
> Camel Version: 2.10.4
> [1]
> http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-td5548338.html#a5723551
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ConsumerTemplate-CamelLocks-deleted-before-file-is-completely-processed-tp5739870.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen