You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by fredbasset <na...@thepyes.com> on 2013/01/24 00:07:25 UTC
Possible bug with multicast shareUnitOfWork
Hi,
I think I may have found a bug with error handling in shareUnitOfWork
multicasts... unless I have fundamentally misunderstood something!
Starting from the camel-archetype-spring archetype and just changing the
context as follows
I would expect the dead letter log to happen once per input message, but
instead it happens once for the first message, and twice for any subsequent
messages (increasing the number of 'throws' in the multicast similarly
increases the dead letter logs.) Sample output from a run is below...
Taking the 'shareUnitOfWork' off causes the double dead-letter processing
for each input message, which is what I would expect. Enabling parallel
processing makes no difference.
Obviously this is massively cut down from the real-world scenario that is
causing me problems!
Issue seen in at least 2.9.2 -> 2.10.3
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Possible-bug-with-multicast-shareUnitOfWork-tp5726103.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Possible bug with multicast shareUnitOfWork
Posted by Willem Jiang <wi...@gmail.com>.
Oh, you are using camel-cxf.
Can you just check the log to see if sync invocation can be time out?
If you setup the timeout, I don’t think camel-cxf producer can block thread forever.
BTW, you can use some thread monitor tool to check what blocks the thread.
If you want to implement a timeout thread pool, you can take a look at this[1]
[1]http://stackoverflow.com/questions/2758612/executorservice-that-interrupts-tasks-after-a-timeout
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On July 8, 2014 at 9:29:12 PM, rubancsp (rubancse@gmail.com) wrote:
> Hi Willem
>
> Thanks for your reply,
>
> already we are configuring the time out changes in camel-cxf.xml as below
> before making the synch call(using worked thread).
>
> >
> name="{http://xml.comcast.com/commercialcustomer/services}CommercialCustomerServicePort.http-conduit">
> > ConnectionTimeout="${workbench.connector.ConnectionTimeout}" />
>
>
>
> But still we are getting the stuck thread issue, just wanted to know if
> there is any time out setting is available in customized thread pool which
> will terminate thread within the configurable amount of time and worked
> thread will be reused it for the next transaction,
>
> Thanks
> Ruban
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Possible-bug-with-multicast-shareUnitOfWork-tp5726103p5753545.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Possible bug with multicast shareUnitOfWork
Posted by rubancsp <ru...@gmail.com>.
Hi Willem
Thanks for your reply,
already we are configuring the time out changes in camel-cxf.xml as below
before making the synch call(using worked thread).
<http-conf:conduit
name="{http://xml.comcast.com/commercialcustomer/services}CommercialCustomerServicePort.http-conduit">
<http-conf:client ReceiveTimeout="${workbench.connector.RecieveTimeout}"
ConnectionTimeout="${workbench.connector.ConnectionTimeout}" />
</http-conf:conduit>
But still we are getting the stuck thread issue, just wanted to know if
there is any time out setting is available in customized thread pool which
will terminate thread within the configurable amount of time and worked
thread will be reused it for the next transaction,
Thanks
Ruban
--
View this message in context: http://camel.465427.n5.nabble.com/Possible-bug-with-multicast-shareUnitOfWork-tp5726103p5753545.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Possible bug with multicast shareUnitOfWork
Posted by Willem Jiang <wi...@gmail.com>.
Here are the answers for you questions.
Q1. The default camel thread pool is using CachedThreadPool which means it doesn’t set up the thread size.
Q2 If the calling thread is blocked, it still there, thread pool doesn’t reuse it.
That could explain why you got the OOM error.
If you setup the max size of thread pool, you may face the reject execution issue, as the thread in the thread pool may all blocked.
My suggestion you should add a timeout for the sync invocation, to avoid blocking the working thread forever.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On July 7, 2014 at 11:30:05 PM, rubancsp (rubancse@gmail.com) wrote:
> Corrected typo error
>
> Hi Willem
>
> We are getting the OOM memory error in our production environment due to the
> lot of stuck threads.
> while analyzed the code we found that one of the camel route which is using
> the mulitcast is the reason for the stuck thread.This is intermittent issue
> .
>
>
> Please find the below code snippet which we used for multi casting.(making
> the two parallel web service call)
>
> Camel version:2.10.3
>
> > loggingLevel="DEBUG">
> > parallelProcessing="true">
>
>
>
>
>
>
>
> Please find the below two logs:
>
> Logs which is causing the stuck thread:
> 30 Jun 2014 15:10:42,823 ********** [] [14.01] ERROR [Camel (camel-1) thread
> #7 - Multicast] invokeCentury - inside century route
>
> Log which is working fine:
>
> 30 Jun 2014 15:10:58,105 *************** [] [14.01] ERROR [Camel (camel-1)
> thread #2 - Multicast] invokeCentury - inside century route
> 30 Jun 2014 15:10:58,105 ************** [14.01] ERROR [Camel (camel-1)
> thread #5 - Multicast] invokeWBServices - inside wb route
>
>
> Here is my questions: 1)we are using the default camel thread pool for the
> making the two synchronous calls and not customizing the default thread
> pool.do we need to make any change in the above code to customize the thread
> pool setting(like min pool size max.pool size etc).
> 2) what will happen to worked thread if we did n't get any response from
> the external system . Thread pool terminate the worked thread or we need to
> handle it in our code.
>
> It would be great if you provide some suggestion on the above issue.
>
> Please let me know if you need any additional details.
>
> Thanks
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Possible-bug-with-multicast-shareUnitOfWork-tp5726103p5753468.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Possible bug with multicast shareUnitOfWork
Posted by rubancsp <ru...@gmail.com>.
Corrected typo error
Hi Willem
We are getting the OOM memory error in our production environment due to the
lot of stuck threads.
while analyzed the code we found that one of the camel route which is using
the mulitcast is the reason for the stuck thread.This is intermittent issue
.
Please find the below code snippet which we used for multi casting.(making
the two parallel web service call)
Camel version:2.10.3
<camel:log message="inside multicast route"
loggingLevel="DEBUG"></camel:log>
<camel:multicast strategyRef="aggregatedData"
parallelProcessing="true">
<to uri="direct:WB" />
<to uri="direct:CEN" />
</camel:multicast>
Please find the below two logs:
Logs which is causing the stuck thread:
30 Jun 2014 15:10:42,823 ********** [] [14.01] ERROR [Camel (camel-1) thread
#7 - Multicast] invokeCentury - inside century route
Log which is working fine:
30 Jun 2014 15:10:58,105 *************** [] [14.01] ERROR [Camel (camel-1)
thread #2 - Multicast] invokeCentury - inside century route
30 Jun 2014 15:10:58,105 ************** [14.01] ERROR [Camel (camel-1)
thread #5 - Multicast] invokeWBServices - inside wb route
Here is my questions: 1)we are using the default camel thread pool for the
making the two synchronous calls and not customizing the default thread
pool.do we need to make any change in the above code to customize the thread
pool setting(like min pool size max.pool size etc).
2) what will happen to worked thread if we did n't get any response from
the external system . Thread pool terminate the worked thread or we need to
handle it in our code.
It would be great if you provide some suggestion on the above issue.
Please let me know if you need any additional details.
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Possible-bug-with-multicast-shareUnitOfWork-tp5726103p5753468.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Possible bug with multicast shareUnitOfWork
Posted by fredbasset <na...@thepyes.com>.
Excellent... I look forward to downloading the fixed version!
Many thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/Possible-bug-with-multicast-shareUnitOfWork-tp5726103p5726182.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Possible bug with multicast shareUnitOfWork
Posted by Willem jiang <wi...@gmail.com>.
I think I found the issue, shareUnitOfWork should create the a new UnitOfWorkProcessor per processing instead of using the last one, as the parent UnitOfWork is different next time. When you using the shareUnitOfWork, the error handler should just get one call.
I will commit the fix after running a whole unit tests with my change.
--
Willem Jiang
Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On Thursday, January 24, 2013 at 3:21 PM, Willem.Jiang wrote:
> Hi,
>
> I can reproduce this kind of error in the trunk by adding a simple unit test
> as you said.
> If I remove the shareUnitOfWork setting, the error handler can get expect
> numbers messages.
> So I just fill a JIRA CAMEL-6005[1] for it.
>
> [1]https://issues.apache.org/jira/browse/CAMEL-6005
>
> Willem
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Possible-bug-with-multicast-shareUnitOfWork-tp5726103p5726113.html
> Sent from the Camel - Users mailing list archive at Nabble.com (http://Nabble.com).
Re: Possible bug with multicast shareUnitOfWork
Posted by "Willem.Jiang" <wi...@gmail.com>.
Hi,
I can reproduce this kind of error in the trunk by adding a simple unit test
as you said.
If I remove the shareUnitOfWork setting, the error handler can get expect
numbers messages.
So I just fill a JIRA CAMEL-6005[1] for it.
[1]https://issues.apache.org/jira/browse/CAMEL-6005
Willem
--
View this message in context: http://camel.465427.n5.nabble.com/Possible-bug-with-multicast-shareUnitOfWork-tp5726103p5726113.html
Sent from the Camel - Users mailing list archive at Nabble.com.