You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Piotr Jagielski <pj...@o2.pl> on 2008/04/10 22:54:47 UTC
Aggregator/resequencer reliability
Hi all,
I'm not sure whether stateful processors e.g. aggregator or resequencer are
persistent and reliable in case of system crash.
I'm using Camel inside Servicemix. When i send some messages which are
passed through aggregator and resequences i notice that enclosing JMS
session is closed and JMS message is commited.
Anyone can help?
There has been a post already but with no answer:
http://www.nabble.com/Better-Aggregator-support-td12564277s22882.html#a12564277
Regards,
Piotr
--
View this message in context: http://www.nabble.com/Aggregator-resequencer-reliability-tp16617652s22882p16617652.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregator/resequencer reliability
Posted by Piotr Jagielski <pj...@o2.pl>.
I did some tests but still can't find correct solution...
While aggregator/resequencer are asynchronous, configuring transactions
didn't help - transaction is commited after invoking process() but messages
are processed later, after batchTimeout.
So I tried to change resequencer to process synchronously but there is a
problem with concurrent consuming on transacted JMS component - multiple
concurrent consumers cause high CPU usage and still some messages are queued
before aggregating.
Any further ideas?
Piotr
RomKal wrote:
>
> 2008/4/11, Piotr Jagielski <pj...@o2.pl>:
>> I've found an issue reported already :
>> https://issues.apache.org/activemq/browse/CAMEL-217
>>
>> What do you think of blocking an exchange by aggregator processor as in
>> delayer processor?
>> Processing could be continued after aggregating
>
> So this way we don't commit transactions associated with messages
> unless they are aggregated?
>
> I'm not really sure if it could work, but there are people who have
> better experience here.Maybe we could store AsyncCallbacks in
> aggregator and asynchronously finish transactions when aggregator
> finishes aggregation.
>
> The only problem here is that many our endpoints doesn't support
> asynchronous invocations, so it would anyway block threads (JMS
> Consumers), I believe.
>
> BTW Maybe aggregator should have such a flag that requires waiting at
> aggregator unless things get aggregated?
>
> Roman
>
>
--
View this message in context: http://www.nabble.com/Aggregator-resequencer-reliability-tp16617652s22882p16677428.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregator/resequencer reliability
Posted by Roman Kalukiewicz <ro...@gmail.com>.
2008/4/11, Piotr Jagielski <pj...@o2.pl>:
> I've found an issue reported already :
> https://issues.apache.org/activemq/browse/CAMEL-217
>
> What do you think of blocking an exchange by aggregator processor as in
> delayer processor?
> Processing could be continued after aggregating
So this way we don't commit transactions associated with messages
unless they are aggregated?
I'm not really sure if it could work, but there are people who have
better experience here.Maybe we could store AsyncCallbacks in
aggregator and asynchronously finish transactions when aggregator
finishes aggregation.
The only problem here is that many our endpoints doesn't support
asynchronous invocations, so it would anyway block threads (JMS
Consumers), I believe.
BTW Maybe aggregator should have such a flag that requires waiting at
aggregator unless things get aggregated?
Roman
Re: Aggregator/resequencer reliability
Posted by Piotr Jagielski <pj...@o2.pl>.
I've found an issue reported already :
https://issues.apache.org/activemq/browse/CAMEL-217
What do you think of blocking an exchange by aggregator processor as in
delayer processor?
Processing could be continued after aggregating
Piotr
RomKal wrote:
>
> 2008/4/10, Piotr Jagielski <pj...@o2.pl>:
>> I'm not sure whether stateful processors e.g. aggregator or resequencer
>> are
>> persistent and reliable in case of system crash.
>> I'm using Camel inside Servicemix. When i send some messages which are
>> passed through aggregator and resequences i notice that enclosing JMS
>> session is closed and JMS message is commited.
>> Anyone can help?
>
> Unfortunately current aggregator and resequencer patterns are not
> persisten and reliable.
> Would you like to create JIRA issue for this to track this requirement?
>
>> There has been a post already but with no answer:
>>
>> http://www.nabble.com/Better-Aggregator-support-td12564277s22882.html#a12564277
>
> In fact this post asks two different questions - one for persistence
> and second one for 'completedPredicate()' that is already implemented.
>
> Roman
>
>
--
View this message in context: http://www.nabble.com/Aggregator-resequencer-reliability-tp16617652s22882p16627875.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregator/resequencer reliability
Posted by Piotr Jagielski <pj...@o2.pl>.
So, when I configure JmsComponent to use transactions as in
http://activemq.apache.org/camel/transactional-client.html and then
configure route:
from("activemq:queue:a")..aggregator(xpath("/msg/id").stringResult()).batchSize(10).batchTimeout(100000)
....
or
from("activemq:queue:a").resequencer(xpath("//priority").numberResult()).stream(new
StreamResequencerConfig(10, 10000))
...
In case of system failure during 100000 batch timeout, after restart
messages will be processed again? That's what i'm looking for..
Regards
James.Strachan wrote:
>
> There's 2 ways to do a reliable resequencer/aggregators. One is to use
> persistence and XA; the other is to use batching and transactions
> (i.e. on a failure, the transaction rolls back and the entire batch is
> replayed). The latter was the first option implemented; using
> transaction batches is less flexible - but it does avoid XA.
>
> We could maybe still use persistence and avoid XA...
> http://activemq.apache.org/should-i-use-xa.html
>
> using the Idempotent Consumer to avoid duplicates?
> http://activemq.apache.org/camel/idempotent-consumer.html
>
> On 11/04/2008, Roman Kalukiewicz <ro...@gmail.com> wrote:
>> 2008/4/10, Piotr Jagielski <pj...@o2.pl>:
>>
>> > I'm not sure whether stateful processors e.g. aggregator or
>> resequencer are
>> > persistent and reliable in case of system crash.
>> > I'm using Camel inside Servicemix. When i send some messages which
>> are
>> > passed through aggregator and resequences i notice that enclosing JMS
>> > session is closed and JMS message is commited.
>> > Anyone can help?
>>
>>
>> Unfortunately current aggregator and resequencer patterns are not
>> persisten and reliable.
>> Would you like to create JIRA issue for this to track this requirement?
>>
>>
>> > There has been a post already but with no answer:
>> >
>> http://www.nabble.com/Better-Aggregator-support-td12564277s22882.html#a12564277
>>
>>
>> In fact this post asks two different questions - one for persistence
>> and second one for 'completedPredicate()' that is already implemented.
>>
>>
>> Roman
>>
>
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://open.iona.com
>
>
--
View this message in context: http://www.nabble.com/Aggregator-resequencer-reliability-tp16617652s22882p16648595.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Aggregator/resequencer reliability
Posted by James Strachan <ja...@gmail.com>.
There's 2 ways to do a reliable resequencer/aggregators. One is to use
persistence and XA; the other is to use batching and transactions
(i.e. on a failure, the transaction rolls back and the entire batch is
replayed). The latter was the first option implemented; using
transaction batches is less flexible - but it does avoid XA.
We could maybe still use persistence and avoid XA...
http://activemq.apache.org/should-i-use-xa.html
using the Idempotent Consumer to avoid duplicates?
http://activemq.apache.org/camel/idempotent-consumer.html
On 11/04/2008, Roman Kalukiewicz <ro...@gmail.com> wrote:
> 2008/4/10, Piotr Jagielski <pj...@o2.pl>:
>
> > I'm not sure whether stateful processors e.g. aggregator or resequencer are
> > persistent and reliable in case of system crash.
> > I'm using Camel inside Servicemix. When i send some messages which are
> > passed through aggregator and resequences i notice that enclosing JMS
> > session is closed and JMS message is commited.
> > Anyone can help?
>
>
> Unfortunately current aggregator and resequencer patterns are not
> persisten and reliable.
> Would you like to create JIRA issue for this to track this requirement?
>
>
> > There has been a post already but with no answer:
> > http://www.nabble.com/Better-Aggregator-support-td12564277s22882.html#a12564277
>
>
> In fact this post asks two different questions - one for persistence
> and second one for 'completedPredicate()' that is already implemented.
>
>
> Roman
>
--
James
-------
http://macstrac.blogspot.com/
Open Source Integration
http://open.iona.com
Re: Aggregator/resequencer reliability
Posted by Roman Kalukiewicz <ro...@gmail.com>.
2008/4/10, Piotr Jagielski <pj...@o2.pl>:
> I'm not sure whether stateful processors e.g. aggregator or resequencer are
> persistent and reliable in case of system crash.
> I'm using Camel inside Servicemix. When i send some messages which are
> passed through aggregator and resequences i notice that enclosing JMS
> session is closed and JMS message is commited.
> Anyone can help?
Unfortunately current aggregator and resequencer patterns are not
persisten and reliable.
Would you like to create JIRA issue for this to track this requirement?
> There has been a post already but with no answer:
> http://www.nabble.com/Better-Aggregator-support-td12564277s22882.html#a12564277
In fact this post asks two different questions - one for persistence
and second one for 'completedPredicate()' that is already implemented.
Roman