You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Vanshul.Chawla" <Va...@target.com> on 2015/07/15 09:06:06 UTC

Camel aggregation in case of server shutdown

Hello All,

We have a scenario wherein we aggregate message coming from a IBM MQ queue for 15 minutes before spitting a batch out to another MQ.
Suppose we have a server shutdown at 10th minute, what will happen to inflight messages? Will these be lost? Or will Fuse server restart get these messages and flush them out to destination.

Thanks and Regards,

Vanshul Chawla

RE: Camel aggregation in case of server shutdown

Posted by "Vanshul.Chawla" <Va...@target.com>.
Ok. Thanks. 

Thanks and Regards,

Vanshul Chawla


-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com] 
Sent: Wednesday, July 15, 2015 12:51 PM
To: users@camel.apache.org
Subject: Re: Camel aggregation in case of server shutdown

On Wed, Jul 15, 2015 at 9:15 AM, Vanshul.Chawla <Va...@target.com> wrote:
> Thanks Claus. What in case someone kills process pid of server? Ie force kill the server?
>

Then they are lost.

There is this component on the way in a future Camel release that may help
http://camel.apache.org/file2

> Thanks and Regards,
>
> Vanshul Chawla
>
>
> -----Original Message-----
> From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Sent: Wednesday, July 15, 2015 12:41 PM
> To: users@camel.apache.org
> Subject: Re: Camel aggregation in case of server shutdown
>
> How do you aggregate them? If you use the Camel aggregator then its by default in-memory only, and you would loose those messages. There is pluggable persistent stores, but they have their own pros/cons.
>
> Its better to stop the consumer that takes in new messages, and wait for the aggregator batch to complete so there is no inflight, and then stop.
>
> You can also enable forceCompletionOnStop on the aggregator then it flushes out all in-flight exchanges so you can stop quicker. In fact it may be the option you are looking for.
>
> On Wed, Jul 15, 2015 at 9:06 AM, Vanshul.Chawla <Va...@target.com> wrote:
>> Hello All,
>>
>> We have a scenario wherein we aggregate message coming from a IBM MQ queue for 15 minutes before spitting a batch out to another MQ.
>> Suppose we have a server shutdown at 10th minute, what will happen to inflight messages? Will these be lost? Or will Fuse server restart get these messages and flush them out to destination.
>>
>> Thanks and Regards,
>>
>> Vanshul Chawla
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2nd edition: http://www.manning.com/ibsen2



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2nd edition: http://www.manning.com/ibsen2

Re: Camel aggregation in case of server shutdown

Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Jul 15, 2015 at 9:15 AM, Vanshul.Chawla
<Va...@target.com> wrote:
> Thanks Claus. What in case someone kills process pid of server? Ie force kill the server?
>

Then they are lost.

There is this component on the way in a future Camel release that may help
http://camel.apache.org/file2

> Thanks and Regards,
>
> Vanshul Chawla
>
>
> -----Original Message-----
> From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Sent: Wednesday, July 15, 2015 12:41 PM
> To: users@camel.apache.org
> Subject: Re: Camel aggregation in case of server shutdown
>
> How do you aggregate them? If you use the Camel aggregator then its by default in-memory only, and you would loose those messages. There is pluggable persistent stores, but they have their own pros/cons.
>
> Its better to stop the consumer that takes in new messages, and wait for the aggregator batch to complete so there is no inflight, and then stop.
>
> You can also enable forceCompletionOnStop on the aggregator then it flushes out all in-flight exchanges so you can stop quicker. In fact it may be the option you are looking for.
>
> On Wed, Jul 15, 2015 at 9:06 AM, Vanshul.Chawla <Va...@target.com> wrote:
>> Hello All,
>>
>> We have a scenario wherein we aggregate message coming from a IBM MQ queue for 15 minutes before spitting a batch out to another MQ.
>> Suppose we have a server shutdown at 10th minute, what will happen to inflight messages? Will these be lost? Or will Fuse server restart get these messages and flush them out to destination.
>>
>> Thanks and Regards,
>>
>> Vanshul Chawla
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2nd edition: http://www.manning.com/ibsen2



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2nd edition: http://www.manning.com/ibsen2

RE: Camel aggregation in case of server shutdown

Posted by "Vanshul.Chawla" <Va...@target.com>.
Thanks Claus. What in case someone kills process pid of server? Ie force kill the server?

Thanks and Regards,

Vanshul Chawla


-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com] 
Sent: Wednesday, July 15, 2015 12:41 PM
To: users@camel.apache.org
Subject: Re: Camel aggregation in case of server shutdown

How do you aggregate them? If you use the Camel aggregator then its by default in-memory only, and you would loose those messages. There is pluggable persistent stores, but they have their own pros/cons.

Its better to stop the consumer that takes in new messages, and wait for the aggregator batch to complete so there is no inflight, and then stop.

You can also enable forceCompletionOnStop on the aggregator then it flushes out all in-flight exchanges so you can stop quicker. In fact it may be the option you are looking for.

On Wed, Jul 15, 2015 at 9:06 AM, Vanshul.Chawla <Va...@target.com> wrote:
> Hello All,
>
> We have a scenario wherein we aggregate message coming from a IBM MQ queue for 15 minutes before spitting a batch out to another MQ.
> Suppose we have a server shutdown at 10th minute, what will happen to inflight messages? Will these be lost? Or will Fuse server restart get these messages and flush them out to destination.
>
> Thanks and Regards,
>
> Vanshul Chawla
>



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2nd edition: http://www.manning.com/ibsen2

Re: Camel aggregation in case of server shutdown

Posted by Claus Ibsen <cl...@gmail.com>.
How do you aggregate them? If you use the Camel aggregator then its by
default in-memory only, and you would loose those messages. There is
pluggable persistent stores, but they have their own pros/cons.

Its better to stop the consumer that takes in new messages, and wait
for the aggregator batch to complete so there is no inflight, and then
stop.

You can also enable forceCompletionOnStop on the aggregator then it
flushes out all in-flight exchanges so you can stop quicker. In fact
it may be the option you are looking for.

On Wed, Jul 15, 2015 at 9:06 AM, Vanshul.Chawla
<Va...@target.com> wrote:
> Hello All,
>
> We have a scenario wherein we aggregate message coming from a IBM MQ queue for 15 minutes before spitting a batch out to another MQ.
> Suppose we have a server shutdown at 10th minute, what will happen to inflight messages? Will these be lost? Or will Fuse server restart get these messages and flush them out to destination.
>
> Thanks and Regards,
>
> Vanshul Chawla
>



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2nd edition: http://www.manning.com/ibsen2

RE: Camel aggregation in case of server shutdown

Posted by "Vanshul.Chawla" <Va...@target.com>.
<route id="EbayRoute-Prod1">
			<camel:from uri="wmq-qm1-Prod1-Ebay:queue:{{financial.WMQ.EbaySubQueue}}"></camel:from>
			<process ref="setHeaders-Prod1" />
			<aggregate strategyRef="myAggregationStrategy-Prod1"
				prop:completionInterval="{{financial.Aggregation.EbayCompletionInterval}}">
				<correlationExpression>
					<constant>true</constant>
				</correlationExpression>
				<camel:to uri="direct:EbayOutQueue1"></camel:to>
			</aggregate>
		</route>
		<route id="EbayRoute2-Prod1">
			<camel:from uri="direct:EbayOutQueue1"></camel:from>
			<process ref="createEbayMessage-Prod">
				<description>
					Original payload message in one queue like our ORIG
					queues
					Generate exception message in java code
				</description>
			</process>
			<camel:to uri="wmq-qm1-Prod1-Ebay:queue:{{financial.WMQ.EbayOutQueue}}"></camel:to>
		</route>


Thanks and Regards,

Vanshul Chawla

-----Original Message-----
From: Vanshul.Chawla [mailto:Vanshul.Chawla@target.com] 
Sent: Wednesday, July 15, 2015 12:36 PM
To: users@camel.apache.org
Subject: Camel aggregation in case of server shutdown

Hello All,

We have a scenario wherein we aggregate message coming from a IBM MQ queue for 15 minutes before spitting a batch out to another MQ.
Suppose we have a server shutdown at 10th minute, what will happen to inflight messages? Will these be lost? Or will Fuse server restart get these messages and flush them out to destination.

Thanks and Regards,

Vanshul Chawla