You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by jfaath <jf...@apache.org> on 2010/04/20 22:32:34 UTC
Remote broker message behavior
I have a camel application that's using a remote ActiveMQ broker instance.
Messages come in to the application via HTTP and are sent to a "inbound"
queue using "inOnly". Messages are then processed and sent to a "processed"
queue. Finally, if an error occurred during processing, those messages are
sent to an "errors" queue.
So, ideally at the end, there should only be messages either in the
"processed" queue or the "errors" queue. Actually, I don't really care if
they end up in the "processed" queue, but I definitely need to see the
messages in the "errors" queue.
However, what ends up happening is, full messages stay in the "inbound"
queue under "pending messages", messages are processed as expected but
nothing ends up in the "processed" queue and while messages resulting in
errors do end up in the "errors" queue, the message body is gone.
I'm using the ActiveMQ web console to inspect the queues. The "processed"
queue is the only one that has a consumer attached to it (which explains why
it ends up with no messages). Is this expected behavior? How can I have
control over what is happening with messages in these queues using camel
routes? Really what I most need fixed is to not lose the body of the
message in the error queue.
-JF
--
View this message in context: http://old.nabble.com/Remote-broker-message-behavior-tp28287920p28287920.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: Remote broker message behavior
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
ActiveMQ also has its own error handling mechanism with Dead Letter Queues.
On Tue, Apr 20, 2010 at 10:32 PM, jfaath <jf...@apache.org> wrote:
>
> I have a camel application that's using a remote ActiveMQ broker instance.
> Messages come in to the application via HTTP and are sent to a "inbound"
> queue using "inOnly". Messages are then processed and sent to a "processed"
> queue. Finally, if an error occurred during processing, those messages are
> sent to an "errors" queue.
>
> So, ideally at the end, there should only be messages either in the
> "processed" queue or the "errors" queue. Actually, I don't really care if
> they end up in the "processed" queue, but I definitely need to see the
> messages in the "errors" queue.
>
> However, what ends up happening is, full messages stay in the "inbound"
> queue under "pending messages", messages are processed as expected but
> nothing ends up in the "processed" queue and while messages resulting in
> errors do end up in the "errors" queue, the message body is gone.
>
> I'm using the ActiveMQ web console to inspect the queues. The "processed"
> queue is the only one that has a consumer attached to it (which explains why
> it ends up with no messages). Is this expected behavior? How can I have
> control over what is happening with messages in these queues using camel
> routes? Really what I most need fixed is to not lose the body of the
> message in the error queue.
>
> -JF
> --
> View this message in context: http://old.nabble.com/Remote-broker-message-behavior-tp28287920p28287920.html
> Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Remote broker message behavior
Posted by jfaath <jf...@apache.org>.
The message is in the proper queue, I just can't see the XML when viewing
through the ActiveMQ console.
I'm hoping to convince the project manager to view the message by a
different means making this issue moot. Thanks for the reply.
willem.jiang wrote:
>
> jfaath wrote:
>> Well, I figured out the one issue with the pending messages. Guess I was
>> confused about the difference between a consumer (using "from") and a
>> producer (using "to").
>>
>> Now I just have to figure out why the camel jms producer removes the
>> message
>> body when sending the message to the queue. At least, that's what I see
>> in
>> the ActiveMQ console. Any ideas?
>>
>>
> Do you mean you can't find any message from the errors queue? or there
> queue?
>
> If you are using Camel 2.x, it may relate to StreamCache issue[1]
>
> [1]http://camel.apache.org/stream-caching.html
>
>
> Willem
>>
>> jfaath wrote:
>>> Sure, here is the route:
>>>
>>> from("servlet:///data-in").convertBodyTo(String.class)
>>> .to("validator:schema-file.xsd")
>>> .inOnly("jms:queue:inbound")
>>> .choice()
>>> .when().xpath("/a:data/a:dataLog1", ns)
>>> .unmarshal(jaxbDf)
>>> .beanRef("dataLog1Processor", "process")
>>> .marshal(jaxbDf)
>>> .to("jms:queue:processed")
>>> .when().xpath("/a:data/a:dataLog2", ns)
>>> .unmarshal(jaxbDf)
>>> .beanRef("dataLog2Processor", "process")
>>> .marshal(jaxbDf)
>>> .to("jms:queue:" + processedQueue)
>>> .otherwise().to("mock:invalid");
>>>
>>> from("jms:queue:processed")
>>> .choice()
>>> .when(header("errorsFound").isEqualTo(true))
>>> .to("jms:queue:errors")
>>> .otherwise().to("mock:finished");
>>>
>>>
>>>
>>>
>>> willem.jiang wrote:
>>>> Hi,
>>>>
>>>> Can we have a look at your camel route?
>>>> Which version of camel are your using?
>>>>
>>>> You issue relates to the ErrorHandler[1], you may need to go through
>>>> the
>>>> wiki document for it.
>>>>
>>>> [1]http://camel.apache.org/error-handler.html
>>>>
>>>> Willem
>>>>
>>>> jfaath wrote:
>>>>> I have a camel application that's using a remote ActiveMQ broker
>>>>> instance.
>>>>> Messages come in to the application via HTTP and are sent to a
>>>>> "inbound"
>>>>> queue using "inOnly". Messages are then processed and sent to a
>>>>> "processed"
>>>>> queue. Finally, if an error occurred during processing, those
>>>>> messages
>>>>> are
>>>>> sent to an "errors" queue.
>>>>>
>>>>> So, ideally at the end, there should only be messages either in the
>>>>> "processed" queue or the "errors" queue. Actually, I don't really
>>>>> care
>>>>> if
>>>>> they end up in the "processed" queue, but I definitely need to see the
>>>>> messages in the "errors" queue.
>>>>>
>>>>> However, what ends up happening is, full messages stay in the
>>>>> "inbound"
>>>>> queue under "pending messages", messages are processed as expected but
>>>>> nothing ends up in the "processed" queue and while messages resulting
>>>>> in
>>>>> errors do end up in the "errors" queue, the message body is gone.
>>>>>
>>>>> I'm using the ActiveMQ web console to inspect the queues. The
>>>>> "processed"
>>>>> queue is the only one that has a consumer attached to it (which
>>>>> explains
>>>>> why
>>>>> it ends up with no messages). Is this expected behavior? How can I
>>>>> have
>>>>> control over what is happening with messages in these queues using
>>>>> camel
>>>>> routes? Really what I most need fixed is to not lose the body of the
>>>>> message in the error queue.
>>>>>
>>>>> -JF
>>>>
>>>>
>>>
>>
>
>
>
--
View this message in context: http://old.nabble.com/Remote-broker-message-behavior-tp28287920p28365128.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: Remote broker message behavior
Posted by Willem Jiang <wi...@gmail.com>.
jfaath wrote:
> Well, I figured out the one issue with the pending messages. Guess I was
> confused about the difference between a consumer (using "from") and a
> producer (using "to").
>
> Now I just have to figure out why the camel jms producer removes the message
> body when sending the message to the queue. At least, that's what I see in
> the ActiveMQ console. Any ideas?
>
>
Do you mean you can't find any message from the errors queue? or there
queue?
If you are using Camel 2.x, it may relate to StreamCache issue[1]
[1]http://camel.apache.org/stream-caching.html
Willem
>
> jfaath wrote:
>> Sure, here is the route:
>>
>> from("servlet:///data-in").convertBodyTo(String.class)
>> .to("validator:schema-file.xsd")
>> .inOnly("jms:queue:inbound")
>> .choice()
>> .when().xpath("/a:data/a:dataLog1", ns)
>> .unmarshal(jaxbDf)
>> .beanRef("dataLog1Processor", "process")
>> .marshal(jaxbDf)
>> .to("jms:queue:processed")
>> .when().xpath("/a:data/a:dataLog2", ns)
>> .unmarshal(jaxbDf)
>> .beanRef("dataLog2Processor", "process")
>> .marshal(jaxbDf)
>> .to("jms:queue:" + processedQueue)
>> .otherwise().to("mock:invalid");
>>
>> from("jms:queue:processed")
>> .choice()
>> .when(header("errorsFound").isEqualTo(true))
>> .to("jms:queue:errors")
>> .otherwise().to("mock:finished");
>>
>>
>>
>>
>> willem.jiang wrote:
>>> Hi,
>>>
>>> Can we have a look at your camel route?
>>> Which version of camel are your using?
>>>
>>> You issue relates to the ErrorHandler[1], you may need to go through the
>>> wiki document for it.
>>>
>>> [1]http://camel.apache.org/error-handler.html
>>>
>>> Willem
>>>
>>> jfaath wrote:
>>>> I have a camel application that's using a remote ActiveMQ broker
>>>> instance.
>>>> Messages come in to the application via HTTP and are sent to a "inbound"
>>>> queue using "inOnly". Messages are then processed and sent to a
>>>> "processed"
>>>> queue. Finally, if an error occurred during processing, those messages
>>>> are
>>>> sent to an "errors" queue.
>>>>
>>>> So, ideally at the end, there should only be messages either in the
>>>> "processed" queue or the "errors" queue. Actually, I don't really care
>>>> if
>>>> they end up in the "processed" queue, but I definitely need to see the
>>>> messages in the "errors" queue.
>>>>
>>>> However, what ends up happening is, full messages stay in the "inbound"
>>>> queue under "pending messages", messages are processed as expected but
>>>> nothing ends up in the "processed" queue and while messages resulting in
>>>> errors do end up in the "errors" queue, the message body is gone.
>>>>
>>>> I'm using the ActiveMQ web console to inspect the queues. The
>>>> "processed"
>>>> queue is the only one that has a consumer attached to it (which explains
>>>> why
>>>> it ends up with no messages). Is this expected behavior? How can I have
>>>> control over what is happening with messages in these queues using camel
>>>> routes? Really what I most need fixed is to not lose the body of the
>>>> message in the error queue.
>>>>
>>>> -JF
>>>
>>>
>>
>
Re: Remote broker message behavior
Posted by jfaath <jf...@apache.org>.
Well, I figured out the one issue with the pending messages. Guess I was
confused about the difference between a consumer (using "from") and a
producer (using "to").
Now I just have to figure out why the camel jms producer removes the message
body when sending the message to the queue. At least, that's what I see in
the ActiveMQ console. Any ideas?
jfaath wrote:
>
> Sure, here is the route:
>
> from("servlet:///data-in").convertBodyTo(String.class)
> .to("validator:schema-file.xsd")
> .inOnly("jms:queue:inbound")
> .choice()
> .when().xpath("/a:data/a:dataLog1", ns)
> .unmarshal(jaxbDf)
> .beanRef("dataLog1Processor", "process")
> .marshal(jaxbDf)
> .to("jms:queue:processed")
> .when().xpath("/a:data/a:dataLog2", ns)
> .unmarshal(jaxbDf)
> .beanRef("dataLog2Processor", "process")
> .marshal(jaxbDf)
> .to("jms:queue:" + processedQueue)
> .otherwise().to("mock:invalid");
>
> from("jms:queue:processed")
> .choice()
> .when(header("errorsFound").isEqualTo(true))
> .to("jms:queue:errors")
> .otherwise().to("mock:finished");
>
>
>
>
> willem.jiang wrote:
>>
>> Hi,
>>
>> Can we have a look at your camel route?
>> Which version of camel are your using?
>>
>> You issue relates to the ErrorHandler[1], you may need to go through the
>> wiki document for it.
>>
>> [1]http://camel.apache.org/error-handler.html
>>
>> Willem
>>
>> jfaath wrote:
>>> I have a camel application that's using a remote ActiveMQ broker
>>> instance.
>>> Messages come in to the application via HTTP and are sent to a "inbound"
>>> queue using "inOnly". Messages are then processed and sent to a
>>> "processed"
>>> queue. Finally, if an error occurred during processing, those messages
>>> are
>>> sent to an "errors" queue.
>>>
>>> So, ideally at the end, there should only be messages either in the
>>> "processed" queue or the "errors" queue. Actually, I don't really care
>>> if
>>> they end up in the "processed" queue, but I definitely need to see the
>>> messages in the "errors" queue.
>>>
>>> However, what ends up happening is, full messages stay in the "inbound"
>>> queue under "pending messages", messages are processed as expected but
>>> nothing ends up in the "processed" queue and while messages resulting in
>>> errors do end up in the "errors" queue, the message body is gone.
>>>
>>> I'm using the ActiveMQ web console to inspect the queues. The
>>> "processed"
>>> queue is the only one that has a consumer attached to it (which explains
>>> why
>>> it ends up with no messages). Is this expected behavior? How can I have
>>> control over what is happening with messages in these queues using camel
>>> routes? Really what I most need fixed is to not lose the body of the
>>> message in the error queue.
>>>
>>> -JF
>>
>>
>>
>
>
--
View this message in context: http://old.nabble.com/Remote-broker-message-behavior-tp28287920p28346703.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: Remote broker message behavior
Posted by jfaath <jf...@apache.org>.
Sure, here is the route:
from("servlet:///data-in").convertBodyTo(String.class)
.to("validator:schema-file.xsd")
.inOnly("jms:queue:inbound")
.choice()
.when().xpath("/a:data/a:dataLog1", ns)
.unmarshal(jaxbDf)
.beanRef("dataLog1Processor", "process")
.marshal(jaxbDf)
.to("jms:queue:processed")
.when().xpath("/a:data/a:dataLog2", ns)
.unmarshal(jaxbDf)
.beanRef("dataLog2Processor", "process")
.marshal(jaxbDf)
.to("jms:queue:" + processedQueue)
.otherwise().to("mock:invalid");
from("jms:queue:processed")
.choice()
.when(header("errorsFound").isEqualTo(true))
.to("jms:queue:errors")
.otherwise().to("mock:finished");
willem.jiang wrote:
>
> Hi,
>
> Can we have a look at your camel route?
> Which version of camel are your using?
>
> You issue relates to the ErrorHandler[1], you may need to go through the
> wiki document for it.
>
> [1]http://camel.apache.org/error-handler.html
>
> Willem
>
> jfaath wrote:
>> I have a camel application that's using a remote ActiveMQ broker
>> instance.
>> Messages come in to the application via HTTP and are sent to a "inbound"
>> queue using "inOnly". Messages are then processed and sent to a
>> "processed"
>> queue. Finally, if an error occurred during processing, those messages
>> are
>> sent to an "errors" queue.
>>
>> So, ideally at the end, there should only be messages either in the
>> "processed" queue or the "errors" queue. Actually, I don't really care
>> if
>> they end up in the "processed" queue, but I definitely need to see the
>> messages in the "errors" queue.
>>
>> However, what ends up happening is, full messages stay in the "inbound"
>> queue under "pending messages", messages are processed as expected but
>> nothing ends up in the "processed" queue and while messages resulting in
>> errors do end up in the "errors" queue, the message body is gone.
>>
>> I'm using the ActiveMQ web console to inspect the queues. The
>> "processed"
>> queue is the only one that has a consumer attached to it (which explains
>> why
>> it ends up with no messages). Is this expected behavior? How can I have
>> control over what is happening with messages in these queues using camel
>> routes? Really what I most need fixed is to not lose the body of the
>> message in the error queue.
>>
>> -JF
>
>
>
--
View this message in context: http://old.nabble.com/Remote-broker-message-behavior-tp28287920p28331966.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: How implement customed transaction section in camel?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
This requires Camel 2.2 onwards.
And check out chapter 9 in the book as it covers using multiple TX demarcations.
On Wed, Apr 21, 2010 at 4:41 PM, ext2 <xu...@tongtech.com> wrote:
>
> Hi:
> Camel 's transaction-policy is applied for all processors in the
> route; but how to use this features to implement a customized transaction
> section in camel? Etc: if I could assign a transaction-policy to a single
> pipe-line processor(not the whole route), I could define custom transaction
> section easily;
>
> To illustrate this usage I give a sample , and I want to know is there
> a-better solution to use camel;
>
> For example: A application which read message from jms and, save the message
> to 2 different database table(A, B). This application asks for this
> transaction features:
>
> Two database table must keep in a same transaction: one failed another
> rollback;
>
> Jms must take part in seperate transaction, that's to say if both database
> success, the jms will commit; if one database failed, I will log the jms
> message in a storage, and jms will commit; But if logging failed, the jms
> will rollback;
>
> And how to full-fill this transaction features using camel?
> Must I separate the complete route into different route piece just for
> transaction support?
> Could camel support the define a transaction range (but not a
> transaction-policy)?
>
> Even I separate the complete route into different route just for transaction
> support, it still has a difficult: how to declared a new transaction for a
> processing route( in this sample , I need to declare a new transaction for
> table-A and table-B processors); if I declared a required-new-transaction
> policy, all the processor in route act in a separate transaction; and it
> cannot full-fill such demand: table A and table B act in a transaction;
> To resolve this problem , I must introduce another route just act as a
> bridge for the new-tx-range);following is a solution use camel's current
> transaction policy to this example, but it's hard to understand:
>
> routeA: (jms)
> (from-jms)-->(required-tx-policy)-->doTry-->(to
> direct:new-tx-op)-->(docatch)-->(log);
>
> routeB:(this is the hardest part for usage)
> (from direct:new-tx-op)-->new-tx -policy-->(to direct:db-op)
>
> Route C:
> (from direct:db-op)-->(required-tx-policy)-->(table A save)-->(table B save)
>
> So does camel support a feature to define transaction range? So the route
> will looks like
>
> (from jms)-->pipeline(required-tx) -->[dotry-->pipeline(new-tx)-->[(table
> A)-->(table B)]-->docatch-->log];
>
> If camel could assign a transaction-policy to single pipe-line process ,
> this feature could be achieved easily;
>
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
How implement customed transaction section in camel?
Posted by ext2 <xu...@tongtech.com>.
Hi:
Camel 's transaction-policy is applied for all processors in the
route; but how to use this features to implement a customized transaction
section in camel? Etc: if I could assign a transaction-policy to a single
pipe-line processor(not the whole route), I could define custom transaction
section easily;
To illustrate this usage I give a sample , and I want to know is there
a-better solution to use camel;
For example: A application which read message from jms and, save the message
to 2 different database table(A, B). This application asks for this
transaction features:
Two database table must keep in a same transaction: one failed another
rollback;
Jms must take part in seperate transaction, that's to say if both database
success, the jms will commit; if one database failed, I will log the jms
message in a storage, and jms will commit; But if logging failed, the jms
will rollback;
And how to full-fill this transaction features using camel?
Must I separate the complete route into different route piece just for
transaction support?
Could camel support the define a transaction range (but not a
transaction-policy)?
Even I separate the complete route into different route just for transaction
support, it still has a difficult: how to declared a new transaction for a
processing route( in this sample , I need to declare a new transaction for
table-A and table-B processors); if I declared a required-new-transaction
policy, all the processor in route act in a separate transaction; and it
cannot full-fill such demand: table A and table B act in a transaction;
To resolve this problem , I must introduce another route just act as a
bridge for the new-tx-range);following is a solution use camel's current
transaction policy to this example, but it's hard to understand:
routeA: (jms)
(from-jms)-->(required-tx-policy)-->doTry-->(to
direct:new-tx-op)-->(docatch)-->(log);
routeB:(this is the hardest part for usage)
(from direct:new-tx-op)-->new-tx -policy-->(to direct:db-op)
Route C:
(from direct:db-op)-->(required-tx-policy)-->(table A save)-->(table B save)
So does camel support a feature to define transaction range? So the route
will looks like
(from jms)-->pipeline(required-tx) -->[dotry-->pipeline(new-tx)-->[(table
A)-->(table B)]-->docatch-->log];
If camel could assign a transaction-policy to single pipe-line process ,
this feature could be achieved easily;
Re: What is the usage for the spring element?
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Apr 21, 2010 at 2:04 PM, ext2 <xu...@tongtech.com> wrote:
>
> Hi :
>
> In camel 1.6x, I must use the <camel:policy> to control transaction; but
> when camel 2.2x , a new <transacted> element is introduced. Is the
> <transacted> is just a simplified for <policy> or it provide some new
> features?
>
Yeah it makes it simpler to use, as you dont need to mess with spring
propagation policies and whatnot if you just want the default cases.
Camel in Action chapter 9 covers transaction in great deals.
> Thanks for any suggestion
>
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
What is the usage for the spring element?
Posted by ext2 <xu...@tongtech.com>.
Hi :
In camel 1.6x, I must use the <camel:policy> to control transaction; but
when camel 2.2x , a new <transacted> element is introduced. Is the
<transacted> is just a simplified for <policy> or it provide some new
features?
Thanks for any suggestion
Re: Remote broker message behavior
Posted by Willem Jiang <wi...@gmail.com>.
Hi,
Can we have a look at your camel route?
Which version of camel are your using?
You issue relates to the ErrorHandler[1], you may need to go through the
wiki document for it.
[1]http://camel.apache.org/error-handler.html
Willem
jfaath wrote:
> I have a camel application that's using a remote ActiveMQ broker instance.
> Messages come in to the application via HTTP and are sent to a "inbound"
> queue using "inOnly". Messages are then processed and sent to a "processed"
> queue. Finally, if an error occurred during processing, those messages are
> sent to an "errors" queue.
>
> So, ideally at the end, there should only be messages either in the
> "processed" queue or the "errors" queue. Actually, I don't really care if
> they end up in the "processed" queue, but I definitely need to see the
> messages in the "errors" queue.
>
> However, what ends up happening is, full messages stay in the "inbound"
> queue under "pending messages", messages are processed as expected but
> nothing ends up in the "processed" queue and while messages resulting in
> errors do end up in the "errors" queue, the message body is gone.
>
> I'm using the ActiveMQ web console to inspect the queues. The "processed"
> queue is the only one that has a consumer attached to it (which explains why
> it ends up with no messages). Is this expected behavior? How can I have
> control over what is happening with messages in these queues using camel
> routes? Really what I most need fixed is to not lose the body of the
> message in the error queue.
>
> -JF