You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by gramanero <gr...@gmail.com> on 2012/06/04 21:51:19 UTC
Transacted vs DeadLetterQueue
I'm wondering if there are any sort of best practices or pros/cons to using
either the dead letter queue or transactions when errors occur while pulling
off messages from a queue and attempting to route them to a cxfrs endpoint
that may not be available.
For example, say I have the following route:
<route id="myRoute" startupOrder="1" >
<from uri="activemq:queue:myQueue"/>
<to uri="cxfrs:http://localhost:9999/MyService" pattern="InOnly"/>
</route>
I have played around with using the dead letter queue and that seems to work
well, but then I started to think about getting the messages off the dead
letter queue and putting them back onto myQueue for processing once the
service becomes available. One way to accomplish this (possibly) is to use
the CronScheduledRoutePolicy such that it wakes up every hour and starts a
route that will pull messages off the dead letter queue (if there are any),
put them onto the myQueue and then stop the route after N number of minutes.
I'm not sure that this is a good idea, but it "one" way to build a more
auto-maintained system. Seems like once I start getting into configuring the
CronScheduledRoutePolicy that the solution becomes more complex then it
really needs to be in terms of the various moving parts.
The other thought is to use the ActiveMQ transaction manager. If the cxfrs
endpoint is not available, then the transaction rolls back which puts the
message back on the queue (or just never really pulls it off) and then waits
a while (via a delay configuration) and tries to route the message again.
One thing I don't really like about this is that if the service is down for
a long period of time, then it seems like we will be in this mode where
messages are constantly trying to be routed to the unavailable cxfrs
endpoint thus sort of thrashing the system.
I'm just looking for some opinions to how this type of problem is typically
solved or some sort of best practice to this use case.
Thanks!
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
Thanks Claus.
So is collision avoidance a different mechanism than the exponential
back-off settings or do they work together?
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714060.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Jun 5, 2012 at 8:12 PM, gramanero <gr...@gmail.com> wrote:
> Thanks for the response Christian.
>
> I have been reading through the Camel in Action and ActiveMQ in Action books
> most of the day today. I'm starting to get a better understanding of how
> transactions work and that definitely seems like the way to go. Also, seems
> like the DLQ and transactions work together, so that I can dump off messages
> in the DLQ if "something bad happens". I think what I want in my case is to
> set up the redelivery policy so that messages stay on the queue until the
> cxfrs endpoint becomes available again. Looks like I can achieve this with a
> value of -1 for the maximumRedelivery configuration. I just need to
> determine a good balance for the other settings (i.e. redeliveryDelay) so
> that I'm not trashing the system by constantly trying to resend messages to
> an unavailable endpoint.
>
> Can someone tell me what the useCollisionAvoidence setting is for? The only
> explanation that I can find is:
>
> "Should the redelivery policy use collision avoidance"
>
> This really isn't telling me a whole lot. I can't seem to find anything in
> either book that discusses it either.
>
The collision avoidance is using another option that dictates a
multiplier value.
So the idea is that this is used to calculate a increasing delay for
redelivery, (eg like network transport layer, having collision
avoidance on the ethernet).
> Thanks.
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714032.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
In reading your email (below) again, it reminds me of the path I started to
go down earlier. I began by taking all messages that caused an exception
and place them into the DLQ right away. THen I was left with the problem of
cleaning up the DLQ by pulling out all good messages, leaving the bad
messages, and routing the good messages back into the route for processing.
What I realized is that my solution for doing this started getting more and
more complex (and maybe that is just inherit in the problem). I started
looking into using the CronScheduledRoutePolicy to start a route at a
specific interval which would pull the messages off the DLQ and put them
back into the primay queue for processing. Then the CronSchedulePolicy
would stop the DLQ extraction route after N minutes and go to sleep.
Obviously I had not thought about "bead messages" at the time, but it just
seems to me that I want to do whatever I can up front to keep the good
messages in the primary queue at all times and only put bad messages into
the DLQ when I know that they are bad.
For instance, if I am getting connection issues with the restful service,
then I don't want to assume that the messages are bad and move them to the
DLQ queue, because now I have to manage them in the DLQ. I would rather
manage them with a redelivery policy and keep retrying every N minutes
knowing full well that our systems will be nback up and running within a
reasonable amount of time. Also, realize that in my case we are not talking
about business critical data. I am dealing with tracking metrics data, so
if the data does not get processed immediately, but soon after, then that
is acceptable.
Had this been for business critical data, though my thought process might
be different. I am enjoying this discussion though!
On Fri, Jun 8, 2012 at 8:05 AM, James Carman-2 [via Camel] <
ml-node+s465427n5714192h28@n5.nabble.com> wrote:
> Camel's redelivery is immediate and doesn't go all the way back to the
> start of the route. The feature I'm hopefully adding today is the
> auto-suspending routes. The route itself will go to "sleep" for a bit to
> allow the problem to be fixed and not needlessly overflow the
> purgatory/dlc
> queues. We're trying to avoid dlc replays if we can. We would rather the
> system auto-heal. Now if the message that goes to purgatory is truly
> "poison" then it will indeed end up in the dlc queue, but that's where it
> belongs anyway.
> On Jun 8, 2012 7:47 AM, "gramanero" <[hidden email]<http://user/SendEmail.jtp?type=node&node=5714192&i=0>>
> wrote:
>
> > So in your example, what is the difference between the Purgatory count
> and
> > using the Redelivery capabilities built into ActiveMQ and Camel
> policies?
> > Maybe I'm missing something, but it seems like the logic wrapped around
> the
> > Purgatory concept is just keeping track of how many times a message has
> > been
> > delivered and caused an exception to happen. This could happen if the
> "to"
> > endpoint is unavailable (i.e. a restful service) and until the service
> does
> > become available the count will go up until the message ends up in the
> DLQ.
> >
> > Like I said, maybe I'm just missing something here.
> >
> > --
> > View this message in context:
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714188.html
>
> > Sent from the Camel - Users mailing list archive at Nabble.com.
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714192.html
> To unsubscribe from Transacted vs DeadLetterQueue, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5713992&code=Z3JhbWFuZXJvQGdtYWlsLmNvbXw1NzEzOTkyfC0xNjAyMDYxMDQz>
> .
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714197.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
Interesting idea. I had not realized that the redelivery doesn't go all the
way back to the start of the route, but now that you mention it that makes
much more sense and explains a few things I had questions about.
My route is extremely simple in that it pulls from the queue and sends the
messag to the restful service. So for me, a redelivery will still
accomplish that, but it is good to know that it is not starting at the
beginning of the route. I was wondering how Camel was able to interact with
the ActiveMQ redelivery mechanism and it sounds like it isn't. So now I"m
wondering how ActiveMQ knows that the message should be sent to the
restulful service when redelivering a message, because the message does get
replayed back to my restulful service and ActiveMQ doesn't know wnything
about the service, but Camel does via the route.
On Fri, Jun 8, 2012 at 8:05 AM, James Carman-2 [via Camel] <
ml-node+s465427n5714192h28@n5.nabble.com> wrote:
> Camel's redelivery is immediate and doesn't go all the way back to the
> start of the route. The feature I'm hopefully adding today is the
> auto-suspending routes. The route itself will go to "sleep" for a bit to
> allow the problem to be fixed and not needlessly overflow the
> purgatory/dlc
> queues. We're trying to avoid dlc replays if we can. We would rather the
> system auto-heal. Now if the message that goes to purgatory is truly
> "poison" then it will indeed end up in the dlc queue, but that's where it
> belongs anyway.
> On Jun 8, 2012 7:47 AM, "gramanero" <[hidden email]<http://user/SendEmail.jtp?type=node&node=5714192&i=0>>
> wrote:
>
> > So in your example, what is the difference between the Purgatory count
> and
> > using the Redelivery capabilities built into ActiveMQ and Camel
> policies?
> > Maybe I'm missing something, but it seems like the logic wrapped around
> the
> > Purgatory concept is just keeping track of how many times a message has
> > been
> > delivered and caused an exception to happen. This could happen if the
> "to"
> > endpoint is unavailable (i.e. a restful service) and until the service
> does
> > become available the count will go up until the message ends up in the
> DLQ.
> >
> > Like I said, maybe I'm just missing something here.
> >
> > --
> > View this message in context:
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714188.html
>
> > Sent from the Camel - Users mailing list archive at Nabble.com.
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714192.html
> To unsubscribe from Transacted vs DeadLetterQueue, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5713992&code=Z3JhbWFuZXJvQGdtYWlsLmNvbXw1NzEzOTkyfC0xNjAyMDYxMDQz>
> .
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714195.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <jc...@carmanconsulting.com>.
Luckily you've got an example you can use! :)
On Jun 8, 2012 8:02 AM, "gramanero" <gr...@gmail.com> wrote:
> Blueprint? Hmmm...I have not heard or seen that yet. lol...great. Another
> rabbit hole for me to dive down into. ;-)
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714190.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
Blueprint? Hmmm...I have not heard or seen that yet. lol...great. Another
rabbit hole for me to dive down into. ;-)
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714190.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
On Fri, Jun 8, 2012 at 10:05 AM, gramanero <gr...@gmail.com> wrote:
> How do you get the message to "bounce off the front of the queue"? Can you
> control whether the message is on the front or back of the queue? You
> brought up an interesting point at the beginning of this thread that
> indicated that a message could get stuck on the front of the queue thus
> preventing all other messages rom processing, so I would be interested to
> know how you are essentially pulling the message of and putting it back on
> the end of the queue. I thought a rollback would essentially keep the
> message on the front of the queue. Its a FIFO queue, right?
>
The rollback will cause the message to go back to the queue. It goes
right back on the "front" of the queue (the place where you remove
messages). Then, when your consumer grabs the next message off the
queue, it will be the message you just rolled back onto the front of
the queue. Then, you do the blacklist check and so forth.
> Overall, an interesting approach. I like the idea of getting messages
> temporarily out of the way until they can "play nice" again. I can see in a
> connection lost scenario that all of the messages will potentially end up in
> purgatory and then be put back on the original queue, hopefully when the
> service is available again. Essentially a holding tank until the situation
> resolves itself. I think the difficulty with any approach is with
> determining why a message is not playing nice. It is bad message content or
> is there some sort of system failure. Bad message content most likely means
> that the message needs to be taken out of rotation permanently (probably not
> to many cases where you can "fix" the message content), but a system failure
> (i.e. consuming service is down) is when we would want to try and re-play
> the message at a later time.
>
You're absolutely correct! There are certain types of errors that
indicate that you've got an unrecoverable error (NullPointerException,
JAXBException, etc.). Therefore, there will also be another feature
which will allow subclasses to specify certain exception types that
should just skip right over the purgatory queue and go straight to the
DLC queue (or "hell" to extend the analogy). I plan on putting all of
this logic into a superclass so that this is "baked in" and subclasses
get it for free!
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
How do you get the message to "bounce off the front of the queue"? Can you
control whether the message is on the front or back of the queue? You
brought up an interesting point at the beginning of this thread that
indicated that a message could get stuck on the front of the queue thus
preventing all other messages rom processing, so I would be interested to
know how you are essentially pulling the message of and putting it back on
the end of the queue. I thought a rollback would essentially keep the
message on the front of the queue. Its a FIFO queue, right?
Overall, an interesting approach. I like the idea of getting messages
temporarily out of the way until they can "play nice" again. I can see in a
connection lost scenario that all of the messages will potentially end up in
purgatory and then be put back on the original queue, hopefully when the
service is available again. Essentially a holding tank until the situation
resolves itself. I think the difficulty with any approach is with
determining why a message is not playing nice. It is bad message content or
is there some sort of system failure. Bad message content most likely means
that the message needs to be taken out of rotation permanently (probably not
to many cases where you can "fix" the message content), but a system failure
(i.e. consuming service is down) is when we would want to try and re-play
the message at a later time.
Thanks for sharing. I'm interesting to see how this pans out for you.
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714205.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
On Fri, Jun 8, 2012 at 8:07 AM, gramanero <gr...@gmail.com> wrote:
>
> If I let the transaction rollback, then I have the message on the queue
> still and then run the risk of trying to process it a second time which
> will result in the same scenario and most likely end up in an endless loop.
> Unless I configure the redelivery policy to only attempt to redeliver N
> times as opposed to an infinite numer of times.
>
With the "purgatory" pattern (best name I could think of at the time),
the transaction is allowed to rollback, but the message is
"blacklisted" and it sort of "bounces" back off the front of the
queue. The next time the consumer grabs the message off the queue, it
recognizes that it has been blacklisted and it forwards it to the
purgatory queue (unless it has already been there too many times, then
it goes to the DLC queue) to sort of take a "time out."
There is a throttled (to keep it from thrashing) consumer on the
purgatory queue that inspects each message that comes off the queue.
If the message has fulfilled its timeout and it's ready to try to come
back and play nice, then it sends it back to where it came from (the
original input queue). If not, then it just puts it back on the
purgatory queue (hence the possibility of thrashing) to be processed
again in a bit.
Again, at some point (hopefully today), I hope to implement a feature
which would allow the original input route (which consumes from the
input queue) to detect (still working on the algorithm) when it is
sending too much stuff to purgatory or DLC. When that happens, it
will spin off a thread to suspend it's consumer. It will also
schedule a timer task to come back and wake it up in a bit when
hopefully the problem (database down, web service unavailable, etc.)
has been addressed and everything is happy again. The input queue
itself will still continue to function and receive messages. The
consumer just won't be pulling messages off the front for a little
while.
The one requirement we have that this does address is that messages
are definitely not lost, since all of the routes are transactional.
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
Perhaps you are correct in that the onException gets fired within the
bounds of the transaction thus making the operation, as a whole, atomic.
However, and keep in mind that I am still learning Camel, from what I can
tell the onException fires within the bounds of the transaction and the
<handled><constant>true</constant></handled> elements are basically telling
Camle that "everything is ok" which means that things do not get rolled
back. In my case, I am trying to catch specific exceptions that are thrown
from the restulful service (which means that no work happened...i.e. no DB
updates or anything), and push the message onto the DLQ to get it out of
the way. Essentially saying that those messages are bad and I don't/can't
process them.
If I let the transaction rollback, then I have the message on the queue
still and then run the risk of trying to process it a second time which
will result in the same scenario and most likely end up in an endless loop.
Unless I configure the redelivery policy to only attempt to redeliver N
times as opposed to an infinite numer of times.
On Fri, Jun 8, 2012 at 7:30 AM, James Carman-2 [via Camel] <
ml-node+s465427n5714184h80@n5.nabble.com> wrote:
> You want it to be atomic with the get from the input queue don't you? It
> would make sense to me.
> On Jun 8, 2012 7:13 AM, "gramanero" <[hidden email]<http://user/SendEmail.jtp?type=node&node=5714184&i=0>>
> wrote:
>
> > Now that you mention it, one might think that, but that does not appear
> to
> > be the case. With the DLQ put in the onException clause, the DLQ put
> does
> > not rollback. I think that is where the Handled clause comes into play.
> >
> > On Fri, Jun 8, 2012 at 7:10 AM, James Carman-2 [via Camel] <
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=5714184&i=1>>
> wrote:
> >
> > > Won't the dlc "put" be in the transaction too? That would rollback
> too,
> > > thus nothing ever happened.
> > > On Jun 8, 2012 6:56 AM, "James Carman" <[hidden email]<
> > http://user/SendEmail.jtp?type=node&node=5714181&i=0>>
> > > wrote:
> > >
> > > > Try it with client cache control. Take a look at my example.
> > > >
> > > > On Fri, Jun 8, 2012 at 6:47 AM, gramanero <[hidden email]<
> > http://user/SendEmail.jtp?type=node&node=5714181&i=1>>
> > > wrote:
> > > > > No, it is not rolling back if you use the Handles element with a
> > > > constant value of true. If you use the Continue element then I
> believe
> > > it
> > > > will roll back.
> > > > >
> > > > > Sent from my iPod
> > > > >
> > > > > On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" <
> > > > [hidden email] <http://user/SendEmail.jtp?type=node&node=5714181&i=2>>
>
> > > wrote:
> > > > >
> > > > >> Your transaction isn't rolling back if you "handle" the
> exception,
> > is
> > > > it?
> > > > >>
> > > > >> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]>
> wrote:
> > > > >>
> > > > >> > I have tested the case of using a route specific onException
> > clause
> > > > within a
> > > > >> > transaction and it appears to work as I would expect (or hope).
> So
> > > I
> > > > have a
> > > > >> > route that is transactional and the final endpoint in the route
> > > > throws an
> > > > >> > exception I forced my restful service to just throw an
> exception).
> > > > Without
> > > > >> > the onException clause the message lands back in the queue as
> you
> > > > would
> > > > >> > expect due to it running within a transaction. With the
> > onException
> > > > clause,
> > > > >> > I look for specific exceptions and if it is one of the
> exceptions
> > > > that I
> > > > >> > have specified I tell tell Camel that the exception has been
> > > > "handled" (via
> > > > >> > the handled clause) and I route the message to the dead letter
> > > queue,
> > > > thus
> > > > >> > moving the "bad message" out of the way of the messages
> remaining
> > > on
> > > > the
> > > > >> > queue. I think the key here is the use of the "handled" clause
> > that
> > > > tells
> > > > >> > Camel that the message has been handled and therefore to NOT
> > > rollback
> > > > the
> > > > >> > transaction. The alternative choice is to tell Camel to
> "continue"
> > > on
> > > > with
> > > > >> > its normal processing which would have rolled back the
> transaction
> > > > and put
> > > > >> > the message back onto the queue (via the "continue" clause...at
> > > least
> > > > I
> > > > >> > think it is a clause).
> > > > >> >
> > > > >> > Hope that helps.
> > > > >> >
> > > > >> > --
> > > > >> > View this message in context:
> > > >
> > >
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
> > > > >> > Sent from the Camel - Users mailing list archive at Nabble.com.
> > > > >>
> > > > >>
> > > > >> If you reply to this email, your message will be added to the
> > > > discussion below:
> > > > >>
> > > >
> > >
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714151.html
> > > > >> To unsubscribe from Transacted vs DeadLetterQueue, click here.
> > > > >> NAML
> > > > >
> > > > >
> > > > > --
> > > > > View this message in context:
> > > >
> > >
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714179.html
> > > > > Sent from the Camel - Users mailing list archive at Nabble.com.
> > > >
> > >
> > >
> > > ------------------------------
> > > If you reply to this email, your message will be added to the
> discussion
> > > below:
> > >
> > >
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714181.html
> > > To unsubscribe from Transacted vs DeadLetterQueue, click here<
> >
> >
> > > .
> > > NAML<
> >
> http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>
> > >
> > >
> >
> >
> > --
> > View this message in context:
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714182.html
>
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714184.html
> To unsubscribe from Transacted vs DeadLetterQueue, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5713992&code=Z3JhbWFuZXJvQGdtYWlsLmNvbXw1NzEzOTkyfC0xNjAyMDYxMDQz>
> .
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714194.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <jc...@carmanconsulting.com>.
You want it to be atomic with the get from the input queue don't you? It
would make sense to me.
On Jun 8, 2012 7:13 AM, "gramanero" <gr...@gmail.com> wrote:
> Now that you mention it, one might think that, but that does not appear to
> be the case. With the DLQ put in the onException clause, the DLQ put does
> not rollback. I think that is where the Handled clause comes into play.
>
> On Fri, Jun 8, 2012 at 7:10 AM, James Carman-2 [via Camel] <
> ml-node+s465427n5714181h87@n5.nabble.com> wrote:
>
> > Won't the dlc "put" be in the transaction too? That would rollback too,
> > thus nothing ever happened.
> > On Jun 8, 2012 6:56 AM, "James Carman" <[hidden email]<
> http://user/SendEmail.jtp?type=node&node=5714181&i=0>>
> > wrote:
> >
> > > Try it with client cache control. Take a look at my example.
> > >
> > > On Fri, Jun 8, 2012 at 6:47 AM, gramanero <[hidden email]<
> http://user/SendEmail.jtp?type=node&node=5714181&i=1>>
> > wrote:
> > > > No, it is not rolling back if you use the Handles element with a
> > > constant value of true. If you use the Continue element then I believe
> > it
> > > will roll back.
> > > >
> > > > Sent from my iPod
> > > >
> > > > On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" <
> > > [hidden email] <http://user/SendEmail.jtp?type=node&node=5714181&i=2>>
> > wrote:
> > > >
> > > >> Your transaction isn't rolling back if you "handle" the exception,
> is
> > > it?
> > > >>
> > > >> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]> wrote:
> > > >>
> > > >> > I have tested the case of using a route specific onException
> clause
> > > within a
> > > >> > transaction and it appears to work as I would expect (or hope). So
> > I
> > > have a
> > > >> > route that is transactional and the final endpoint in the route
> > > throws an
> > > >> > exception I forced my restful service to just throw an exception).
> > > Without
> > > >> > the onException clause the message lands back in the queue as you
> > > would
> > > >> > expect due to it running within a transaction. With the
> onException
> > > clause,
> > > >> > I look for specific exceptions and if it is one of the exceptions
> > > that I
> > > >> > have specified I tell tell Camel that the exception has been
> > > "handled" (via
> > > >> > the handled clause) and I route the message to the dead letter
> > queue,
> > > thus
> > > >> > moving the "bad message" out of the way of the messages remaining
> > on
> > > the
> > > >> > queue. I think the key here is the use of the "handled" clause
> that
> > > tells
> > > >> > Camel that the message has been handled and therefore to NOT
> > rollback
> > > the
> > > >> > transaction. The alternative choice is to tell Camel to "continue"
> > on
> > > with
> > > >> > its normal processing which would have rolled back the transaction
> > > and put
> > > >> > the message back onto the queue (via the "continue" clause...at
> > least
> > > I
> > > >> > think it is a clause).
> > > >> >
> > > >> > Hope that helps.
> > > >> >
> > > >> > --
> > > >> > View this message in context:
> > >
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
> > > >> > Sent from the Camel - Users mailing list archive at Nabble.com.
> > > >>
> > > >>
> > > >> If you reply to this email, your message will be added to the
> > > discussion below:
> > > >>
> > >
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714151.html
> > > >> To unsubscribe from Transacted vs DeadLetterQueue, click here.
> > > >> NAML
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > >
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714179.html
> > > > Sent from the Camel - Users mailing list archive at Nabble.com.
> > >
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714181.html
> > To unsubscribe from Transacted vs DeadLetterQueue, click here<
> http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5713992&code=Z3JhbWFuZXJvQGdtYWlsLmNvbXw1NzEzOTkyfC0xNjAyMDYxMDQz
> >
> > .
> > NAML<
> http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714182.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
Now that you mention it, one might think that, but that does not appear to
be the case. With the DLQ put in the onException clause, the DLQ put does
not rollback. I think that is where the Handled clause comes into play.
On Fri, Jun 8, 2012 at 7:10 AM, James Carman-2 [via Camel] <
ml-node+s465427n5714181h87@n5.nabble.com> wrote:
> Won't the dlc "put" be in the transaction too? That would rollback too,
> thus nothing ever happened.
> On Jun 8, 2012 6:56 AM, "James Carman" <[hidden email]<http://user/SendEmail.jtp?type=node&node=5714181&i=0>>
> wrote:
>
> > Try it with client cache control. Take a look at my example.
> >
> > On Fri, Jun 8, 2012 at 6:47 AM, gramanero <[hidden email]<http://user/SendEmail.jtp?type=node&node=5714181&i=1>>
> wrote:
> > > No, it is not rolling back if you use the Handles element with a
> > constant value of true. If you use the Continue element then I believe
> it
> > will roll back.
> > >
> > > Sent from my iPod
> > >
> > > On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" <
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=5714181&i=2>>
> wrote:
> > >
> > >> Your transaction isn't rolling back if you "handle" the exception, is
> > it?
> > >>
> > >> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]> wrote:
> > >>
> > >> > I have tested the case of using a route specific onException clause
> > within a
> > >> > transaction and it appears to work as I would expect (or hope). So
> I
> > have a
> > >> > route that is transactional and the final endpoint in the route
> > throws an
> > >> > exception I forced my restful service to just throw an exception).
> > Without
> > >> > the onException clause the message lands back in the queue as you
> > would
> > >> > expect due to it running within a transaction. With the onException
> > clause,
> > >> > I look for specific exceptions and if it is one of the exceptions
> > that I
> > >> > have specified I tell tell Camel that the exception has been
> > "handled" (via
> > >> > the handled clause) and I route the message to the dead letter
> queue,
> > thus
> > >> > moving the "bad message" out of the way of the messages remaining
> on
> > the
> > >> > queue. I think the key here is the use of the "handled" clause that
> > tells
> > >> > Camel that the message has been handled and therefore to NOT
> rollback
> > the
> > >> > transaction. The alternative choice is to tell Camel to "continue"
> on
> > with
> > >> > its normal processing which would have rolled back the transaction
> > and put
> > >> > the message back onto the queue (via the "continue" clause...at
> least
> > I
> > >> > think it is a clause).
> > >> >
> > >> > Hope that helps.
> > >> >
> > >> > --
> > >> > View this message in context:
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
> > >> > Sent from the Camel - Users mailing list archive at Nabble.com.
> > >>
> > >>
> > >> If you reply to this email, your message will be added to the
> > discussion below:
> > >>
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714151.html
> > >> To unsubscribe from Transacted vs DeadLetterQueue, click here.
> > >> NAML
> > >
> > >
> > > --
> > > View this message in context:
> >
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714179.html
> > > Sent from the Camel - Users mailing list archive at Nabble.com.
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714181.html
> To unsubscribe from Transacted vs DeadLetterQueue, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5713992&code=Z3JhbWFuZXJvQGdtYWlsLmNvbXw1NzEzOTkyfC0xNjAyMDYxMDQz>
> .
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714182.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <jc...@carmanconsulting.com>.
Won't the dlc "put" be in the transaction too? That would rollback too,
thus nothing ever happened.
On Jun 8, 2012 6:56 AM, "James Carman" <ja...@carmanconsulting.com> wrote:
> Try it with client cache control. Take a look at my example.
>
> On Fri, Jun 8, 2012 at 6:47 AM, gramanero <gr...@gmail.com> wrote:
> > No, it is not rolling back if you use the Handles element with a
> constant value of true. If you use the Continue element then I believe it
> will roll back.
> >
> > Sent from my iPod
> >
> > On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" <
> ml-node+s465427n5714151h57@n5.nabble.com> wrote:
> >
> >> Your transaction isn't rolling back if you "handle" the exception, is
> it?
> >>
> >> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]> wrote:
> >>
> >> > I have tested the case of using a route specific onException clause
> within a
> >> > transaction and it appears to work as I would expect (or hope). So I
> have a
> >> > route that is transactional and the final endpoint in the route
> throws an
> >> > exception I forced my restful service to just throw an exception).
> Without
> >> > the onException clause the message lands back in the queue as you
> would
> >> > expect due to it running within a transaction. With the onException
> clause,
> >> > I look for specific exceptions and if it is one of the exceptions
> that I
> >> > have specified I tell tell Camel that the exception has been
> "handled" (via
> >> > the handled clause) and I route the message to the dead letter queue,
> thus
> >> > moving the "bad message" out of the way of the messages remaining on
> the
> >> > queue. I think the key here is the use of the "handled" clause that
> tells
> >> > Camel that the message has been handled and therefore to NOT rollback
> the
> >> > transaction. The alternative choice is to tell Camel to "continue" on
> with
> >> > its normal processing which would have rolled back the transaction
> and put
> >> > the message back onto the queue (via the "continue" clause...at least
> I
> >> > think it is a clause).
> >> >
> >> > Hope that helps.
> >> >
> >> > --
> >> > View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
> >> > Sent from the Camel - Users mailing list archive at Nabble.com.
> >>
> >>
> >> If you reply to this email, your message will be added to the
> discussion below:
> >>
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714151.html
> >> To unsubscribe from Transacted vs DeadLetterQueue, click here.
> >> NAML
> >
> >
> > --
> > View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714179.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
Try it with client cache control. Take a look at my example.
On Fri, Jun 8, 2012 at 6:47 AM, gramanero <gr...@gmail.com> wrote:
> No, it is not rolling back if you use the Handles element with a constant value of true. If you use the Continue element then I believe it will roll back.
>
> Sent from my iPod
>
> On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" <ml...@n5.nabble.com> wrote:
>
>> Your transaction isn't rolling back if you "handle" the exception, is it?
>>
>> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]> wrote:
>>
>> > I have tested the case of using a route specific onException clause within a
>> > transaction and it appears to work as I would expect (or hope). So I have a
>> > route that is transactional and the final endpoint in the route throws an
>> > exception I forced my restful service to just throw an exception). Without
>> > the onException clause the message lands back in the queue as you would
>> > expect due to it running within a transaction. With the onException clause,
>> > I look for specific exceptions and if it is one of the exceptions that I
>> > have specified I tell tell Camel that the exception has been "handled" (via
>> > the handled clause) and I route the message to the dead letter queue, thus
>> > moving the "bad message" out of the way of the messages remaining on the
>> > queue. I think the key here is the use of the "handled" clause that tells
>> > Camel that the message has been handled and therefore to NOT rollback the
>> > transaction. The alternative choice is to tell Camel to "continue" on with
>> > its normal processing which would have rolled back the transaction and put
>> > the message back onto the queue (via the "continue" clause...at least I
>> > think it is a clause).
>> >
>> > Hope that helps.
>> >
>> > --
>> > View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
>> > Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>> If you reply to this email, your message will be added to the discussion below:
>> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714151.html
>> To unsubscribe from Transacted vs DeadLetterQueue, click here.
>> NAML
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714179.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
No, it is not rolling back if you use the Handles element with a constant value of true. If you use the Continue element then I believe it will roll back.
Sent from my iPod
On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" <ml...@n5.nabble.com> wrote:
> Your transaction isn't rolling back if you "handle" the exception, is it?
>
> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]> wrote:
>
> > I have tested the case of using a route specific onException clause within a
> > transaction and it appears to work as I would expect (or hope). So I have a
> > route that is transactional and the final endpoint in the route throws an
> > exception I forced my restful service to just throw an exception). Without
> > the onException clause the message lands back in the queue as you would
> > expect due to it running within a transaction. With the onException clause,
> > I look for specific exceptions and if it is one of the exceptions that I
> > have specified I tell tell Camel that the exception has been "handled" (via
> > the handled clause) and I route the message to the dead letter queue, thus
> > moving the "bad message" out of the way of the messages remaining on the
> > queue. I think the key here is the use of the "handled" clause that tells
> > Camel that the message has been handled and therefore to NOT rollback the
> > transaction. The alternative choice is to tell Camel to "continue" on with
> > its normal processing which would have rolled back the transaction and put
> > the message back onto the queue (via the "continue" clause...at least I
> > think it is a clause).
> >
> > Hope that helps.
> >
> > --
> > View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714151.html
> To unsubscribe from Transacted vs DeadLetterQueue, click here.
> NAML
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714179.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <jc...@carmanconsulting.com>.
Camel's redelivery is immediate and doesn't go all the way back to the
start of the route. The feature I'm hopefully adding today is the
auto-suspending routes. The route itself will go to "sleep" for a bit to
allow the problem to be fixed and not needlessly overflow the purgatory/dlc
queues. We're trying to avoid dlc replays if we can. We would rather the
system auto-heal. Now if the message that goes to purgatory is truly
"poison" then it will indeed end up in the dlc queue, but that's where it
belongs anyway.
On Jun 8, 2012 7:47 AM, "gramanero" <gr...@gmail.com> wrote:
> So in your example, what is the difference between the Purgatory count and
> using the Redelivery capabilities built into ActiveMQ and Camel policies?
> Maybe I'm missing something, but it seems like the logic wrapped around the
> Purgatory concept is just keeping track of how many times a message has
> been
> delivered and caused an exception to happen. This could happen if the "to"
> endpoint is unavailable (i.e. a restful service) and until the service does
> become available the count will go up until the message ends up in the DLQ.
>
> Like I said, maybe I'm just missing something here.
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714188.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
So in your example, what is the difference between the Purgatory count and
using the Redelivery capabilities built into ActiveMQ and Camel policies?
Maybe I'm missing something, but it seems like the logic wrapped around the
Purgatory concept is just keeping track of how many times a message has been
delivered and caused an exception to happen. This could happen if the "to"
endpoint is unavailable (i.e. a restful service) and until the service does
become available the count will go up until the message ends up in the DLQ.
Like I said, maybe I'm just missing something here.
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714188.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <jc...@carmanconsulting.com>.
Well you might have to do quite a bit of copy/paste if you use the XML. I
would also suggest switching to blueprint as opposed to spring. You'll
avoid a lot of headaches!
On Jun 8, 2012 7:43 AM, "gramanero" <gr...@gmail.com> wrote:
> I'm going through your example now. Only one problem for me, we are a
> Micosoft shop. Much of the skill set of the developers is Microsoft-only
> and
> very little Java experience. Having said that, I myself have developed in
> Java for 5-years, but that was seven years ago. While the language is not
> foreign to me, I need to try and build a system that others can maintain
> which means that I am looking to do everything in Spring DSL and not Java
> DSL. I "might" be able to take your concepts though and apply them to
> Spring
> DSL. We actually took a look at using NServiceBus, but fell in love with
> the
> Apache stack and all the Camel has to offer (+ServiceMi, ActiveMQ, etc).
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714186.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
I'm going through your example now. Only one problem for me, we are a
Micosoft shop. Much of the skill set of the developers is Microsoft-only and
very little Java experience. Having said that, I myself have developed in
Java for 5-years, but that was seven years ago. While the language is not
foreign to me, I need to try and build a system that others can maintain
which means that I am looking to do everything in Spring DSL and not Java
DSL. I "might" be able to take your concepts though and apply them to Spring
DSL. We actually took a look at using NServiceBus, but fell in love with the
Apache stack and all the Camel has to offer (+ServiceMi, ActiveMQ, etc).
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714186.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
I've been doing a little playing around with this problem today (and
this week). I'm using a "purgatory" queue to get the "poison" out of
the way temporarily. I go ahead and let the transactions rollback and
put the message back on the queue, but I "blacklist" the message so
that I know to send it to purgatory. I've got an example here:
https://github.com/jwcarman/smx-example/blob/master/camel/src/test/java/com/carmanconsulting/smx/example/camel/route/TestPurgatoryIdea.java
I plan on adding in route auto-suspension, also. If a route sees too
many things going to purgatory, then it can suspend itself (using a
separate thread) and schedule a Timer task to come back and resume it
at some later point (hopefully whatever was wrong is now fixed).
Enjoy!
On Thu, Jun 7, 2012 at 12:26 PM, James Carman
<ja...@carmanconsulting.com> wrote:
> Sorry, didn't see the rest of your message. Anyway, the problem
> you're going to face here is that suppose you do some stuff (database
> inserts, place messages on queues, etc.) between your input queue and
> your web service. That "stuff" needs to get rolled back when you have
> an issue and route your message to the DLC. If you tell Camel you've
> "handled" the exception, the transaction doesn't rollback and your
> message goes to DLC. So, you are sending your message to the
> dead-letter channel where you may potentially replay it later and some
> of the "stuff" that needs to go on during the processing of that
> message has already taken place as a result of the committed
> transaction. Therein lies the issue.
>
> On Thu, Jun 7, 2012 at 12:23 PM, James Carman
> <ja...@carmanconsulting.com> wrote:
>> Your transaction isn't rolling back if you "handle" the exception, is it?
>>
>> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <gr...@gmail.com> wrote:
>>> I have tested the case of using a route specific onException clause within a
>>> transaction and it appears to work as I would expect (or hope). So I have a
>>> route that is transactional and the final endpoint in the route throws an
>>> exception I forced my restful service to just throw an exception). Without
>>> the onException clause the message lands back in the queue as you would
>>> expect due to it running within a transaction. With the onException clause,
>>> I look for specific exceptions and if it is one of the exceptions that I
>>> have specified I tell tell Camel that the exception has been "handled" (via
>>> the handled clause) and I route the message to the dead letter queue, thus
>>> moving the "bad message" out of the way of the messages remaining on the
>>> queue. I think the key here is the use of the "handled" clause that tells
>>> Camel that the message has been handled and therefore to NOT rollback the
>>> transaction. The alternative choice is to tell Camel to "continue" on with
>>> its normal processing which would have rolled back the transaction and put
>>> the message back onto the queue (via the "continue" clause...at least I
>>> think it is a clause).
>>>
>>> Hope that helps.
>>>
>>> --
>>> View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
Sorry, didn't see the rest of your message. Anyway, the problem
you're going to face here is that suppose you do some stuff (database
inserts, place messages on queues, etc.) between your input queue and
your web service. That "stuff" needs to get rolled back when you have
an issue and route your message to the DLC. If you tell Camel you've
"handled" the exception, the transaction doesn't rollback and your
message goes to DLC. So, you are sending your message to the
dead-letter channel where you may potentially replay it later and some
of the "stuff" that needs to go on during the processing of that
message has already taken place as a result of the committed
transaction. Therein lies the issue.
On Thu, Jun 7, 2012 at 12:23 PM, James Carman
<ja...@carmanconsulting.com> wrote:
> Your transaction isn't rolling back if you "handle" the exception, is it?
>
> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <gr...@gmail.com> wrote:
>> I have tested the case of using a route specific onException clause within a
>> transaction and it appears to work as I would expect (or hope). So I have a
>> route that is transactional and the final endpoint in the route throws an
>> exception I forced my restful service to just throw an exception). Without
>> the onException clause the message lands back in the queue as you would
>> expect due to it running within a transaction. With the onException clause,
>> I look for specific exceptions and if it is one of the exceptions that I
>> have specified I tell tell Camel that the exception has been "handled" (via
>> the handled clause) and I route the message to the dead letter queue, thus
>> moving the "bad message" out of the way of the messages remaining on the
>> queue. I think the key here is the use of the "handled" clause that tells
>> Camel that the message has been handled and therefore to NOT rollback the
>> transaction. The alternative choice is to tell Camel to "continue" on with
>> its normal processing which would have rolled back the transaction and put
>> the message back onto the queue (via the "continue" clause...at least I
>> think it is a clause).
>>
>> Hope that helps.
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
Your transaction isn't rolling back if you "handle" the exception, is it?
On Thu, Jun 7, 2012 at 12:21 PM, gramanero <gr...@gmail.com> wrote:
> I have tested the case of using a route specific onException clause within a
> transaction and it appears to work as I would expect (or hope). So I have a
> route that is transactional and the final endpoint in the route throws an
> exception I forced my restful service to just throw an exception). Without
> the onException clause the message lands back in the queue as you would
> expect due to it running within a transaction. With the onException clause,
> I look for specific exceptions and if it is one of the exceptions that I
> have specified I tell tell Camel that the exception has been "handled" (via
> the handled clause) and I route the message to the dead letter queue, thus
> moving the "bad message" out of the way of the messages remaining on the
> queue. I think the key here is the use of the "handled" clause that tells
> Camel that the message has been handled and therefore to NOT rollback the
> transaction. The alternative choice is to tell Camel to "continue" on with
> its normal processing which would have rolled back the transaction and put
> the message back onto the queue (via the "continue" clause...at least I
> think it is a clause).
>
> Hope that helps.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
I have tested the case of using a route specific onException clause within a
transaction and it appears to work as I would expect (or hope). So I have a
route that is transactional and the final endpoint in the route throws an
exception I forced my restful service to just throw an exception). Without
the onException clause the message lands back in the queue as you would
expect due to it running within a transaction. With the onException clause,
I look for specific exceptions and if it is one of the exceptions that I
have specified I tell tell Camel that the exception has been "handled" (via
the handled clause) and I route the message to the dead letter queue, thus
moving the "bad message" out of the way of the messages remaining on the
queue. I think the key here is the use of the "handled" clause that tells
Camel that the message has been handled and therefore to NOT rollback the
transaction. The alternative choice is to tell Camel to "continue" on with
its normal processing which would have rolled back the transaction and put
the message back onto the queue (via the "continue" clause...at least I
think it is a clause).
Hope that helps.
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <jc...@carmanconsulting.com>.
Does onException() happen within the transaction too? If so then your
rollback will rollback your DLC put also.
On Jun 7, 2012 9:18 AM, "gramanero" <gr...@gmail.com> wrote:
> Yes, that is correct and actually in my case that "may" be what I want to
> have happen (i.e. the message gets put back onto queue A). However
> yesterday
> I did some digging into the onException clause and I'm finding it very
> useful. For example, if when "something bad happens" an exception is
> thrown,
> I can essentially catch that exception in Spring DSL and if it is a
> specific
> exception that I an looking for then I can decide what to do with the
> message. I can put it onto the deadletter queue, perhaps in the case of an
> argument exception, or I can stick it back onto the queue and have the
> route
> utilize my redelivery policy, especially if the exception was a
> connectivity
> issue with the cxfrs endpoint that I am trying to route the message to.
>
> Putting the message onto the DLQ move "poison pills" out of the way for the
> good messages to be routed to the cxfrs endpoint thus allowing message to
> flow through the route again. Seems to work pretty well. The issue I have
> now is that my cxfrs endpoint is a .NET restful service and if I throw a
> .NET exception from the service, it does not get translated to an
> equivalent
> java exception. Makes sense because I'm not sure that there would be a
> simple translation, but it does make it more difficult to determine which
> exceptions I want to look for a then route to the DLQ.
>
> Not sure if that helps you, but it seems to be a possible approach for me.
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714128.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
Yes, that is correct and actually in my case that "may" be what I want to
have happen (i.e. the message gets put back onto queue A). However yesterday
I did some digging into the onException clause and I'm finding it very
useful. For example, if when "something bad happens" an exception is thrown,
I can essentially catch that exception in Spring DSL and if it is a specific
exception that I an looking for then I can decide what to do with the
message. I can put it onto the deadletter queue, perhaps in the case of an
argument exception, or I can stick it back onto the queue and have the route
utilize my redelivery policy, especially if the exception was a connectivity
issue with the cxfrs endpoint that I am trying to route the message to.
Putting the message onto the DLQ move "poison pills" out of the way for the
good messages to be routed to the cxfrs endpoint thus allowing message to
flow through the route again. Seems to work pretty well. The issue I have
now is that my cxfrs endpoint is a .NET restful service and if I throw a
.NET exception from the service, it does not get translated to an equivalent
java exception. Makes sense because I'm not sure that there would be a
simple translation, but it does make it more difficult to determine which
exceptions I want to look for a then route to the DLQ.
Not sure if that helps you, but it seems to be a possible approach for me.
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714128.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
On Wed, Jun 6, 2012 at 11:35 PM, Claus Ibsen <cl...@gmail.com> wrote:
>
> Nothing is reliable.
>
> You would need to use for example Camel's dead letter channel, and
> send the messages to another JMS queue, which is the DLQ.
> And then use transactions so it appears as a atomic operation.
With transactions and DLC both playing at the same time, you run into
troubles, though. Suppose you have a route from queue A -> queue B
which does a bunch of "stuff" in the middle which you want to have
atomic (database inserts, putting messages on other queues, etc.).
Thus, you use transactions of course. Now, if something happens along
the way, the transaction will rollback. Rolling back the transaction
would also cause you to put the message back onto queue A. Then
you're right back where you started.
Re: Transacted vs DeadLetterQueue
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Jun 6, 2012 at 6:13 PM, James Carman <ja...@carmanconsulting.com> wrote:
> What if I want to do this the "camel way"? Is there any support in Camel
> to handle this type of situation reliably without losing messages?
>
Nothing is reliable.
You would need to use for example Camel's dead letter channel, and
send the messages to another JMS queue, which is the DLQ.
And then use transactions so it appears as a atomic operation.
> On Wed, Jun 6, 2012 at 12:11 PM, Claus Ibsen <cl...@gmail.com> wrote:
>
>> On Wed, Jun 6, 2012 at 6:08 PM, James Carman <ja...@carmanconsulting.com>
>> wrote:
>> > What happens if you have a "poisonous" (just plain bad) message on the
>> queue
>> > and not a system outage? You'll continually put the message back on the
>> > queue (by rolling back your transaction) and retry it, thereby blocking
>> all
>> > the subsequent messages in the queue. Right?
>> >
>>
>> Yes, but a broker usually have a way to configure a upper cap, and
>> move the poisonous message to a DLQ.
>> This is broker specific and you need to read up on what your broker
>> support.
>>
>>
>> > --
>> > View this message in context:
>> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714078.html
>> > Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
What if I want to do this the "camel way"? Is there any support in Camel
to handle this type of situation reliably without losing messages?
On Wed, Jun 6, 2012 at 12:11 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Wed, Jun 6, 2012 at 6:08 PM, James Carman <ja...@carmanconsulting.com>
> wrote:
> > What happens if you have a "poisonous" (just plain bad) message on the
> queue
> > and not a system outage? You'll continually put the message back on the
> > queue (by rolling back your transaction) and retry it, thereby blocking
> all
> > the subsequent messages in the queue. Right?
> >
>
> Yes, but a broker usually have a way to configure a upper cap, and
> move the poisonous message to a DLQ.
> This is broker specific and you need to read up on what your broker
> support.
>
>
> > --
> > View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714078.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Re: Transacted vs DeadLetterQueue
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Jun 6, 2012 at 6:08 PM, James Carman <ja...@carmanconsulting.com> wrote:
> What happens if you have a "poisonous" (just plain bad) message on the queue
> and not a system outage? You'll continually put the message back on the
> queue (by rolling back your transaction) and retry it, thereby blocking all
> the subsequent messages in the queue. Right?
>
Yes, but a broker usually have a way to configure a upper cap, and
move the poisonous message to a DLQ.
This is broker specific and you need to read up on what your broker support.
> --
> View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714078.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Transacted vs DeadLetterQueue
Posted by James Carman <ja...@carmanconsulting.com>.
What happens if you have a "poisonous" (just plain bad) message on the queue
and not a system outage? You'll continually put the message back on the
queue (by rolling back your transaction) and retry it, thereby blocking all
the subsequent messages in the queue. Right?
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714078.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by gramanero <gr...@gmail.com>.
Thanks for the response Christian.
I have been reading through the Camel in Action and ActiveMQ in Action books
most of the day today. I'm starting to get a better understanding of how
transactions work and that definitely seems like the way to go. Also, seems
like the DLQ and transactions work together, so that I can dump off messages
in the DLQ if "something bad happens". I think what I want in my case is to
set up the redelivery policy so that messages stay on the queue until the
cxfrs endpoint becomes available again. Looks like I can achieve this with a
value of -1 for the maximumRedelivery configuration. I just need to
determine a good balance for the other settings (i.e. redeliveryDelay) so
that I'm not trashing the system by constantly trying to resend messages to
an unavailable endpoint.
Can someone tell me what the useCollisionAvoidence setting is for? The only
explanation that I can find is:
"Should the redelivery policy use collision avoidance"
This really isn't telling me a whole lot. I can't seem to find anything in
either book that discusses it either.
Thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714032.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Posted by Christian Müller <ch...@gmail.com>.
If you should not loose messages, you should use a transacted route and
leverage on the ActiveMQ redelivery capabilities [1].
[1] http://activemq.apache.org/redelivery-policy.html
Best,
Christian
On Mon, Jun 4, 2012 at 9:51 PM, gramanero <gr...@gmail.com> wrote:
> I'm wondering if there are any sort of best practices or pros/cons to using
> either the dead letter queue or transactions when errors occur while
> pulling
> off messages from a queue and attempting to route them to a cxfrs endpoint
> that may not be available.
>
> For example, say I have the following route:
>
> <route id="myRoute" startupOrder="1" >
> <from uri="activemq:queue:myQueue"/>
> <to uri="cxfrs:http://localhost:9999/MyService" pattern="InOnly"/>
> </route>
>
> I have played around with using the dead letter queue and that seems to
> work
> well, but then I started to think about getting the messages off the dead
> letter queue and putting them back onto myQueue for processing once the
> service becomes available. One way to accomplish this (possibly) is to use
> the CronScheduledRoutePolicy such that it wakes up every hour and starts a
> route that will pull messages off the dead letter queue (if there are any),
> put them onto the myQueue and then stop the route after N number of
> minutes.
> I'm not sure that this is a good idea, but it "one" way to build a more
> auto-maintained system. Seems like once I start getting into configuring
> the
> CronScheduledRoutePolicy that the solution becomes more complex then it
> really needs to be in terms of the various moving parts.
>
> The other thought is to use the ActiveMQ transaction manager. If the cxfrs
> endpoint is not available, then the transaction rolls back which puts the
> message back on the queue (or just never really pulls it off) and then
> waits
> a while (via a delay configuration) and tries to route the message again.
> One thing I don't really like about this is that if the service is down for
> a long period of time, then it seems like we will be in this mode where
> messages are constantly trying to be routed to the unavailable cxfrs
> endpoint thus sort of thrashing the system.
>
> I'm just looking for some opinions to how this type of problem is typically
> solved or some sort of best practice to this use case.
>
> Thanks!
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>