You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Li Li <fa...@gmail.com> on 2014/02/24 12:07:19 UTC
does AUTO_ACKNOWLEDGE guarantee once and only once semantic for queue consumer?
I've read the jms specification and questions by google such as
http://stackoverflow.com/questions/2991412/anyone-know-exactly-which-jms-messages-will-be-redelivered-in-client-acknowledge
. But I still confused about it.
Another question is: how can a consumer tell the broker to
redelevery a message to other consumers(maybe again to this consumer)
e.g. we have some real time tasks and here is code:
class MyConsumer implements MessageListener{
public void onMessage(Message msg) {
doPhrase1();
if(leftTime < avgTime1){
//Tell broker I can't finish it in time,
//you should redelevery to other consumers
//do some cleanup job
}
doPhrase2();
if(leftTime < avgTime1){
//Tell broker I can't finish it in time,
//you should redelevery to other consumers
//do some cleanup job
}
doPhrase3();
}
Re: does AUTO_ACKNOWLEDGE guarantee once and only once semantic for
queue consumer?
Posted by Gary Tully <ga...@gmail.com>.
in the absense of xa, I think we can do better with the redeliveryflag
such that it can be used as a reliable indication of duplicate
dispatch. In that way, an application can avoid doing duplicate checks
when the flag is not set, even after a broker failure.
It will come as a cost, but it opens up a new class of usecases.
see: https://issues.apache.org/jira/browse/AMQ-5068
On 25 February 2014 02:54, David Jencks <da...@yahoo.com> wrote:
> Using activemq with a jta transaction manager and using the appropriate xa interfaces (such as through the resource adapter) will guarantee that a message is delivered exactly once. Many people prefer to use idempotent messages to avoid the xa overhead.
>
> david jencks
>
>
> On Feb 24, 2014, at 6:45 PM, Li Li <fa...@gmail.com> wrote:
>
>> thanks. Since JMS can't guarantee only once, it's the client's
>> responsibility to make things correct. it should either using
>> idempotent processing(running a task twice is no harm) or using other
>> tools(such as zookeeper or database to make data consistent). So I
>> think there is still many dirty work to do in some strict environment.
>> is there any practical usage of JMS(activemq) in real world in
>> industry such as financial or e-commerce?
>> on the other hand, does JMS guarantee at least once delivery? What's
>> the redelevery mean?
>> as said in https://activemq.apache.org/message-redelivery-and-dlq-handling.html
>>
>> Messages are redelivered to a client when any of the following occurs:
>>
>> A transacted session is used and rollback() is called.
>> A transacted session is closed before commit is called.
>> A session is using CLIENT_ACKNOWLEDGE and Session.recover() is called.
>>
>> I notice that when a client crashed in the onMessage method and
>> restart. the message will be dispatched to another consumer with
>> getJMSRedelivered()==true. it this message a redelivered one?
>> also, if the consumer is very slow(maybe hang for an hour,stay in
>> onMessage so it's not acknowledged) and the message has a short TTL,
>> will it be redelivered to another consumer or to dead letter queue?
>>
>>
>> On Tue, Feb 25, 2014 at 10:10 AM, artnaseef <ar...@artnaseef.com> wrote:
>>> The use-case sounds like one that needs more than just ActiveMQ to solve
>>> well.
>>>
>>> In JMS, redeliveries of messages must be expected and handled by
>>> applications. There are cases where the JMS redelivered property is set,
>>> but the application may not have touched the message ever before - such as
>>> when the message is dispatched in the prefetch for the client and the client
>>> terminates without getting to the message. Even with prefetch 0, the same
>>> problem can happen.
>>>
>>> The JMS redelivered property is a hint that lets an application know that
>>> there is a risk the message was actually handled before.
>>>
>>> My preference is to think of JMS guarantees as a great way to simplify the
>>> normal processing path in an application as it eliminates the need to
>>> consider how messages can get lost in transit and protecting against the
>>> same. However, it's not intended to be a way to signal to the broker to try
>>> a different consumer. And it doesn't eliminate considerations under error
>>> cases - such as a consumer terminating in the middle of processing the
>>> message.
>>>
>>> On the practical front, rejecting a message so that the broker resends it
>>> will introduce notable delay. It may also result in the message being
>>> redelivered to the same client. Also, I would need to check -- there may
>>> also be a TTL issue in which case the message could end up stuck on the
>>> broker.
>>>
>>> In your use-case, I would consider ActiveMQ as an underlying component to a
>>> complete solution. Perhaps using camel with a server that routes tasks to
>>> task engines. Also, I recommend considering a proactive approach to the
>>> problem. Waiting until the task reaches a task engine to have it rejected
>>> may lead to problems. For example, what if the fastest task engine is too
>>> slow for the task? The task will starve. Using feedback from the task
>>> engines to the routing engine, the routing engine can decide the best task
>>> engine using heuristics (such as best fit - not always the fastest engine).
>>>
>>> To get the guarantee a task only ever executes once, you'll need a reliable
>>> lock/state engine. Zookeeper comes to mind. Note that many applications
>>> prefer to use an idempotent processing model so that duplicate processing
>>> causes no problems.
>>>
>>> Hope this helps.
>>>
>>>
>>>
>>> --
>>> View this message in context: http://activemq.2283324.n4.nabble.com/does-AUTO-ACKNOWLEDGE-guarantee-once-and-only-once-semantic-for-queue-consumer-tp4678300p4678338.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://redhat.com
http://blog.garytully.com
Re: does AUTO_ACKNOWLEDGE guarantee once and only once semantic for queue consumer?
Posted by David Jencks <da...@yahoo.com>.
Using activemq with a jta transaction manager and using the appropriate xa interfaces (such as through the resource adapter) will guarantee that a message is delivered exactly once. Many people prefer to use idempotent messages to avoid the xa overhead.
david jencks
On Feb 24, 2014, at 6:45 PM, Li Li <fa...@gmail.com> wrote:
> thanks. Since JMS can't guarantee only once, it's the client's
> responsibility to make things correct. it should either using
> idempotent processing(running a task twice is no harm) or using other
> tools(such as zookeeper or database to make data consistent). So I
> think there is still many dirty work to do in some strict environment.
> is there any practical usage of JMS(activemq) in real world in
> industry such as financial or e-commerce?
> on the other hand, does JMS guarantee at least once delivery? What's
> the redelevery mean?
> as said in https://activemq.apache.org/message-redelivery-and-dlq-handling.html
>
> Messages are redelivered to a client when any of the following occurs:
>
> A transacted session is used and rollback() is called.
> A transacted session is closed before commit is called.
> A session is using CLIENT_ACKNOWLEDGE and Session.recover() is called.
>
> I notice that when a client crashed in the onMessage method and
> restart. the message will be dispatched to another consumer with
> getJMSRedelivered()==true. it this message a redelivered one?
> also, if the consumer is very slow(maybe hang for an hour,stay in
> onMessage so it's not acknowledged) and the message has a short TTL,
> will it be redelivered to another consumer or to dead letter queue?
>
>
> On Tue, Feb 25, 2014 at 10:10 AM, artnaseef <ar...@artnaseef.com> wrote:
>> The use-case sounds like one that needs more than just ActiveMQ to solve
>> well.
>>
>> In JMS, redeliveries of messages must be expected and handled by
>> applications. There are cases where the JMS redelivered property is set,
>> but the application may not have touched the message ever before - such as
>> when the message is dispatched in the prefetch for the client and the client
>> terminates without getting to the message. Even with prefetch 0, the same
>> problem can happen.
>>
>> The JMS redelivered property is a hint that lets an application know that
>> there is a risk the message was actually handled before.
>>
>> My preference is to think of JMS guarantees as a great way to simplify the
>> normal processing path in an application as it eliminates the need to
>> consider how messages can get lost in transit and protecting against the
>> same. However, it's not intended to be a way to signal to the broker to try
>> a different consumer. And it doesn't eliminate considerations under error
>> cases - such as a consumer terminating in the middle of processing the
>> message.
>>
>> On the practical front, rejecting a message so that the broker resends it
>> will introduce notable delay. It may also result in the message being
>> redelivered to the same client. Also, I would need to check -- there may
>> also be a TTL issue in which case the message could end up stuck on the
>> broker.
>>
>> In your use-case, I would consider ActiveMQ as an underlying component to a
>> complete solution. Perhaps using camel with a server that routes tasks to
>> task engines. Also, I recommend considering a proactive approach to the
>> problem. Waiting until the task reaches a task engine to have it rejected
>> may lead to problems. For example, what if the fastest task engine is too
>> slow for the task? The task will starve. Using feedback from the task
>> engines to the routing engine, the routing engine can decide the best task
>> engine using heuristics (such as best fit - not always the fastest engine).
>>
>> To get the guarantee a task only ever executes once, you'll need a reliable
>> lock/state engine. Zookeeper comes to mind. Note that many applications
>> prefer to use an idempotent processing model so that duplicate processing
>> causes no problems.
>>
>> Hope this helps.
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/does-AUTO-ACKNOWLEDGE-guarantee-once-and-only-once-semantic-for-queue-consumer-tp4678300p4678338.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: does AUTO_ACKNOWLEDGE guarantee once and only once semantic for
queue consumer?
Posted by Li Li <fa...@gmail.com>.
thanks. Since JMS can't guarantee only once, it's the client's
responsibility to make things correct. it should either using
idempotent processing(running a task twice is no harm) or using other
tools(such as zookeeper or database to make data consistent). So I
think there is still many dirty work to do in some strict environment.
is there any practical usage of JMS(activemq) in real world in
industry such as financial or e-commerce?
on the other hand, does JMS guarantee at least once delivery? What's
the redelevery mean?
as said in https://activemq.apache.org/message-redelivery-and-dlq-handling.html
Messages are redelivered to a client when any of the following occurs:
A transacted session is used and rollback() is called.
A transacted session is closed before commit is called.
A session is using CLIENT_ACKNOWLEDGE and Session.recover() is called.
I notice that when a client crashed in the onMessage method and
restart. the message will be dispatched to another consumer with
getJMSRedelivered()==true. it this message a redelivered one?
also, if the consumer is very slow(maybe hang for an hour,stay in
onMessage so it's not acknowledged) and the message has a short TTL,
will it be redelivered to another consumer or to dead letter queue?
On Tue, Feb 25, 2014 at 10:10 AM, artnaseef <ar...@artnaseef.com> wrote:
> The use-case sounds like one that needs more than just ActiveMQ to solve
> well.
>
> In JMS, redeliveries of messages must be expected and handled by
> applications. There are cases where the JMS redelivered property is set,
> but the application may not have touched the message ever before - such as
> when the message is dispatched in the prefetch for the client and the client
> terminates without getting to the message. Even with prefetch 0, the same
> problem can happen.
>
> The JMS redelivered property is a hint that lets an application know that
> there is a risk the message was actually handled before.
>
> My preference is to think of JMS guarantees as a great way to simplify the
> normal processing path in an application as it eliminates the need to
> consider how messages can get lost in transit and protecting against the
> same. However, it's not intended to be a way to signal to the broker to try
> a different consumer. And it doesn't eliminate considerations under error
> cases - such as a consumer terminating in the middle of processing the
> message.
>
> On the practical front, rejecting a message so that the broker resends it
> will introduce notable delay. It may also result in the message being
> redelivered to the same client. Also, I would need to check -- there may
> also be a TTL issue in which case the message could end up stuck on the
> broker.
>
> In your use-case, I would consider ActiveMQ as an underlying component to a
> complete solution. Perhaps using camel with a server that routes tasks to
> task engines. Also, I recommend considering a proactive approach to the
> problem. Waiting until the task reaches a task engine to have it rejected
> may lead to problems. For example, what if the fastest task engine is too
> slow for the task? The task will starve. Using feedback from the task
> engines to the routing engine, the routing engine can decide the best task
> engine using heuristics (such as best fit - not always the fastest engine).
>
> To get the guarantee a task only ever executes once, you'll need a reliable
> lock/state engine. Zookeeper comes to mind. Note that many applications
> prefer to use an idempotent processing model so that duplicate processing
> causes no problems.
>
> Hope this helps.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/does-AUTO-ACKNOWLEDGE-guarantee-once-and-only-once-semantic-for-queue-consumer-tp4678300p4678338.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: does AUTO_ACKNOWLEDGE guarantee once and only once semantic for
queue consumer?
Posted by artnaseef <ar...@artnaseef.com>.
The use-case sounds like one that needs more than just ActiveMQ to solve
well.
In JMS, redeliveries of messages must be expected and handled by
applications. There are cases where the JMS redelivered property is set,
but the application may not have touched the message ever before - such as
when the message is dispatched in the prefetch for the client and the client
terminates without getting to the message. Even with prefetch 0, the same
problem can happen.
The JMS redelivered property is a hint that lets an application know that
there is a risk the message was actually handled before.
My preference is to think of JMS guarantees as a great way to simplify the
normal processing path in an application as it eliminates the need to
consider how messages can get lost in transit and protecting against the
same. However, it's not intended to be a way to signal to the broker to try
a different consumer. And it doesn't eliminate considerations under error
cases - such as a consumer terminating in the middle of processing the
message.
On the practical front, rejecting a message so that the broker resends it
will introduce notable delay. It may also result in the message being
redelivered to the same client. Also, I would need to check -- there may
also be a TTL issue in which case the message could end up stuck on the
broker.
In your use-case, I would consider ActiveMQ as an underlying component to a
complete solution. Perhaps using camel with a server that routes tasks to
task engines. Also, I recommend considering a proactive approach to the
problem. Waiting until the task reaches a task engine to have it rejected
may lead to problems. For example, what if the fastest task engine is too
slow for the task? The task will starve. Using feedback from the task
engines to the routing engine, the routing engine can decide the best task
engine using heuristics (such as best fit - not always the fastest engine).
To get the guarantee a task only ever executes once, you'll need a reliable
lock/state engine. Zookeeper comes to mind. Note that many applications
prefer to use an idempotent processing model so that duplicate processing
causes no problems.
Hope this helps.
--
View this message in context: http://activemq.2283324.n4.nabble.com/does-AUTO-ACKNOWLEDGE-guarantee-once-and-only-once-semantic-for-queue-consumer-tp4678300p4678338.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: does AUTO_ACKNOWLEDGE guarantee once and only once semantic for
queue consumer?
Posted by Yin Wang <su...@gmail.com>.
BTW,AUTO_ACKNOWLEDGE can not guarantee once and only once semantic since
the ack is sent async.
You can image such a case when your consumer has succeeded in processing a
message
while the consumer application is crashed before the ack is sent, you will
receive that message again if your applications get restarted.
Seems what you really want to solve is to distribute real task messages
evenly between consumers.
As default a consumer has a prefetch to allow broker to push messages to it
in batch manner,
if a consumer is slow to process message then many pending messages will be
accumulated
and other consumers may suffer starvation.In this case those real tasks
wont get executed in time.
You can just use pull mechanism for your consumers so that your consumer
can pull real task message from broker one by one.
Fast consumers can have a chance to pull more messages than slow consumers.
If consumers still can not process real tasks in time, increase consumers
or tune performance for consumers.
2014-02-24 19:07 GMT+08:00 Li Li <fa...@gmail.com>:
> I've read the jms specification and questions by google such as
>
> http://stackoverflow.com/questions/2991412/anyone-know-exactly-which-jms-messages-will-be-redelivered-in-client-acknowledge
> . But I still confused about it.
> Another question is: how can a consumer tell the broker to
> redelevery a message to other consumers(maybe again to this consumer)
> e.g. we have some real time tasks and here is code:
> class MyConsumer implements MessageListener{
>
> public void onMessage(Message msg) {
> doPhrase1();
> if(leftTime < avgTime1){
> //Tell broker I can't finish it in time,
> //you should redelevery to other consumers
> //do some cleanup job
> }
> doPhrase2();
> if(leftTime < avgTime1){
> //Tell broker I can't finish it in time,
> //you should redelevery to other consumers
> //do some cleanup job
> }
> doPhrase3();
> }
>
Re: does AUTO_ACKNOWLEDGE guarantee once and only once semantic for
queue consumer?
Posted by Li Li <fa...@gmail.com>.
thanks, there is another question: I can I make sure a task is
processed once and only once?
do I need do like this:
onMessage(){
if(msg.getJMSRedelivered()){
//check some central status system such as database to see whether
it's finished
//if finished
return;
else //partial done
undo partial work
do real work
}else{
do real work
}
On Mon, Feb 24, 2014 at 10:55 PM, Yin Wang <su...@gmail.com> wrote:
> Seems what you really want to solve is to distribute real task messages
> evenly between consumers.
> As default a consumer has a prefetch to allow broker to push messages to it
> in batch manner,
> if a consumer is slow to process message then many pending messages will be
> accumulated
> and other consumers may suffer starvation.In this case those real tasks
> wont get executed in time.
> You can just use pull mechanism for your consumers so that your consumer
> can pull real task message from broker one by one.
> Fast consumers can have a chance to pull more messages than slow consumers.
> If consumers still can not process real tasks in time, increase consumers
> or tune performance for consumers.
>
>
> 2014-02-24 19:07 GMT+08:00 Li Li <fa...@gmail.com>:
>
>> I've read the jms specification and questions by google such as
>>
>> http://stackoverflow.com/questions/2991412/anyone-know-exactly-which-jms-messages-will-be-redelivered-in-client-acknowledge
>> . But I still confused about it.
>> Another question is: how can a consumer tell the broker to
>> redelevery a message to other consumers(maybe again to this consumer)
>> e.g. we have some real time tasks and here is code:
>> class MyConsumer implements MessageListener{
>>
>> public void onMessage(Message msg) {
>> doPhrase1();
>> if(leftTime < avgTime1){
>> //Tell broker I can't finish it in time,
>> //you should redelevery to other consumers
>> //do some cleanup job
>> }
>> doPhrase2();
>> if(leftTime < avgTime1){
>> //Tell broker I can't finish it in time,
>> //you should redelevery to other consumers
>> //do some cleanup job
>> }
>> doPhrase3();
>> }
>>
Re: does AUTO_ACKNOWLEDGE guarantee once and only once semantic for
queue consumer?
Posted by Yin Wang <su...@gmail.com>.
Seems what you really want to solve is to distribute real task messages
evenly between consumers.
As default a consumer has a prefetch to allow broker to push messages to it
in batch manner,
if a consumer is slow to process message then many pending messages will be
accumulated
and other consumers may suffer starvation.In this case those real tasks
wont get executed in time.
You can just use pull mechanism for your consumers so that your consumer
can pull real task message from broker one by one.
Fast consumers can have a chance to pull more messages than slow consumers.
If consumers still can not process real tasks in time, increase consumers
or tune performance for consumers.
2014-02-24 19:07 GMT+08:00 Li Li <fa...@gmail.com>:
> I've read the jms specification and questions by google such as
>
> http://stackoverflow.com/questions/2991412/anyone-know-exactly-which-jms-messages-will-be-redelivered-in-client-acknowledge
> . But I still confused about it.
> Another question is: how can a consumer tell the broker to
> redelevery a message to other consumers(maybe again to this consumer)
> e.g. we have some real time tasks and here is code:
> class MyConsumer implements MessageListener{
>
> public void onMessage(Message msg) {
> doPhrase1();
> if(leftTime < avgTime1){
> //Tell broker I can't finish it in time,
> //you should redelevery to other consumers
> //do some cleanup job
> }
> doPhrase2();
> if(leftTime < avgTime1){
> //Tell broker I can't finish it in time,
> //you should redelevery to other consumers
> //do some cleanup job
> }
> doPhrase3();
> }
>