You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Se...@troika.ru on 2011/03/22 14:03:34 UTC
Qpid CLIENT_ACKNOWLEDGE
Hi there
I'm working with qpid 0.8 by means of JMS API and I'm using
CLIENT_ACKNOWLEDGE mode to get higher performance comparing to
transactinal processing.
As a rule performance is rather pure in both cases. In transactional mode
I can receive about 1.6K messages per second, in CLIENT_ACKNOWLEDGE mode
I'm able to receive about 3.6K messages per second and in AUTO_ACKNOWLEDGE
mode I'm able to receive about 70K messages per second.
Are there any recommendations how to improve performance and get a
guaranted message delivery?
Best Regards,
Sergey
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
Re: Qpid CLIENT_ACKNOWLEDGE
Posted by fadams <fr...@blueyonder.co.uk>.
Hello all,
I hate to contradict, but I noted figures of:
sync publish 15k msg/sec
tx size =2 48 msg/sec (thats right just 48 messages per sec)
tx size=1000 14k msg/sec
mentioned by Rajith in a previous post.
I seem to be getting something very different. I'm using Qpid 0.8 with C++
Broker and Java JMS Clients
With 1K message size I get
With no explicit sync (default behaviour) 21000 messages/sec
With sync_publish='all' 1420 messages/sec
With transacted session tx size = 1 1125 messages/sec
With transacted session tx size = 500 20750 messages/sec
With 50K message size I get
With no explicit sync (default behaviour) 1500 messages/sec
With sync_publish='all' 332 messages/sec
With transacted session tx size = 1 314 messages/sec
With transacted session tx size = 500 1550 messages/sec
I do wonder about the use of sync_publish though! When I use the connection
URL approach, e.g.
connectionfactory.ConnectionFactory =
amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672'&sync_publish='all'
it behaves as expected, however if I do
-Dsync_publish=all
there is no effect......
I had a quick grep through the Java client code and I suspect a bug in
AMQConnection.java as this has a line that says:
"_syncPublish =
System.getProperty((ClientProperties.SYNC_ACK_PROP_NAME),_syncPublish);"
The SYNC_ACK_PROP_NAME looks wrong and I guess should really say
SYNC_PUBLISH_PROP_NAME.
I re-read one of Rajith's previous posts and note that "Therefore I
recommend you use -Dsync_ack=persistent" is mentioned, but given the mixture
of that and "-Dsync_publish={persistent|transient|all}. " mentioned in the
same post I wonder if Rajith has perhaps accidentally used the (non-working)
-Dsync_publish=all option with the test giving the result "sync publish
15k msg/sec"?? so that value may really be for the default mode?
I'm also really surprised at the 48 messages/second result with a tx size of
2. As I say above even with 50K messages I'm still getting 314 messages per
second with Java client and C++ broker and tx size of 1 (I'm running clients
and broker on the same box, so I'd expect it to be even faster if I ran my
clients on a separate box).
Regards,
fadams
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Qpid-CLIENT-ACKNOWLEDGE-tp6196226p6210606.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Qpid CLIENT_ACKNOWLEDGE
Posted by Rajith Attapattu <ra...@gmail.com>.
Sorry was sick yesterday, catching up on the email now.
On Thu, Mar 24, 2011 at 6:42 AM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:
> Hi Rajith,
>
> Thanks (and oops).
>
> On the stats for sync-publish vs transactions, guessing these are all
> transient -> with the Java broker ?
>
>
Nope these are all durable with the C++ broker.
> For persistent messages, which you'd need for reliable delivery with the
> Java broker, the figures would be closer together since the I/O write is the
> larger cost.
>
Performance apart, using transactions IMO is not a good idea unless it's
extremely necessary.
>
> Thanks,
> Marnie
>
> On Wed, Mar 23, 2011 at 8:20 PM, Rajith Attapattu <ra...@gmail.com>wrote:
>
>> Marnie,
>>
>> Looks like you forgot to hit replyAll.
>> So I am forwarding your email to the list along with my comments.
>>
>>
>> On Wed, Mar 23, 2011 at 12:47 PM, Marnie McCormack <
>> marnie.mccormack@googlemail.com> wrote:
>>
>>> Sorry, which broker are you usiong Sergey ? I'd assumed the Java broker
>>> ...
>>>
>>
>> Since the Java broker supports 0-10 now, you could use sync_publish.
>>
>>
>>>
>>> I'd probably disagree that you can achieve guaranteed delivery without a
>>> transaction or its equivalent (which sync publish effectively is ?).
>>>
>>
>> Synchronous publishing is not equivalent to transactions, but I understand
>> what you meant here. You were probably alluding to the fact that in both
>> cases messages are published in a synchronous way. i.e When the tx completes
>> (or when the send method returns in sync-publish) you know that the broker
>> has acked the message.
>>
>> However transactions have a lot more overhead than a simple sync publish,
>> hence a higher performance penalty with transactions.
>> For example look at the following relative numbers for a producer/consumer
>> pair using the same setup.
>>
>> sync publish 15k msg/sec
>> tx size =2 48 msg/sec (thats right just 48 messages per sec)
>> tx size=1000 14k msg/sec
>>
>> Also from an application point of view, non transactional code is more
>> simple and easy to write than transactional code.
>> I would normally stay away from transactions unless I really have to. It
>> basically negates a lot of benefits of using messaging.
>>
>> Regards,
>>
>> Rajith
>>
>>
>>>
>>> Regards,
>>> Marnie
>>>
>>> On Wed, Mar 23, 2011 at 4:21 PM, Rajith Attapattu <ra...@gmail.com>wrote:
>>>
>>>> Hi Sergey,
>>>>
>>>> For Qpid to guarantee 'at-least-once' delivery on publish, you don't
>>>> need to use transactions.
>>>>
>>>> Broker failures
>>>> -----------------
>>>> (1) In a stand-alone-broker you need to use persistence to ensure that
>>>> messages survive a broker crash.
>>>> (2) In a clustered broker you may be able to get away without using
>>>> persistence (*)
>>>>
>>>> (*) Only the c++ broker supports clustering.
>>>> If you don't use persistence, then in the event of a total cluster
>>>> failure you will loose messages.
>>>> Therefore it depends on how much you can tolerate these occasional
>>>> failures.
>>>>
>>>> Application/Client lib failures
>>>> ---------------------------------
>>>> In the event of a failure in the application that sends the messages,
>>>> you may loose messages.
>>>> i.e if there are unacked messages in the client libs replay buffer, they
>>>> will be lost.
>>>> You could avoid that by using
>>>>
>>>> For JMS client
>>>> -------------------
>>>> (1) Using synchronous publishing. Use
>>>> -Dsync_publish={persistent|transient|all}.
>>>> (2) Transaction - as Marnie pointed out.
>>>>
>>>> Performance wise there isn't much of a difference between option 1 & 2
>>>> if you use large batches (ex 1000 msg/batch) for transactions.
>>>> But ff you use small tx batches then there is an appreciable difference
>>>> in performance.
>>>>
>>>> Therefore I recommend you use -Dsync_ack=persistent.
>>>> Also the application code is much simpler when writing non transactional
>>>> code.
>>>>
>>>> For other clients
>>>> --------------------
>>>> If you can handle reliable replay at the application level then you
>>>> could just use asynchronous publishing without worrying about application
>>>> failure.
>>>> Ex. If your application is retrieving orders from a queue and processing
>>>> it before sending a reply, then you could ack the order messsage after the
>>>> broker acks your reply message.
>>>>
>>>> Hope this helps.
>>>>
>>>> Rajith
>>>>
>>>>
>>>> On Wed, Mar 23, 2011 at 11:21 AM, Marnie McCormack <
>>>> marnie.mccormack@googlemail.com> wrote:
>>>>
>>>>> Hi Sergey,
>>>>>
>>>>> To achieve guaranteed message delivery you need to be using
>>>>> transactions on
>>>>> *publish*. Qpid then guarantees 'at least once' delivery. You'd offset
>>>>> the
>>>>> reliable delivery cost against the cost or message loss during a
>>>>> failure
>>>>> from sending non-transacted messages. The use of transactions and
>>>>> persistence for reliable messaging adds cost in the shape of I/O.
>>>>>
>>>>> However, you could opt not to use transactions on consume since you
>>>>> cannot
>>>>> conceptually lose a message during consumption - if there's a failure
>>>>> and a
>>>>> message ack doesn't get back to the broker then you'll simply get the
>>>>> message back again on restart. This assumes that you
>>>>> listen/catch/handle
>>>>> errors and exception on consume such that you don't have application
>>>>> level
>>>>> side effects.
>>>>>
>>>>> Regards,
>>>>> Marnie
>>>>>
>>>>> On Tue, Mar 22, 2011 at 1:03 PM, <Se...@troika.ru> wrote:
>>>>>
>>>>> > Hi there
>>>>> >
>>>>> > I'm working with qpid 0.8 by means of JMS API and I'm using
>>>>> > CLIENT_ACKNOWLEDGE mode to get higher performance comparing to
>>>>> > transactinal processing.
>>>>> > As a rule performance is rather pure in both cases. In transactional
>>>>> mode
>>>>> > I can receive about 1.6K messages per second, in CLIENT_ACKNOWLEDGE
>>>>> mode
>>>>> > I'm able to receive about 3.6K messages per second and in
>>>>> AUTO_ACKNOWLEDGE
>>>>> > mode I'm able to receive about 70K messages per second.
>>>>> >
>>>>> > Are there any recommendations how to improve performance and get a
>>>>> > guaranted message delivery?
>>>>> >
>>>>> >
>>>>> > Best Regards,
>>>>> > Sergey
>>>>> > _______________________________________________________
>>>>> >
>>>>> > The information contained in this message may be privileged and conf
>>>>> > idential and protected from disclosure. If you are not the original
>>>>> intended
>>>>> > recipient, you are hereby notified that any review, retransmission,
>>>>> > dissemination, or other use of, or taking of any action in reliance
>>>>> upon,
>>>>> > this information is prohibited. If you have received this
>>>>> communication in
>>>>> > error, please notify the sender immediately by replying to this
>>>>> message and
>>>>> > delete it from your computer. Thank you for your cooperation. Troika
>>>>> Dialog,
>>>>> > Russia.
>>>>> > If you need assistance please contact our Contact Center (+7495)
>>>>> 258 0500
>>>>> > or go to www.troika.ru/eng/Contacts/system.wbp
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Qpid CLIENT_ACKNOWLEDGE
Posted by Marnie McCormack <ma...@googlemail.com>.
Hi Rajith,
Thanks (and oops).
On the stats for sync-publish vs transactions, guessing these are all
transient -> with the Java broker ?
For persistent messages, which you'd need for reliable delivery with the
Java broker, the figures would be closer together since the I/O write is the
larger cost.
Thanks,
Marnie
On Wed, Mar 23, 2011 at 8:20 PM, Rajith Attapattu <ra...@gmail.com>wrote:
> Marnie,
>
> Looks like you forgot to hit replyAll.
> So I am forwarding your email to the list along with my comments.
>
>
> On Wed, Mar 23, 2011 at 12:47 PM, Marnie McCormack <
> marnie.mccormack@googlemail.com> wrote:
>
>> Sorry, which broker are you usiong Sergey ? I'd assumed the Java broker
>> ...
>>
>
> Since the Java broker supports 0-10 now, you could use sync_publish.
>
>
>>
>> I'd probably disagree that you can achieve guaranteed delivery without a
>> transaction or its equivalent (which sync publish effectively is ?).
>>
>
> Synchronous publishing is not equivalent to transactions, but I understand
> what you meant here. You were probably alluding to the fact that in both
> cases messages are published in a synchronous way. i.e When the tx completes
> (or when the send method returns in sync-publish) you know that the broker
> has acked the message.
>
> However transactions have a lot more overhead than a simple sync publish,
> hence a higher performance penalty with transactions.
> For example look at the following relative numbers for a producer/consumer
> pair using the same setup.
>
> sync publish 15k msg/sec
> tx size =2 48 msg/sec (thats right just 48 messages per sec)
> tx size=1000 14k msg/sec
>
> Also from an application point of view, non transactional code is more
> simple and easy to write than transactional code.
> I would normally stay away from transactions unless I really have to. It
> basically negates a lot of benefits of using messaging.
>
> Regards,
>
> Rajith
>
>
>>
>> Regards,
>> Marnie
>>
>> On Wed, Mar 23, 2011 at 4:21 PM, Rajith Attapattu <ra...@gmail.com>wrote:
>>
>>> Hi Sergey,
>>>
>>> For Qpid to guarantee 'at-least-once' delivery on publish, you don't need
>>> to use transactions.
>>>
>>> Broker failures
>>> -----------------
>>> (1) In a stand-alone-broker you need to use persistence to ensure that
>>> messages survive a broker crash.
>>> (2) In a clustered broker you may be able to get away without using
>>> persistence (*)
>>>
>>> (*) Only the c++ broker supports clustering.
>>> If you don't use persistence, then in the event of a total cluster
>>> failure you will loose messages.
>>> Therefore it depends on how much you can tolerate these occasional
>>> failures.
>>>
>>> Application/Client lib failures
>>> ---------------------------------
>>> In the event of a failure in the application that sends the messages, you
>>> may loose messages.
>>> i.e if there are unacked messages in the client libs replay buffer, they
>>> will be lost.
>>> You could avoid that by using
>>>
>>> For JMS client
>>> -------------------
>>> (1) Using synchronous publishing. Use
>>> -Dsync_publish={persistent|transient|all}.
>>> (2) Transaction - as Marnie pointed out.
>>>
>>> Performance wise there isn't much of a difference between option 1 & 2 if
>>> you use large batches (ex 1000 msg/batch) for transactions.
>>> But ff you use small tx batches then there is an appreciable difference
>>> in performance.
>>>
>>> Therefore I recommend you use -Dsync_ack=persistent.
>>> Also the application code is much simpler when writing non transactional
>>> code.
>>>
>>> For other clients
>>> --------------------
>>> If you can handle reliable replay at the application level then you could
>>> just use asynchronous publishing without worrying about application failure.
>>> Ex. If your application is retrieving orders from a queue and processing
>>> it before sending a reply, then you could ack the order messsage after the
>>> broker acks your reply message.
>>>
>>> Hope this helps.
>>>
>>> Rajith
>>>
>>>
>>> On Wed, Mar 23, 2011 at 11:21 AM, Marnie McCormack <
>>> marnie.mccormack@googlemail.com> wrote:
>>>
>>>> Hi Sergey,
>>>>
>>>> To achieve guaranteed message delivery you need to be using transactions
>>>> on
>>>> *publish*. Qpid then guarantees 'at least once' delivery. You'd offset
>>>> the
>>>> reliable delivery cost against the cost or message loss during a failure
>>>> from sending non-transacted messages. The use of transactions and
>>>> persistence for reliable messaging adds cost in the shape of I/O.
>>>>
>>>> However, you could opt not to use transactions on consume since you
>>>> cannot
>>>> conceptually lose a message during consumption - if there's a failure
>>>> and a
>>>> message ack doesn't get back to the broker then you'll simply get the
>>>> message back again on restart. This assumes that you listen/catch/handle
>>>> errors and exception on consume such that you don't have application
>>>> level
>>>> side effects.
>>>>
>>>> Regards,
>>>> Marnie
>>>>
>>>> On Tue, Mar 22, 2011 at 1:03 PM, <Se...@troika.ru> wrote:
>>>>
>>>> > Hi there
>>>> >
>>>> > I'm working with qpid 0.8 by means of JMS API and I'm using
>>>> > CLIENT_ACKNOWLEDGE mode to get higher performance comparing to
>>>> > transactinal processing.
>>>> > As a rule performance is rather pure in both cases. In transactional
>>>> mode
>>>> > I can receive about 1.6K messages per second, in CLIENT_ACKNOWLEDGE
>>>> mode
>>>> > I'm able to receive about 3.6K messages per second and in
>>>> AUTO_ACKNOWLEDGE
>>>> > mode I'm able to receive about 70K messages per second.
>>>> >
>>>> > Are there any recommendations how to improve performance and get a
>>>> > guaranted message delivery?
>>>> >
>>>> >
>>>> > Best Regards,
>>>> > Sergey
>>>> > _______________________________________________________
>>>> >
>>>> > The information contained in this message may be privileged and conf
>>>> > idential and protected from disclosure. If you are not the original
>>>> intended
>>>> > recipient, you are hereby notified that any review, retransmission,
>>>> > dissemination, or other use of, or taking of any action in reliance
>>>> upon,
>>>> > this information is prohibited. If you have received this
>>>> communication in
>>>> > error, please notify the sender immediately by replying to this
>>>> message and
>>>> > delete it from your computer. Thank you for your cooperation. Troika
>>>> Dialog,
>>>> > Russia.
>>>> > If you need assistance please contact our Contact Center (+7495) 258
>>>> 0500
>>>> > or go to www.troika.ru/eng/Contacts/system.wbp
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Qpid CLIENT_ACKNOWLEDGE
Posted by Se...@troika.ru.
Marnie, Rajith thanks a lot.
We try to omit to use transactions because of performance penalty. To
achieve the maximum performance and have our messages guaranteed delivered
we have the following configuration:
1. cluster of two qpid brokers
2. all queues are cluster-durable
3. all messages are non-persistent (once qpid cluster collapses to a
single broker all messages will become durable due to cluster-durable
queues)
4. c++ code publishes messages
5. java code receives messages and uses client_ack mode
I would say in that case we can achieve maximum performance having all the
messages guaranteed delivered. What do you think?
Do we have to publish messages synchronously in our c++ code?
Best Regards,
Sergey Zhemzhitsky
Rajith Attapattu <ra...@gmail.com>
23.03.2011 23:21
Please respond to
users@qpid.apache.org
To
Marnie McCormack <ma...@googlemail.com>, users@qpid.apache.org
cc
Subject
Re: Qpid CLIENT_ACKNOWLEDGE
Marnie,
Looks like you forgot to hit replyAll.
So I am forwarding your email to the list along with my comments.
On Wed, Mar 23, 2011 at 12:47 PM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:
> Sorry, which broker are you usiong Sergey ? I'd assumed the Java broker
...
>
Since the Java broker supports 0-10 now, you could use sync_publish.
>
> I'd probably disagree that you can achieve guaranteed delivery without a
> transaction or its equivalent (which sync publish effectively is ?).
>
Synchronous publishing is not equivalent to transactions, but I understand
what you meant here. You were probably alluding to the fact that in both
cases messages are published in a synchronous way. i.e When the tx
completes
(or when the send method returns in sync-publish) you know that the broker
has acked the message.
However transactions have a lot more overhead than a simple sync publish,
hence a higher performance penalty with transactions.
For example look at the following relative numbers for a producer/consumer
pair using the same setup.
sync publish 15k msg/sec
tx size =2 48 msg/sec (thats right just 48 messages per sec)
tx size=1000 14k msg/sec
Also from an application point of view, non transactional code is more
simple and easy to write than transactional code.
I would normally stay away from transactions unless I really have to. It
basically negates a lot of benefits of using messaging.
Regards,
Rajith
>
> Regards,
> Marnie
>
> On Wed, Mar 23, 2011 at 4:21 PM, Rajith Attapattu
<ra...@gmail.com>wrote:
>
>> Hi Sergey,
>>
>> For Qpid to guarantee 'at-least-once' delivery on publish, you don't
need
>> to use transactions.
>>
>> Broker failures
>> -----------------
>> (1) In a stand-alone-broker you need to use persistence to ensure that
>> messages survive a broker crash.
>> (2) In a clustered broker you may be able to get away without using
>> persistence (*)
>>
>> (*) Only the c++ broker supports clustering.
>> If you don't use persistence, then in the event of a total
cluster
>> failure you will loose messages.
>> Therefore it depends on how much you can tolerate these
occasional
>> failures.
>>
>> Application/Client lib failures
>> ---------------------------------
>> In the event of a failure in the application that sends the messages,
you
>> may loose messages.
>> i.e if there are unacked messages in the client libs replay buffer,
they
>> will be lost.
>> You could avoid that by using
>>
>> For JMS client
>> -------------------
>> (1) Using synchronous publishing. Use
>> -Dsync_publish={persistent|transient|all}.
>> (2) Transaction - as Marnie pointed out.
>>
>> Performance wise there isn't much of a difference between option 1 & 2
if
>> you use large batches (ex 1000 msg/batch) for transactions.
>> But ff you use small tx batches then there is an appreciable difference
in
>> performance.
>>
>> Therefore I recommend you use -Dsync_ack=persistent.
>> Also the application code is much simpler when writing non
transactional
>> code.
>>
>> For other clients
>> --------------------
>> If you can handle reliable replay at the application level then you
could
>> just use asynchronous publishing without worrying about application
failure.
>> Ex. If your application is retrieving orders from a queue and
processing
>> it before sending a reply, then you could ack the order messsage after
the
>> broker acks your reply message.
>>
>> Hope this helps.
>>
>> Rajith
>>
>>
>> On Wed, Mar 23, 2011 at 11:21 AM, Marnie McCormack <
>> marnie.mccormack@googlemail.com> wrote:
>>
>>> Hi Sergey,
>>>
>>> To achieve guaranteed message delivery you need to be using
transactions
>>> on
>>> *publish*. Qpid then guarantees 'at least once' delivery. You'd offset
>>> the
>>> reliable delivery cost against the cost or message loss during a
failure
>>> from sending non-transacted messages. The use of transactions and
>>> persistence for reliable messaging adds cost in the shape of I/O.
>>>
>>> However, you could opt not to use transactions on consume since you
>>> cannot
>>> conceptually lose a message during consumption - if there's a failure
and
>>> a
>>> message ack doesn't get back to the broker then you'll simply get the
>>> message back again on restart. This assumes that you
listen/catch/handle
>>> errors and exception on consume such that you don't have application
>>> level
>>> side effects.
>>>
>>> Regards,
>>> Marnie
>>>
>>> On Tue, Mar 22, 2011 at 1:03 PM, <Se...@troika.ru> wrote:
>>>
>>> > Hi there
>>> >
>>> > I'm working with qpid 0.8 by means of JMS API and I'm using
>>> > CLIENT_ACKNOWLEDGE mode to get higher performance comparing to
>>> > transactinal processing.
>>> > As a rule performance is rather pure in both cases. In transactional
>>> mode
>>> > I can receive about 1.6K messages per second, in CLIENT_ACKNOWLEDGE
>>> mode
>>> > I'm able to receive about 3.6K messages per second and in
>>> AUTO_ACKNOWLEDGE
>>> > mode I'm able to receive about 70K messages per second.
>>> >
>>> > Are there any recommendations how to improve performance and get a
>>> > guaranted message delivery?
>>> >
>>> >
>>> > Best Regards,
>>> > Sergey
>>> > _______________________________________________________
>>> >
>>> > The information contained in this message may be privileged and conf
>>> > idential and protected from disclosure. If you are not the original
>>> intended
>>> > recipient, you are hereby notified that any review, retransmission,
>>> > dissemination, or other use of, or taking of any action in reliance
>>> upon,
>>> > this information is prohibited. If you have received this
communication
>>> in
>>> > error, please notify the sender immediately by replying to this
message
>>> and
>>> > delete it from your computer. Thank you for your cooperation. Troika
>>> Dialog,
>>> > Russia.
>>> > If you need assistance please contact our Contact Center (+7495)
258
>>> 0500
>>> > or go to www.troika.ru/eng/Contacts/system.wbp
>>> >
>>> >
>>>
>>
>>
>
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
Re: Qpid CLIENT_ACKNOWLEDGE
Posted by Rajith Attapattu <ra...@gmail.com>.
Marnie,
Looks like you forgot to hit replyAll.
So I am forwarding your email to the list along with my comments.
On Wed, Mar 23, 2011 at 12:47 PM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:
> Sorry, which broker are you usiong Sergey ? I'd assumed the Java broker ...
>
Since the Java broker supports 0-10 now, you could use sync_publish.
>
> I'd probably disagree that you can achieve guaranteed delivery without a
> transaction or its equivalent (which sync publish effectively is ?).
>
Synchronous publishing is not equivalent to transactions, but I understand
what you meant here. You were probably alluding to the fact that in both
cases messages are published in a synchronous way. i.e When the tx completes
(or when the send method returns in sync-publish) you know that the broker
has acked the message.
However transactions have a lot more overhead than a simple sync publish,
hence a higher performance penalty with transactions.
For example look at the following relative numbers for a producer/consumer
pair using the same setup.
sync publish 15k msg/sec
tx size =2 48 msg/sec (thats right just 48 messages per sec)
tx size=1000 14k msg/sec
Also from an application point of view, non transactional code is more
simple and easy to write than transactional code.
I would normally stay away from transactions unless I really have to. It
basically negates a lot of benefits of using messaging.
Regards,
Rajith
>
> Regards,
> Marnie
>
> On Wed, Mar 23, 2011 at 4:21 PM, Rajith Attapattu <ra...@gmail.com>wrote:
>
>> Hi Sergey,
>>
>> For Qpid to guarantee 'at-least-once' delivery on publish, you don't need
>> to use transactions.
>>
>> Broker failures
>> -----------------
>> (1) In a stand-alone-broker you need to use persistence to ensure that
>> messages survive a broker crash.
>> (2) In a clustered broker you may be able to get away without using
>> persistence (*)
>>
>> (*) Only the c++ broker supports clustering.
>> If you don't use persistence, then in the event of a total cluster
>> failure you will loose messages.
>> Therefore it depends on how much you can tolerate these occasional
>> failures.
>>
>> Application/Client lib failures
>> ---------------------------------
>> In the event of a failure in the application that sends the messages, you
>> may loose messages.
>> i.e if there are unacked messages in the client libs replay buffer, they
>> will be lost.
>> You could avoid that by using
>>
>> For JMS client
>> -------------------
>> (1) Using synchronous publishing. Use
>> -Dsync_publish={persistent|transient|all}.
>> (2) Transaction - as Marnie pointed out.
>>
>> Performance wise there isn't much of a difference between option 1 & 2 if
>> you use large batches (ex 1000 msg/batch) for transactions.
>> But ff you use small tx batches then there is an appreciable difference in
>> performance.
>>
>> Therefore I recommend you use -Dsync_ack=persistent.
>> Also the application code is much simpler when writing non transactional
>> code.
>>
>> For other clients
>> --------------------
>> If you can handle reliable replay at the application level then you could
>> just use asynchronous publishing without worrying about application failure.
>> Ex. If your application is retrieving orders from a queue and processing
>> it before sending a reply, then you could ack the order messsage after the
>> broker acks your reply message.
>>
>> Hope this helps.
>>
>> Rajith
>>
>>
>> On Wed, Mar 23, 2011 at 11:21 AM, Marnie McCormack <
>> marnie.mccormack@googlemail.com> wrote:
>>
>>> Hi Sergey,
>>>
>>> To achieve guaranteed message delivery you need to be using transactions
>>> on
>>> *publish*. Qpid then guarantees 'at least once' delivery. You'd offset
>>> the
>>> reliable delivery cost against the cost or message loss during a failure
>>> from sending non-transacted messages. The use of transactions and
>>> persistence for reliable messaging adds cost in the shape of I/O.
>>>
>>> However, you could opt not to use transactions on consume since you
>>> cannot
>>> conceptually lose a message during consumption - if there's a failure and
>>> a
>>> message ack doesn't get back to the broker then you'll simply get the
>>> message back again on restart. This assumes that you listen/catch/handle
>>> errors and exception on consume such that you don't have application
>>> level
>>> side effects.
>>>
>>> Regards,
>>> Marnie
>>>
>>> On Tue, Mar 22, 2011 at 1:03 PM, <Se...@troika.ru> wrote:
>>>
>>> > Hi there
>>> >
>>> > I'm working with qpid 0.8 by means of JMS API and I'm using
>>> > CLIENT_ACKNOWLEDGE mode to get higher performance comparing to
>>> > transactinal processing.
>>> > As a rule performance is rather pure in both cases. In transactional
>>> mode
>>> > I can receive about 1.6K messages per second, in CLIENT_ACKNOWLEDGE
>>> mode
>>> > I'm able to receive about 3.6K messages per second and in
>>> AUTO_ACKNOWLEDGE
>>> > mode I'm able to receive about 70K messages per second.
>>> >
>>> > Are there any recommendations how to improve performance and get a
>>> > guaranted message delivery?
>>> >
>>> >
>>> > Best Regards,
>>> > Sergey
>>> > _______________________________________________________
>>> >
>>> > The information contained in this message may be privileged and conf
>>> > idential and protected from disclosure. If you are not the original
>>> intended
>>> > recipient, you are hereby notified that any review, retransmission,
>>> > dissemination, or other use of, or taking of any action in reliance
>>> upon,
>>> > this information is prohibited. If you have received this communication
>>> in
>>> > error, please notify the sender immediately by replying to this message
>>> and
>>> > delete it from your computer. Thank you for your cooperation. Troika
>>> Dialog,
>>> > Russia.
>>> > If you need assistance please contact our Contact Center (+7495) 258
>>> 0500
>>> > or go to www.troika.ru/eng/Contacts/system.wbp
>>> >
>>> >
>>>
>>
>>
>
Re: Qpid CLIENT_ACKNOWLEDGE
Posted by Rajith Attapattu <ra...@gmail.com>.
Hi Sergey,
For Qpid to guarantee 'at-least-once' delivery on publish, you don't need to
use transactions.
Broker failures
-----------------
(1) In a stand-alone-broker you need to use persistence to ensure that
messages survive a broker crash.
(2) In a clustered broker you may be able to get away without using
persistence (*)
(*) Only the c++ broker supports clustering.
If you don't use persistence, then in the event of a total cluster
failure you will loose messages.
Therefore it depends on how much you can tolerate these occasional
failures.
Application/Client lib failures
---------------------------------
In the event of a failure in the application that sends the messages, you
may loose messages.
i.e if there are unacked messages in the client libs replay buffer, they
will be lost.
You could avoid that by using
For JMS client
-------------------
(1) Using synchronous publishing. Use
-Dsync_publish={persistent|transient|all}.
(2) Transaction - as Marnie pointed out.
Performance wise there isn't much of a difference between option 1 & 2 if
you use large batches (ex 1000 msg/batch) for transactions.
But ff you use small tx batches then there is an appreciable difference in
performance.
Therefore I recommend you use -Dsync_ack=persistent.
Also the application code is much simpler when writing non transactional
code.
For other clients
--------------------
If you can handle reliable replay at the application level then you could
just use asynchronous publishing without worrying about application failure.
Ex. If your application is retrieving orders from a queue and processing it
before sending a reply, then you could ack the order messsage after the
broker acks your reply message.
Hope this helps.
Rajith
On Wed, Mar 23, 2011 at 11:21 AM, Marnie McCormack <
marnie.mccormack@googlemail.com> wrote:
> Hi Sergey,
>
> To achieve guaranteed message delivery you need to be using transactions on
> *publish*. Qpid then guarantees 'at least once' delivery. You'd offset the
> reliable delivery cost against the cost or message loss during a failure
> from sending non-transacted messages. The use of transactions and
> persistence for reliable messaging adds cost in the shape of I/O.
>
> However, you could opt not to use transactions on consume since you cannot
> conceptually lose a message during consumption - if there's a failure and a
> message ack doesn't get back to the broker then you'll simply get the
> message back again on restart. This assumes that you listen/catch/handle
> errors and exception on consume such that you don't have application level
> side effects.
>
> Regards,
> Marnie
>
> On Tue, Mar 22, 2011 at 1:03 PM, <Se...@troika.ru> wrote:
>
> > Hi there
> >
> > I'm working with qpid 0.8 by means of JMS API and I'm using
> > CLIENT_ACKNOWLEDGE mode to get higher performance comparing to
> > transactinal processing.
> > As a rule performance is rather pure in both cases. In transactional mode
> > I can receive about 1.6K messages per second, in CLIENT_ACKNOWLEDGE mode
> > I'm able to receive about 3.6K messages per second and in
> AUTO_ACKNOWLEDGE
> > mode I'm able to receive about 70K messages per second.
> >
> > Are there any recommendations how to improve performance and get a
> > guaranted message delivery?
> >
> >
> > Best Regards,
> > Sergey
> > _______________________________________________________
> >
> > The information contained in this message may be privileged and conf
> > idential and protected from disclosure. If you are not the original
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, or other use of, or taking of any action in reliance upon,
> > this information is prohibited. If you have received this communication
> in
> > error, please notify the sender immediately by replying to this message
> and
> > delete it from your computer. Thank you for your cooperation. Troika
> Dialog,
> > Russia.
> > If you need assistance please contact our Contact Center (+7495) 258
> 0500
> > or go to www.troika.ru/eng/Contacts/system.wbp
> >
> >
>
Re: Qpid CLIENT_ACKNOWLEDGE
Posted by Marnie McCormack <ma...@googlemail.com>.
Hi Sergey,
To achieve guaranteed message delivery you need to be using transactions on
*publish*. Qpid then guarantees 'at least once' delivery. You'd offset the
reliable delivery cost against the cost or message loss during a failure
from sending non-transacted messages. The use of transactions and
persistence for reliable messaging adds cost in the shape of I/O.
However, you could opt not to use transactions on consume since you cannot
conceptually lose a message during consumption - if there's a failure and a
message ack doesn't get back to the broker then you'll simply get the
message back again on restart. This assumes that you listen/catch/handle
errors and exception on consume such that you don't have application level
side effects.
Regards,
Marnie
On Tue, Mar 22, 2011 at 1:03 PM, <Se...@troika.ru> wrote:
> Hi there
>
> I'm working with qpid 0.8 by means of JMS API and I'm using
> CLIENT_ACKNOWLEDGE mode to get higher performance comparing to
> transactinal processing.
> As a rule performance is rather pure in both cases. In transactional mode
> I can receive about 1.6K messages per second, in CLIENT_ACKNOWLEDGE mode
> I'm able to receive about 3.6K messages per second and in AUTO_ACKNOWLEDGE
> mode I'm able to receive about 70K messages per second.
>
> Are there any recommendations how to improve performance and get a
> guaranted message delivery?
>
>
> Best Regards,
> Sergey
> _______________________________________________________
>
> The information contained in this message may be privileged and conf
> idential and protected from disclosure. If you are not the original intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, or other use of, or taking of any action in reliance upon,
> this information is prohibited. If you have received this communication in
> error, please notify the sender immediately by replying to this message and
> delete it from your computer. Thank you for your cooperation. Troika Dialog,
> Russia.
> If you need assistance please contact our Contact Center (+7495) 258 0500
> or go to www.troika.ru/eng/Contacts/system.wbp
>
>