You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by uromahn <ul...@ulrichromahn.net> on 2014/01/08 19:38:08 UTC
Serious Bug in AMQP 1.0 JMS Client with persistent messages
I believe I found a serious bug in the current (latest version from source)
qpid-amqp-1-0-client-jms library.
I think the client violates the delivery guarantee of JMS with persistent
messages.
Here is my test:
1. Start a qpid-cpp broker. Create a persistent queue on that broker
2. Start a test Java application using the amqp 1.0 JMS client. The
application will write 10,000 persistent messages into the just created
queue.
3. After some time while the client is still writing messages into the
queue, I kill the broker (kill -9 <pid>) simulating the worst disaster
scenario
4. My client code throws an exception and reports 4,963 messages sent
5. After re-starting qpidd, the queue statistics reports only 4,816 messages
in the queue!
Here are some of the relevant code snippets from the client:
// Creating a Session
Session session = brokerConn.getConnection().createSession(false,
Session.AUTO_ACKNOWLEDGE); // NOTE AUTO_ACKNOWLEGE does not have any effect
with nessage producer
// Create a destination
Destination dest = session.createQueue(queueName);
// Creating a MessageProducer
MessageProducer msgProducer = session.createProducer(dest);
msgProducer.setDeliveryMode(2); // 2 == persistent
And then in a loop 1000 times with a random TextMessage I call:
msgProducer.send(msg);
So, as you can see, nothing special here.
So, it looks like the client is sending the message to the broker and the
"send(msg)" method returns *before* either the broker acknowledged receiving
and persisting the message, or the client does some internal caching.
By the way, I see the exact same issue when using ActiveMQ 5.9.0 as the
broker using the AMQP connector.
However, it is also possible that the bug is within the proton library on
the server side which is being used in both the qpid-cpp broker (proton-c)
and ActiveMQ 5.9.0 (proton-j) to support AMQP 1-0.
I already created a JIRA to report this potential bug:
https://issues.apache.org/jira/browse/QPID-5455
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
Quick update: re-ran the test using a qpid-0.24 Java broker with the same
result. A "kill -9" lost 42 messages from 491 reported sent.
Still not clear whether it is a client bug or a bug in the proton messaging
library.
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602417.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
Ok, I did the following:
1. got the latest sources for ActiveMQ 5.10-SNAPSHOT and built from source
2. got the latest sources for qpid and build amqp-1-0-client-jms from source
3. started ActiveMQ 5.10-SNAPSHOT enabling levelDB (Java) and NIO on the
AMQP connector
4. re-built my test program with the latest amqp-1-0-client-jms libraries
After re-running my test, I still have one message (the last one sent)
missing after killing my broker.
However, when I change to the standard amqp connector (non-NIO), I don't
have a missing message. Looks like there is a lingering bug in the amqp+nio
connector on ActiveMQ as well.
Now comes the $1Mill question (which is probably wrong in this mailing list
since it belongs to the ActiveMQ community): is there a chance to back-port
those fixes to ActiveMQ 5.9.0? If you would provide me with a list of
commits or changed files, I can probably attempt this myself, but I'd rather
have the official committers do that and provide a 5.9.1 release.
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602455.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
I was doing that but was missing your commit by just a few minutes.
Re-building just now to see how it changes...
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602452.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Timothy Bish <ta...@gmail.com>.
On 01/08/2014 05:34 PM, uromahn wrote:
> With the patch added to my local sources, I ran a bunch of different tests
> against an ActiveMQ 5.9.0 and 5.10-SNAPSHOT broker using the patched as well
> as unpatched client library.
>
> To summarize my findings:
> 1. running the patched client against ActiveMQ 5.9.0 without sync-publish
> set at all does not seem to force sync-publish which could be proven by
> message loss when killing the broker
> 2. running the patched or unpatched client against ActiveMQ 5.9.0 with
> sync-publish=true set explicitly is publishing one single message and then
> "hanging" on the send() call
> 3. running the patched client against ActiveMQ 5.10-SNAPSHOT without
> sync-publish set at all seems to force sync-publish as demonstrated by *no*
> message loss when killing the broker
> 4. running the unpatched client against ActiveMQ 5.10-SNAPSHOT also seems to
> force sync-publish as demonstrated by *no* message loss when killing the
> broker.
> 5. running the patched or unpatched client against ActiveMQ 5.10-SNAPSHOT
> with sync-publish=true explicitly set is publishing one single message and
> then "hanging" on the send() call
>
> In addition to the "hanging" of the send() call with sync-publish=true set,
> I also sense an issue within ActiveMQ as well.
>
>
>
>
> --
> View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602448.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
You need to build ActiveMQ from source or wait for a new SNAPSHOT build
as the sync send issue was resolved today with the fix for:
https://issues.apache.org/jira/browse/AMQ-4741
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
Quick correction of my last post:
it seems that I still have a message loss when running against ActiveMQ
5.10-SNAPSHOT without setting sync-publish, although the loss seems to be
consistently limited to a single message - the last one published.
So, there seems to be something really odd going on.
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602450.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
With the patch added to my local sources, I ran a bunch of different tests
against an ActiveMQ 5.9.0 and 5.10-SNAPSHOT broker using the patched as well
as unpatched client library.
To summarize my findings:
1. running the patched client against ActiveMQ 5.9.0 without sync-publish
set at all does not seem to force sync-publish which could be proven by
message loss when killing the broker
2. running the patched or unpatched client against ActiveMQ 5.9.0 with
sync-publish=true set explicitly is publishing one single message and then
"hanging" on the send() call
3. running the patched client against ActiveMQ 5.10-SNAPSHOT without
sync-publish set at all seems to force sync-publish as demonstrated by *no*
message loss when killing the broker
4. running the unpatched client against ActiveMQ 5.10-SNAPSHOT also seems to
force sync-publish as demonstrated by *no* message loss when killing the
broker.
5. running the patched or unpatched client against ActiveMQ 5.10-SNAPSHOT
with sync-publish=true explicitly set is publishing one single message and
then "hanging" on the send() call
In addition to the "hanging" of the send() call with sync-publish=true set,
I also sense an issue within ActiveMQ as well.
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602448.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
Thanks for creating a patch on such a short notice. I am testing it out on my
side right now.
-Uli
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602435.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Rob Godfrey <ro...@gmail.com>.
Looks good - I'll commit it later tonight
-- Rob
On 8 January 2014 21:54, Timothy Bish <ta...@gmail.com> wrote:
> On 01/08/2014 03:41 PM, Rob Godfrey wrote:
>
>> On 8 January 2014 21:27, Fraser Adams <fr...@blueyonder.co.uk>
>> wrote:
>>
>> On 08/01/14 18:55, Rob Godfrey wrote:
>>>
>>> Hi Uli,
>>>>
>>>> To enable synchronous publishing you need to either set the Java system
>>>> property "qpid.sync_publish" to true, or have sync-publish=true as one
>>>> of
>>>> the URL options in your connection URL.
>>>>
>>>> I agree it should be the default, and we can look to change this in a
>>>> future release.
>>>>
>>>> Hmm, I'd really rather you didn't do that. Async behaviour has been the
>>> default behaviour with Qpid from year dot. so changing it is pretty
>>> likely
>>> to bite someone nastily.
>>>
>>> I'd agree that async behaviour technically means that by default Qpid
>>> breaches JMS guarantees, but TBH I think it's a little late in the day
>>> now
>>> to be going switching the default. It's also make the default behaviour
>>> inconsistent with qpid::messaging which doesn't seem ideal.
>>>
>>>
>>> Given this is a totally separate client, (and is vastly different in
>> many
>> other ways - like different connection URLs, different address formats,
>> etc.) and it's probably being used more widely against non-qpid
>> brokers/services than it is against Qpid right now, then I'm not sure I'm
>> too worried about refleting the behaviour of the AMQP 0-8/9/9-1/10client.
>> I think the bigger issue in switching the default right now is that it
>> would seem that it would break anyone trying to use it with ActiveMQ.
>>
>>
>> As we've discussed before not everyone has guaranteed delivery right up
>>> there at the top end of things they give a damn about, in my case it's
>>> all
>>> about performance and I can take the hit if I lose the odd message -
>>> plenty
>>> more where they came from :-) I probably wouldn't be wildly amused to
>>> upgrade to Qpid version "x" only to find a massive performance regression
>>> that might force me to get a non-trivial number of clients to change some
>>> bit of config. or other.
>>>
>>>
>>> The "intelligent" change would be to only use syn publishing with
>> persistent but non-transactional messaging. If you are using transient
>> you
>> clearly aren't looking for guaranteed delivery anyway. If you are using
>> transation, you get your guarantee at the commit. This is the "expected"
>> behaviour for JMS (though most implementations offer a switch to turn it
>> off so people can get better performance... in which case you have to
>> wonder why they are using persistent messaging in the first place)....
>>
>> Anyway given the issues with ActiveMQ - I won't be changing it for the
>> moment anyways.
>>
>> -- Rob
>>
>>
>> Frase
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>>>
>>> I submitted a patch for the issue that was opened for this thread:
>
> https://issues.apache.org/jira/browse/QPID-5455
>
> It enables sync sends either when the flag is set or the JMS DeliveryMode
> that was requested is PERSISTENT and there is no current TX in progress.
> Testing against a 0.26-SNAPSHOT with the fix and an ActiveMQ 5.10-SNAPSHOT
> things seem to be working fine.
>
> --
> Tim Bish
> Sr Software Engineer | RedHat Inc.
> tim.bish@redhat.com | www.fusesource.com | www.redhat.com
> skype: tabish121 | twitter: @tabish121
> blog: http://timbish.blogspot.com/
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Timothy Bish <ta...@gmail.com>.
On 01/08/2014 03:41 PM, Rob Godfrey wrote:
> On 8 January 2014 21:27, Fraser Adams <fr...@blueyonder.co.uk> wrote:
>
>> On 08/01/14 18:55, Rob Godfrey wrote:
>>
>>> Hi Uli,
>>>
>>> To enable synchronous publishing you need to either set the Java system
>>> property "qpid.sync_publish" to true, or have sync-publish=true as one of
>>> the URL options in your connection URL.
>>>
>>> I agree it should be the default, and we can look to change this in a
>>> future release.
>>>
>> Hmm, I'd really rather you didn't do that. Async behaviour has been the
>> default behaviour with Qpid from year dot. so changing it is pretty likely
>> to bite someone nastily.
>>
>> I'd agree that async behaviour technically means that by default Qpid
>> breaches JMS guarantees, but TBH I think it's a little late in the day now
>> to be going switching the default. It's also make the default behaviour
>> inconsistent with qpid::messaging which doesn't seem ideal.
>>
>>
> Given this is a totally separate client, (and is vastly different in many
> other ways - like different connection URLs, different address formats,
> etc.) and it's probably being used more widely against non-qpid
> brokers/services than it is against Qpid right now, then I'm not sure I'm
> too worried about refleting the behaviour of the AMQP 0-8/9/9-1/10 client.
> I think the bigger issue in switching the default right now is that it
> would seem that it would break anyone trying to use it with ActiveMQ.
>
>
>> As we've discussed before not everyone has guaranteed delivery right up
>> there at the top end of things they give a damn about, in my case it's all
>> about performance and I can take the hit if I lose the odd message - plenty
>> more where they came from :-) I probably wouldn't be wildly amused to
>> upgrade to Qpid version "x" only to find a massive performance regression
>> that might force me to get a non-trivial number of clients to change some
>> bit of config. or other.
>>
>>
> The "intelligent" change would be to only use syn publishing with
> persistent but non-transactional messaging. If you are using transient you
> clearly aren't looking for guaranteed delivery anyway. If you are using
> transation, you get your guarantee at the commit. This is the "expected"
> behaviour for JMS (though most implementations offer a switch to turn it
> off so people can get better performance... in which case you have to
> wonder why they are using persistent messaging in the first place)....
>
> Anyway given the issues with ActiveMQ - I won't be changing it for the
> moment anyways.
>
> -- Rob
>
>
>> Frase
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
I submitted a patch for the issue that was opened for this thread:
https://issues.apache.org/jira/browse/QPID-5455
It enables sync sends either when the flag is set or the JMS
DeliveryMode that was requested is PERSISTENT and there is no current TX
in progress. Testing against a 0.26-SNAPSHOT with the fix and an
ActiveMQ 5.10-SNAPSHOT things seem to be working fine.
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Rob Godfrey <ro...@gmail.com>.
On 8 January 2014 21:27, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> On 08/01/14 18:55, Rob Godfrey wrote:
>
>> Hi Uli,
>>
>> To enable synchronous publishing you need to either set the Java system
>> property "qpid.sync_publish" to true, or have sync-publish=true as one of
>> the URL options in your connection URL.
>>
>> I agree it should be the default, and we can look to change this in a
>> future release.
>>
>
> Hmm, I'd really rather you didn't do that. Async behaviour has been the
> default behaviour with Qpid from year dot. so changing it is pretty likely
> to bite someone nastily.
>
> I'd agree that async behaviour technically means that by default Qpid
> breaches JMS guarantees, but TBH I think it's a little late in the day now
> to be going switching the default. It's also make the default behaviour
> inconsistent with qpid::messaging which doesn't seem ideal.
>
>
Given this is a totally separate client, (and is vastly different in many
other ways - like different connection URLs, different address formats,
etc.) and it's probably being used more widely against non-qpid
brokers/services than it is against Qpid right now, then I'm not sure I'm
too worried about refleting the behaviour of the AMQP 0-8/9/9-1/10 client.
I think the bigger issue in switching the default right now is that it
would seem that it would break anyone trying to use it with ActiveMQ.
> As we've discussed before not everyone has guaranteed delivery right up
> there at the top end of things they give a damn about, in my case it's all
> about performance and I can take the hit if I lose the odd message - plenty
> more where they came from :-) I probably wouldn't be wildly amused to
> upgrade to Qpid version "x" only to find a massive performance regression
> that might force me to get a non-trivial number of clients to change some
> bit of config. or other.
>
>
The "intelligent" change would be to only use syn publishing with
persistent but non-transactional messaging. If you are using transient you
clearly aren't looking for guaranteed delivery anyway. If you are using
transation, you get your guarantee at the commit. This is the "expected"
behaviour for JMS (though most implementations offer a switch to turn it
off so people can get better performance... in which case you have to
wonder why they are using persistent messaging in the first place)....
Anyway given the issues with ActiveMQ - I won't be changing it for the
moment anyways.
-- Rob
> Frase
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 08/01/14 18:55, Rob Godfrey wrote:
> Hi Uli,
>
> To enable synchronous publishing you need to either set the Java system
> property "qpid.sync_publish" to true, or have sync-publish=true as one of
> the URL options in your connection URL.
>
> I agree it should be the default, and we can look to change this in a
> future release.
Hmm, I'd really rather you didn't do that. Async behaviour has been the
default behaviour with Qpid from year dot. so changing it is pretty
likely to bite someone nastily.
I'd agree that async behaviour technically means that by default Qpid
breaches JMS guarantees, but TBH I think it's a little late in the day
now to be going switching the default. It's also make the default
behaviour inconsistent with qpid::messaging which doesn't seem ideal.
As we've discussed before not everyone has guaranteed delivery right up
there at the top end of things they give a damn about, in my case it's
all about performance and I can take the hit if I lose the odd message -
plenty more where they came from :-) I probably wouldn't be wildly
amused to upgrade to Qpid version "x" only to find a massive performance
regression that might force me to get a non-trivial number of clients to
change some bit of config. or other.
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Rob Godfrey <ro...@gmail.com>.
What sort of environment are you using?
Just testing on my laptop (Macbook with SSD) over loopback and using small
("Hello World") messages I get ~1100msg/s sync publish on the Java Broker
with the Derby store and ~1600msg/s with the Java Broker and BDB store.
-- Rob
On 8 January 2014 21:15, uromahn <ul...@ulrichromahn.net> wrote:
> Rob,
>
> I just completed another round of testing and here is a consolidated
> status:
>
> Client (latest version build from trunk) using anyc-publish=true:
> *Qpid Java Broker 0.24:* enqueued and dequeued 1000 messages. Writes about
> 8.5 msgs/sec and reads about 60 msgs/sec.
> *Qpid CPP Broker (latest source build from trunk):* enueued and dequeued
> 100
> messages (1000 would take too long). Writes about 1 msg/sec and reads about
> 700 msgs/sec.
> *ActiveMQ 5.9.0 Broker (with AMQP 1.0 Connector):* tried to enqueue and
> dequeue 1000 messages. Publisher.send(msg) was "stuck" at the first
> message.
> Broker reported one enqueue message but client never returned from the
> first
> send() call.
> *Apache Apollo 1.6 Broker:* enqueued and dequeued 1000 messages. Writes
> about 33 msgs/sec and reads about 890 msgs/sec.
>
> So, I see two issues here:
> 1. performance against a qpid-cpp broker is unacceptably slow - looks like
> some protocol issue on the CPP broker to me
> 2. hangs publishing to an ActiveMQ broker - looks like some protocol issue
> on the AMQP connector in ActiveMQ to me,
>
> Not sure how to proceed from here. Do you want me to create JIRAs for the
> cpp-broker and ActiveMQ or would you rather do that?
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602424.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Rob Godfrey <ro...@gmail.com>.
On 8 January 2014 21:15, uromahn <ul...@ulrichromahn.net> wrote:
> Rob,
>
> I just completed another round of testing and here is a consolidated
> status:
>
> Client (latest version build from trunk) using anyc-publish=true:
> *Qpid Java Broker 0.24:* enqueued and dequeued 1000 messages. Writes about
> 8.5 msgs/sec and reads about 60 msgs/sec.
> *Qpid CPP Broker (latest source build from trunk):* enueued and dequeued
> 100
> messages (1000 would take too long). Writes about 1 msg/sec and reads about
> 700 msgs/sec.
> *ActiveMQ 5.9.0 Broker (with AMQP 1.0 Connector):* tried to enqueue and
> dequeue 1000 messages. Publisher.send(msg) was "stuck" at the first
> message.
> Broker reported one enqueue message but client never returned from the
> first
> send() call.
> *Apache Apollo 1.6 Broker:* enqueued and dequeued 1000 messages. Writes
> about 33 msgs/sec and reads about 890 msgs/sec.
>
> So, I see two issues here:
> 1. performance against a qpid-cpp broker is unacceptably slow - looks like
> some protocol issue on the CPP broker to me
> 2. hangs publishing to an ActiveMQ broker - looks like some protocol issue
> on the AMQP connector in ActiveMQ to me,
>
> Not sure how to proceed from here. Do you want me to create JIRAs for the
> cpp-broker and ActiveMQ or would you rather do that?
>
>
If you could, that would be great. At some point I may try to optimise the
Java Broker a bit more for this case (there are two fairly obvious
improvements that would speed up things here... but it's not really been a
use case we have focussed on).
Thanks for all your help in teasing out these issues!
Cheers,
Rob
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602424.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
Rob,
I just completed another round of testing and here is a consolidated status:
Client (latest version build from trunk) using anyc-publish=true:
*Qpid Java Broker 0.24:* enqueued and dequeued 1000 messages. Writes about
8.5 msgs/sec and reads about 60 msgs/sec.
*Qpid CPP Broker (latest source build from trunk):* enueued and dequeued 100
messages (1000 would take too long). Writes about 1 msg/sec and reads about
700 msgs/sec.
*ActiveMQ 5.9.0 Broker (with AMQP 1.0 Connector):* tried to enqueue and
dequeue 1000 messages. Publisher.send(msg) was "stuck" at the first message.
Broker reported one enqueue message but client never returned from the first
send() call.
*Apache Apollo 1.6 Broker:* enqueued and dequeued 1000 messages. Writes
about 33 msgs/sec and reads about 890 msgs/sec.
So, I see two issues here:
1. performance against a qpid-cpp broker is unacceptably slow - looks like
some protocol issue on the CPP broker to me
2. hangs publishing to an ActiveMQ broker - looks like some protocol issue
on the AMQP connector in ActiveMQ to me,
Not sure how to proceed from here. Do you want me to create JIRAs for the
cpp-broker and ActiveMQ or would you rather do that?
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602424.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Uli,
On 8 Jan 2014 20:29, "uromahn" <ul...@ulrichromahn.net> wrote:
>
> I changed my test to use 'sync-publish=true' in my connection URL as
> suggested.
> However, now I am facing another issue:
> Publishing to my qpid-cpp broker becomes extremely slow: less than 1
msg/sec
> compared to 1000+ msgs/sec with async publish!
> Publishing to an ActiveMQ 5.9.0 broker seems to hang in the
> producer.send(msg) method. I see one message being enqueued in the broker
> and then nothing else happens and the client just hangs there. I suspect
> that the client is waiting for some acknowledgement which the AMQP
Connector
> in ActiveMQ never sends.
>
> So, it appears that the sync-publish mode is simply unusable.
>
So, IIRC, the client is simply using the standard AMQP mechanism to
establish a synchronous acknowledgement. I haven't tested in a while, but
I would certainly expect to get > 1msg/s.
I'll try it out later tonight against the Java Broker, with a sample size
of 2 where 1 doesn't work it's a bit difficult to determine if it's a
client or broker issue with performance
I'll also look at making the use of sync more intelligent in the client
(i.e. it's required on non transactional persistent messages, but not
otherwise). I think it's only the way it is currently as historically
that's how the older Qpid JMS clients have "worked".
-- Rob
>
>
> --
> View this message in context:
http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602420.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
I changed my test to use 'sync-publish=true' in my connection URL as
suggested.
However, now I am facing another issue:
Publishing to my qpid-cpp broker becomes extremely slow: less than 1 msg/sec
compared to 1000+ msgs/sec with async publish!
Publishing to an ActiveMQ 5.9.0 broker seems to hang in the
producer.send(msg) method. I see one message being enqueued in the broker
and then nothing else happens and the client just hangs there. I suspect
that the client is waiting for some acknowledgement which the AMQP Connector
in ActiveMQ never sends.
So, it appears that the sync-publish mode is simply unusable.
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602420.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by uromahn <ul...@ulrichromahn.net>.
Rob,
saw your reply after my second post.
I will try that out immediately.
As you mentioned, the client should be "intelligent" enough to switch off
async publishing by default when a producer was set to persistent delivery
mode since the client is expected to honor the JMS delivery guarantee in
this case.
-Uli
--
View this message in context: http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408p7602419.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Serious Bug in AMQP 1.0 JMS Client with persistent messages
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Uli,
To enable synchronous publishing you need to either set the Java system
property "qpid.sync_publish" to true, or have sync-publish=true as one of
the URL options in your connection URL.
I agree it should be the default, and we can look to change this in a
future release.
-- Rob
On 8 January 2014 19:38, uromahn <ul...@ulrichromahn.net> wrote:
> I believe I found a serious bug in the current (latest version from source)
> qpid-amqp-1-0-client-jms library.
>
> I think the client violates the delivery guarantee of JMS with persistent
> messages.
> Here is my test:
>
> 1. Start a qpid-cpp broker. Create a persistent queue on that broker
> 2. Start a test Java application using the amqp 1.0 JMS client. The
> application will write 10,000 persistent messages into the just created
> queue.
> 3. After some time while the client is still writing messages into the
> queue, I kill the broker (kill -9 <pid>) simulating the worst disaster
> scenario
> 4. My client code throws an exception and reports 4,963 messages sent
> 5. After re-starting qpidd, the queue statistics reports only 4,816
> messages
> in the queue!
>
> Here are some of the relevant code snippets from the client:
>
> // Creating a Session
> Session session = brokerConn.getConnection().createSession(false,
> Session.AUTO_ACKNOWLEDGE); // NOTE AUTO_ACKNOWLEGE does not have any effect
> with nessage producer
> // Create a destination
> Destination dest = session.createQueue(queueName);
> // Creating a MessageProducer
> MessageProducer msgProducer = session.createProducer(dest);
> msgProducer.setDeliveryMode(2); // 2 == persistent
>
> And then in a loop 1000 times with a random TextMessage I call:
> msgProducer.send(msg);
>
> So, as you can see, nothing special here.
>
> So, it looks like the client is sending the message to the broker and the
> "send(msg)" method returns *before* either the broker acknowledged
> receiving
> and persisting the message, or the client does some internal caching.
>
> By the way, I see the exact same issue when using ActiveMQ 5.9.0 as the
> broker using the AMQP connector.
> However, it is also possible that the bug is within the proton library on
> the server side which is being used in both the qpid-cpp broker (proton-c)
> and ActiveMQ 5.9.0 (proton-j) to support AMQP 1-0.
>
> I already created a JIRA to report this potential bug:
> https://issues.apache.org/jira/browse/QPID-5455
>
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Serious-Bug-in-AMQP-1-0-JMS-Client-with-persistent-messages-tp7602408.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>