You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by uromahn <ul...@ulrichromahn.net> on 2014/01/04 01:08:18 UTC
Interesting Persistent Messaging Performance
I have just ran a very simple performance test to compare ActiveMQ 5.9.0 to
some other broker.
This message is rather long but has some surprising and also unexpected
results, so please bear with me and read on...
I have used the following setup:
Broker:
* Virtual Machine with CentOS 6.0 64bit
* 2 virtual CPUs and 6GB RAM
* latest Oracle JVM (1.7.0_45)
* ActiveMQ 5.9.0 (built from source)
* Openwire and AMQP connectors are both configure using NIO
Client:
* Client running on separate VM connected via corporate Gigabit LAN to
broker VM
* Client using the "standard" ActiveMQ JMS client in one test and the
Qpid-amqp-1-0-client-jms library for another test (the qpid-amqp library was
built from source using the latest head with a recent patch)
* The ActiveMQ JMS client was connecting with caching and pre-fetch enabled.
This feature does not exist with the AMQP client library.
The test consists of a very simple scenario:
1. En-queue 1000 messages (1024 bytes message body) in a single threaded
application using a single connection and session
2. After completing step #1, de-queue those 1000 messages in the same single
threaded application using the same connection but with a new session
I measured the time it took to en-queue and de-queue each message using
System.nanoTime() with a resolution of +/- 100 nanoseconds. I essentially
took the time before I called producer.send(msg) and then took the time
immediately after the call returned.
The same measurement was applied with the de-queue - I took the time before
a consumer.receive(timeout) and immediately after.
I configured and measured my ActiveMQ 5.9.0 broker with three different
persistent stores:
1. KahaDB (the default)
2. LevelDB (Java)
3. LevelDB (native via JNI)
I installed the native LevelDB 1.7 Rel2 from the standard CentOS repo.
Since I am particularly interested in performance of persistent messages, I
will only present those results here:
Here are the results using the "native" ActiveMQ client via openwire.
======================================
Persistent Store | en-queue | de-queue
-------------------------------------------------
KahaDB | 57.9 ms | 0.3 ms
-------------------------------------------------
LevelDB (Java) | 28.6 ms | 0.2 ms
-------------------------------------------------
LevelDB (JNI) | 26.3 ms | 0.2 ms
======================================
It is visible that LevelDB seems about 2x faster than KahaDB.
And now comes the surprise, at least to me. Here are the results connecting
with the qpid-amqp-1-0-client-jms library.
======================================
Persistent Store | en-queue | de-queue
-------------------------------------------------
KahaDB | 24.0 ms | 25.7 ms
-------------------------------------------------
LevelDB (Java) | 0.9 ms | 1.0 ms
-------------------------------------------------
LevelDB (JNI) | 0.9 ms | 0.9 ms
======================================
(Note: the different de-queuing times for KahaDB between openwire and AMQP
is most likely due to caching with the ActiveMQ client.)
As you can see, performance with KahaDB is about 2x on AMQP compared to
openwire. However, it is more than 25x with LevelDB using AMQP!!
So here are my questions: why is the performance between openwire and AMQP
so much different? Is this a fluke or a bug in one of the libraries or did I
miss something in my tests?
There is also a lesson learned for me: performance between LeveDB Java and
native (JNI) seems almost identical and hence it may not worth the hassle to
deal with the native LevelDB.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Interesting Persistent Messaging Performance
Posted by Timothy Bish <ta...@gmail.com>.
On 01/08/2014 02:32 PM, uromahn wrote:
> Final update: this is not a bug in the AMQP 1-0 JMS client but rather a
> "feature". Turns out that the AMQP client uses async publishing by default
> boosting the performance but violating the delivery guarantee of persistent
> messages.
>
> However, when using the sync publishing mode, it appears that the amqp
> client has another bug which makes it extremely slow and hence practically
> unusable.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001p4676129.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Patches have been applied to both ActiveMQ trunk and to the QPid AMQP
JMS client code that should address the sync send issue. Messages that
are sent with the persistent flag set outside of a TX should now be sent
synchronously and the broker should now return the correct state on
acceptance to ensure the client doesn't block indefinitely waiting for
its sync send to complete.
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/
Re: Interesting Persistent Messaging Performance
Posted by artnaseef <ar...@artnaseef.com>.
Thanks for the update, that's good-to-know.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001p4676130.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Interesting Persistent Messaging Performance
Posted by uromahn <ul...@ulrichromahn.net>.
Final update: this is not a bug in the AMQP 1-0 JMS client but rather a
"feature". Turns out that the AMQP client uses async publishing by default
boosting the performance but violating the delivery guarantee of persistent
messages.
However, when using the sync publishing mode, it appears that the amqp
client has another bug which makes it extremely slow and hence practically
unusable.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001p4676129.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Interesting Persistent Messaging Performance
Posted by uromahn <ul...@ulrichromahn.net>.
Quick update:
I re-ran my tests against a qpid-cpp broker with a persistent queue and saw
the exact same issue, i.e. I am losing messages when the broker gets killed
while my client writes messages to the queue.
So, this appears to be a bug in the Qpid-amqp-1-0-client-jms library. I will
post a message to that board and then file a JIRA.
-Uli
--
View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001p4676125.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Interesting Persistent Messaging Performance
Posted by uromahn <ul...@ulrichromahn.net>.
I am installing and running ActiveMQ 5.9.0 with almost default configuration.
The only difference was changing <kahaDB> to <levelDB> as
persistenceAdapter.
Also, I am not using a "test script" but instead a rather simple Java JMS
application using standard JMS 1.1 API.
Here are some of the relevant code snippets:
// Creating a Session
Session session = brokerConn.getConnection().createSession(false,
Session.AUTO_ACKNOWLEDGE); // NOTE AUTO_ACKNOWLEGE does not have any effect
with nessage producer
// Create a destination
Destination dest = session.createQueue(queueName);
// Creating a MessageProducer
MessageProducer msgProducer = session.createProducer(dest);
msgProducer.setDeliveryMode(2); // 2 == persistent
And then in a loop 1000 times with a random TextMessage I call:
msgProducer.send(msg);
So, as you can see, nothing special here.
However, I think I know where the performance difference comes from.
I believe there is a bug in the AMQP Connector (for which I will create a
JIRA).
Here is my test:
1. Write 10,000 messages into a persistent queue using the AMQP 1-0 JMS
client. Sometime in the middle of this, I kill my ActiveMQ process (kill -9
<pid> on Linux). The client errors out and I report the number of sent
messages in the log file
2. Same as #1, except I am using the ActiveMQ 5.9.0 client in my application
In case #1, my client reported 5,738 messages written. However, after
re-starting ActiveMQ, the queue only contained 5,642 messages - I "lost" 96
messages!!
In case #2, my client reported 4,322 messages written and after re-starting
ActiveMQ, the queue contained exactly 4,322 messages (same result regardless
whether I use KahaDB, LevelDB Java, or LevelDB JNI).
It appears that the AMQP Connector within ActiveMQ is acknowledging the
receipt of messages to the client *before* they actually got persisted to
disk. This would also explain the performance difference in my tests above.
Before I file a bug, I run the same tests against a qpid-cpp broker and see
whether the bug is in the AMQP 1-0 JMS client or in the ActiveMQ AMQP
Connector.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001p4676124.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Interesting Persistent Messaging Performance
Posted by Christian Posta <ch...@gmail.com>.
Wanna post your configs and test scripts?
On Tue, Jan 7, 2014 at 3:04 PM, uromahn <ul...@ulrichromahn.net> wrote:
> The AMQP client does not use asynchronous communication with the server. This
> is embedded in the AMQP protocol, so the client sends the message and the
> call to "send(msg)" only returns after the client received an ack
> confirmation from the broker via the protocol.
>
> What I don't know, however, is how the AMQP protocol connector within
> ActiveMQ works. It could be that the connector confirms the message before
> it actually got committed to the persistent store and hence the significant
> different performance numbers.
> The best way to test this would be to send a large number of messages to the
> broker, then kill the broker process and compare the number of sent messages
> on the client with the number of messages enqueued on the broker. If there
> is a difference (i.e. less messages on the broker), then we lost some
> messages and the guarantee is violated. This would then be a clear bug in
> the AMQP connector within ActiveMQ.
>
> You may hear from me back on the result of my test.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001p4676090.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Christian Posta
http://www.christianposta.com/blog
twitter: @christianposta
Re: Interesting Persistent Messaging Performance
Posted by uromahn <ul...@ulrichromahn.net>.
The AMQP client does not use asynchronous communication with the server. This
is embedded in the AMQP protocol, so the client sends the message and the
call to "send(msg)" only returns after the client received an ack
confirmation from the broker via the protocol.
What I don't know, however, is how the AMQP protocol connector within
ActiveMQ works. It could be that the connector confirms the message before
it actually got committed to the persistent store and hence the significant
different performance numbers.
The best way to test this would be to send a large number of messages to the
broker, then kill the broker process and compare the number of sent messages
on the client with the number of messages enqueued on the broker. If there
is a difference (i.e. less messages on the broker), then we lost some
messages and the guarantee is violated. This would then be a clear bug in
the AMQP connector within ActiveMQ.
You may hear from me back on the result of my test.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001p4676090.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Interesting Persistent Messaging Performance
Posted by artnaseef <ar...@artnaseef.com>.
Take careful look at how the send method works in both cases. I don't know
about the amqp interface, but the openwire connector can use asynchronous
communication with the server - meaning when the send() method returns, the
broker has not necessarily received the message, much less persisted it.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-tp4676001p4676012.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.