You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by kaipa <al...@mtu-net.ru> on 2006/08/24 17:53:32 UTC
Bad AMQ Linux performance
Hi,
I've read through archives and found that some people experience the same
problem but there were no solution. Tests work great fast on Windows Laptop
but unbarely slow on Linux server.
- I have the standard configuration from Latest AMQ release (4.*). The only
difference that I have tried Kaha persistence that works much faster than
JDBCJournal. I tried JDBCJournal as well -- perfomance difference is the
same.
- I use tcp transport
- Queue is persistent
- Session is transactional
First pair of tests commit session after every message. Second pair commit
after all messages are sent or received. I am especially concerned about
first pair that is too slow.
Windows:
Testing QService.sendMessage()
500 ops 453ms
Average speed: 1103 ops/s
Testing QService.consumeMessage()
500 ops 484ms
Average speed: 1033 ops/s
Testing QService.sendMessageNoCommit()
500 ops 188ms
Average speed: 2659 ops/s
Testing QService.consumeMessageNoCommit()
500 ops 125ms
Average speed: 4000 ops/s
Linux:
[java] Testing QService.sendMessage()
[java] 500 ops 22290ms
[java] Average speed: 22 ops/s
[java] Testing QService.consumeMessage()
[java] 500 ops 40366ms
[java] Average speed: 12 ops/s
[java] Testing QService.sendMessageNoCommit()
[java] 500 ops 926ms
[java] Average speed: 539 ops/s
[java] Testing QService.consumeMessageNoCommit()
[java] 500 ops 1017ms
[java] Average speed: 491 ops/s
I suspect it is something with activemq io libraries. Java web services work
fine on the same server, the problem is only with activemq.
Please, advise where I can look into
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5966799
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by aioaneid <ai...@gmail.com>.
I stumbled over pretty much the same issue on Linux:
http://issues.apache.org/activemq/browse/AMQ-905
I don't send msg transactionally, but I consume them like that. For sending
msg the only way I could improve performance with a single producer thread
is by specifying jms.useAsyncSend=true, but I'm not sure of the significance
of that parameter.
As for message consumption, I haven't come across a quick fix as yet.
How are you able to consume messages so fast in a non-transactional way ?
I'm using jencks and message driven pojos but couldn't make it consume any
faster than 11 msg/second.
kaipa wrote:
>
> Hi,
>
> I've read through archives and found that some people experience the same
> problem but there were no solution. Tests work great fast on Windows
> Laptop but unbarely slow on Linux server.
>
> - I have the standard configuration from Latest AMQ release (4.*). The
> only difference that I have tried Kaha persistence that works much faster
> than JDBCJournal. I tried JDBCJournal as well -- perfomance difference is
> the same.
> - I use tcp transport
> - Queue is persistent
> - Session is transactional
>
> First pair of tests commit session after every message. Second pair commit
> after all messages are sent or received. I am especially concerned about
> first pair that is too slow.
>
> Windows:
>
> Testing QService.sendMessage()
> 500 ops 453ms
> Average speed: 1103 ops/s
>
> Testing QService.consumeMessage()
> 500 ops 484ms
> Average speed: 1033 ops/s
>
> Testing QService.sendMessageNoCommit()
> 500 ops 188ms
> Average speed: 2659 ops/s
>
> Testing QService.consumeMessageNoCommit()
> 500 ops 125ms
> Average speed: 4000 ops/s
>
> Linux:
>
> [java] Testing QService.sendMessage()
> [java] 500 ops 22290ms
> [java] Average speed: 22 ops/s
>
> [java] Testing QService.consumeMessage()
> [java] 500 ops 40366ms
> [java] Average speed: 12 ops/s
>
> [java] Testing QService.sendMessageNoCommit()
> [java] 500 ops 926ms
> [java] Average speed: 539 ops/s
>
> [java] Testing QService.consumeMessageNoCommit()
> [java] 500 ops 1017ms
> [java] Average speed: 491 ops/s
>
> I suspect it is something with activemq io libraries. Java web services
> work fine on the same server, the problem is only with activemq.
>
> Please, advise where I can look into
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a6152154
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by AlB <Br...@avaya.com>.
I've seen this same issue. I've been using a profiler and found that a wait
is caused by commit in the producer (ultimately occurs in:
edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take). I
put in a counter to emulate "chunking up" the messages; that is, don't
perform a commit with every send; do the commit every 100th send. This had a
noticeable effect, confirmed by the profiler. I've also confirmed that this
same issue occurs when a subscriber commits as well.
I was curious if the solution offered in an earlier message was scheduled to
go in a patch or future release. I checked the issue mentioned above
(http://issues.apache.org/activemq/browse/AMQ-905) and didn't see any
further entires.
Thanks
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a7225431
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Bad AMQ Linux performance
Posted by kaipa <al...@mtu-net.ru>.
I can not build now:
--------------
1 required artifact is missing.
for artifact:
org.apache.maven.plugins:maven-jar-plugin:maven-plugin:2.1-20060826.162339-6
from the specified remote repositories:
central (http://repo1.maven.org/maven2),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
apache-snapshots (http://people.apache.org/maven-snapshot-repository)
James.Strachan wrote:
>
> On 8/25/06, kaipa <al...@mtu-net.ru> wrote:
>>
>> Right. But options are still not initialized.
>> I have stepped though using debugger -- for some reason it does not
>> recognize the option in
>> IntrospectionSupport.setProperties(tcpTransport, options);
>
> Sorry about that - I should have written a test case first before
> asking you to try it. I've just added a test case and fixed the bug
> that was preventing you configure these properties - want to try now
> with the latest from svn?
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a6016381
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by James Strachan <ja...@gmail.com>.
On 8/25/06, kaipa <al...@mtu-net.ru> wrote:
>
> Right. But options are still not initialized.
> I have stepped though using debugger -- for some reason it does not
> recognize the option in
> IntrospectionSupport.setProperties(tcpTransport, options);
Sorry about that - I should have written a test case first before
asking you to try it. I've just added a test case and fixed the bug
that was preventing you configure these properties - want to try now
with the latest from svn?
--
James
-------
http://radio.weblogs.com/0112098/
Re: Bad AMQ Linux performance
Posted by kaipa <al...@mtu-net.ru>.
Right. But options are still not initialized.
I have stepped though using debugger -- for some reason it does not
recognize the option in
IntrospectionSupport.setProperties(tcpTransport, options);
James.Strachan wrote:
>
> http://svn.apache.org/viewvc?view=rev&revision=436745
>
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5986085
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by James Strachan <ja...@gmail.com>.
On 8/25/06, kaipa <al...@mtu-net.ru> wrote:
>
> Hm, I used failover transport over tcp and just did not see an exception. Now
> I get an exception:
>
> javax.jms.JMSException: Could not connect to broker URL:
> tcp://qservice.webamg.local:61616?tcpNoDelay=false. Reason:
> java.lang.IllegalArgumentException: Invalid connect parameters:
> {tcpNoDelay=false}
>
> Looks like the code does not have your changes. I took the latest from
> trunk. What class have you changed? What revision?
http://svn.apache.org/viewvc?view=rev&revision=436745
--
James
-------
http://radio.weblogs.com/0112098/
Re: Bad AMQ Linux performance
Posted by kaipa <al...@mtu-net.ru>.
Hm, I used failover transport over tcp and just did not see an exception. Now
I get an exception:
javax.jms.JMSException: Could not connect to broker URL:
tcp://qservice.webamg.local:61616?tcpNoDelay=false. Reason:
java.lang.IllegalArgumentException: Invalid connect parameters:
{tcpNoDelay=false}
Looks like the code does not have your changes. I took the latest from
trunk. What class have you changed? What revision?
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5984940
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by James Strachan <ja...@gmail.com>.
What are the statistics like on the box? Is it swapping or IO bound?
Does it have plenty of RAM?
On 8/25/06, kaipa <al...@mtu-net.ru> wrote:
> >> No this does not work. If I specify tcpNoDelay option -- I can not
> >> connect to
> >> the broker.
> >
> > Why, what's the error?
>
> It waits forever.
Which version? tcpNoDelay=true or tcpNoDelay=false?
--
James
-------
http://radio.weblogs.com/0112098/
Re: Bad AMQ Linux performance
Posted by kaipa <al...@mtu-net.ru>.
>>
>> No this does not work. If I specify tcpNoDelay option -- I can not
>> connect to
>> the broker.
>
> Why, what's the error?
It waits forever.
>> Anyway, I doubt that it can make such a big difference.
>
> I do otherwise I wouldn't have taken the time to make the patch to the
> code and then wrttien an email to you about it :)
>
> This setting toggles on and off whether or not TCP waits for the
> packet to fill up before its sent to the other process. Turning it on
> and off can have a massive effect on performance (latency versus
> throughput). You are seeing a latency issue.
>
>
Yes, maybe. But I have never experienced anything similar before with same
computers.
>> There is something
>> else that blocks transactions. Note, if I turn transactions off --
>> everything works fast on Linux as well.
>
> Are you using persistence? If not I think the transactions are a bit
> of a red herring as when using non persistence without transactions we
> do everything asynchronous anyway. Persistent messaging should be
> compariable in performance between transactions and no transactions.
>
Yes, I am using persistence. But as I noted in one of previous posts, if I
turn persistence off -- the same, linux broker is slow in transactional
mode.
James
-------
http://radio.weblogs.com/0112098/
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5984479
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by James Strachan <ja...@gmail.com>.
On 8/25/06, kaipa <al...@mtu-net.ru> wrote:
>
> No this does not work. If I specify tcpNoDelay option -- I can not connect to
> the broker.
Why, what's the error?
> Anyway, I doubt that it can make such a big difference.
I do otherwise I wouldn't have taken the time to make the patch to the
code and then wrttien an email to you about it :)
This setting toggles on and off whether or not TCP waits for the
packet to fill up before its sent to the other process. Turning it on
and off can have a massive effect on performance (latency versus
throughput). You are seeing a latency issue.
> There is something
> else that blocks transactions. Note, if I turn transactions off --
> everything works fast on Linux as well.
Are you using persistence? If not I think the transactions are a bit
of a red herring as when using non persistence without transactions we
do everything asynchronous anyway. Persistent messaging should be
compariable in performance between transactions and no transactions.
--
James
-------
http://radio.weblogs.com/0112098/
Re: Bad AMQ Linux performance
Posted by kaipa <al...@mtu-net.ru>.
No this does not work. If I specify tcpNoDelay option -- I can not connect to
the broker.
Anyway, I doubt that it can make such a big difference. There is something
else that blocks transactions. Note, if I turn transactions off --
everything works fast on Linux as well.
I have tried different Linux installation (even different distribution) --
same thing.
James.Strachan wrote:
>
> The only thing I can think of is you're hitting the Nagler part of TCP
> - the TCP_NODELAY setting.
>
> I've just patched SVN HEAD to allow you to explicitly set/unset the
> Socket.setTcpNoDelay() option. If you grab the latest source code
> http://incubator.apache.org/activemq/source.html
>
> then building it
> http://incubator.apache.org/activemq/building.html
>
> you should be able to try linux with these 2 URLs to see if it makes a
> difference...
>
> tcp://localhost:61616?tcpNoDelay=true
> tcp://localhost:61616?tcpNoDelay=false
>
> Let us know if thats it.
>
>
> On 8/25/06, kaipa <al...@mtu-net.ru> wrote:
>>
>> Hi James,
>>
>> Thank you for reply. I still can not get it solved.
>>
>> 1. Yes, I send a number of messages to the queue first and then start to
>> receive
>> 2. If I disable persistence Linux is still that bad.
>> 3. However, if I disable transactions -- everything works fine and fast
>> both
>> with/without persistence. So the problem is in how transactions are
>> handled,
>> I believe.
>> 4. Broker is running on local file system.
>>
>> I have tried simpler test scenario: send-commit-receive-commit. On the
>> same
>> connection and session.
>>
>> Windows:
>> Testing QService.sendMessage() followed by QService.receiveMessage()
>> 1000 ops 1578ms
>> Average speed: 633 ops/s
>>
>> Linux:
>> Testing QService.sendMessage() followed by QService.receiveMessage()
>> 1000 ops 202094ms
>> Average speed: 4 ops/s
>>
>> I am absolutely lost!
>>
>> I have also tried to split message consumption into several threads and
>> sessions -- that improves message consumption proportionally but this is
>> not
>> the root cause.
>>
>>
>>
>>
>> James.Strachan wrote:
>> >
>> > BTW for the consumer tests, are there pleny of messages on the queue
>> > before you start trying to consume? How does the performance differ
>> > between linux and windows if you disable persistence?
>> >
>> > The only real difference between the two - assuming things are not
>> > CPU, network or IO bound is the performance of the disk writes.
>> > Persistent sending/consuming of messages by default requires blocking
>> > the client until the broker does a sync-to-disk of the journal.
>> >
>> > Is the broker running on a local file system or a shared network drive?
>> >
>> > On 8/24/06, kaipa <al...@mtu-net.ru> wrote:
>> >> Hi,
>> >>
>> >> I've read through archives and found that some people experience the
>> same
>> >> problem but there were no solution. Tests work great fast on Windows
>> >> Laptop
>> >> but unbarely slow on Linux server.
>> >>
>> >> - I have the standard configuration from Latest AMQ release (4.*). The
>> >> only
>> >> difference that I have tried Kaha persistence that works much faster
>> than
>> >> JDBCJournal. I tried JDBCJournal as well -- perfomance difference is
>> the
>> >> same.
>> >> - I use tcp transport
>> >> - Queue is persistent
>> >> - Session is transactional
>> >>
>> >> First pair of tests commit session after every message. Second pair
>> >> commit
>> >> after all messages are sent or received. I am especially concerned
>> about
>> >> first pair that is too slow.
>> >>
>> >> Windows:
>> >>
>> >> Testing QService.sendMessage()
>> >> 500 ops 453ms
>> >> Average speed: 1103 ops/s
>> >>
>> >> Testing QService.consumeMessage()
>> >> 500 ops 484ms
>> >> Average speed: 1033 ops/s
>> >>
>> >> Testing QService.sendMessageNoCommit()
>> >> 500 ops 188ms
>> >> Average speed: 2659 ops/s
>> >>
>> >> Testing QService.consumeMessageNoCommit()
>> >> 500 ops 125ms
>> >> Average speed: 4000 ops/s
>> >>
>> >> Linux:
>> >>
>> >> [java] Testing QService.sendMessage()
>> >> [java] 500 ops 22290ms
>> >> [java] Average speed: 22 ops/s
>> >>
>> >> [java] Testing QService.consumeMessage()
>> >> [java] 500 ops 40366ms
>> >> [java] Average speed: 12 ops/s
>> >>
>> >> [java] Testing QService.sendMessageNoCommit()
>> >> [java] 500 ops 926ms
>> >> [java] Average speed: 539 ops/s
>> >>
>> >> [java] Testing QService.consumeMessageNoCommit()
>> >> [java] 500 ops 1017ms
>> >> [java] Average speed: 491 ops/s
>> >>
>> >> I suspect it is something with activemq io libraries. Java web
>> services
>> >> work
>> >> fine on the same server, the problem is only with activemq.
>> >>
>> >> Please, advise where I can look into
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5966799
>> >> Sent from the ActiveMQ - User forum at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> >
>> > James
>> > -------
>> > http://radio.weblogs.com/0112098/
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5980491
>> Sent from the ActiveMQ - User forum at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5983897
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by kaipa <al...@mtu-net.ru>.
James,
I have finally tried that. No good news. There is no or very liittle
difference when using tcpNoDelay option.
James.Strachan wrote:
>
> The only thing I can think of is you're hitting the Nagler part of TCP
> - the TCP_NODELAY setting.
>
> I've just patched SVN HEAD to allow you to explicitly set/unset the
> Socket.setTcpNoDelay() option. If you grab the latest source code
> http://incubator.apache.org/activemq/source.html
>
> then building it
> http://incubator.apache.org/activemq/building.html
>
> you should be able to try linux with these 2 URLs to see if it makes a
> difference...
>
> tcp://localhost:61616?tcpNoDelay=true
> tcp://localhost:61616?tcpNoDelay=false
>
> Let us know if thats it.
>
>
> On 8/25/06, kaipa <al...@mtu-net.ru> wrote:
>>
>> Hi James,
>>
>> Thank you for reply. I still can not get it solved.
>>
>> 1. Yes, I send a number of messages to the queue first and then start to
>> receive
>> 2. If I disable persistence Linux is still that bad.
>> 3. However, if I disable transactions -- everything works fine and fast
>> both
>> with/without persistence. So the problem is in how transactions are
>> handled,
>> I believe.
>> 4. Broker is running on local file system.
>>
>> I have tried simpler test scenario: send-commit-receive-commit. On the
>> same
>> connection and session.
>>
>> Windows:
>> Testing QService.sendMessage() followed by QService.receiveMessage()
>> 1000 ops 1578ms
>> Average speed: 633 ops/s
>>
>> Linux:
>> Testing QService.sendMessage() followed by QService.receiveMessage()
>> 1000 ops 202094ms
>> Average speed: 4 ops/s
>>
>> I am absolutely lost!
>>
>> I have also tried to split message consumption into several threads and
>> sessions -- that improves message consumption proportionally but this is
>> not
>> the root cause.
>>
>>
>>
>>
>> James.Strachan wrote:
>> >
>> > BTW for the consumer tests, are there pleny of messages on the queue
>> > before you start trying to consume? How does the performance differ
>> > between linux and windows if you disable persistence?
>> >
>> > The only real difference between the two - assuming things are not
>> > CPU, network or IO bound is the performance of the disk writes.
>> > Persistent sending/consuming of messages by default requires blocking
>> > the client until the broker does a sync-to-disk of the journal.
>> >
>> > Is the broker running on a local file system or a shared network drive?
>> >
>> > On 8/24/06, kaipa <al...@mtu-net.ru> wrote:
>> >> Hi,
>> >>
>> >> I've read through archives and found that some people experience the
>> same
>> >> problem but there were no solution. Tests work great fast on Windows
>> >> Laptop
>> >> but unbarely slow on Linux server.
>> >>
>> >> - I have the standard configuration from Latest AMQ release (4.*). The
>> >> only
>> >> difference that I have tried Kaha persistence that works much faster
>> than
>> >> JDBCJournal. I tried JDBCJournal as well -- perfomance difference is
>> the
>> >> same.
>> >> - I use tcp transport
>> >> - Queue is persistent
>> >> - Session is transactional
>> >>
>> >> First pair of tests commit session after every message. Second pair
>> >> commit
>> >> after all messages are sent or received. I am especially concerned
>> about
>> >> first pair that is too slow.
>> >>
>> >> Windows:
>> >>
>> >> Testing QService.sendMessage()
>> >> 500 ops 453ms
>> >> Average speed: 1103 ops/s
>> >>
>> >> Testing QService.consumeMessage()
>> >> 500 ops 484ms
>> >> Average speed: 1033 ops/s
>> >>
>> >> Testing QService.sendMessageNoCommit()
>> >> 500 ops 188ms
>> >> Average speed: 2659 ops/s
>> >>
>> >> Testing QService.consumeMessageNoCommit()
>> >> 500 ops 125ms
>> >> Average speed: 4000 ops/s
>> >>
>> >> Linux:
>> >>
>> >> [java] Testing QService.sendMessage()
>> >> [java] 500 ops 22290ms
>> >> [java] Average speed: 22 ops/s
>> >>
>> >> [java] Testing QService.consumeMessage()
>> >> [java] 500 ops 40366ms
>> >> [java] Average speed: 12 ops/s
>> >>
>> >> [java] Testing QService.sendMessageNoCommit()
>> >> [java] 500 ops 926ms
>> >> [java] Average speed: 539 ops/s
>> >>
>> >> [java] Testing QService.consumeMessageNoCommit()
>> >> [java] 500 ops 1017ms
>> >> [java] Average speed: 491 ops/s
>> >>
>> >> I suspect it is something with activemq io libraries. Java web
>> services
>> >> work
>> >> fine on the same server, the problem is only with activemq.
>> >>
>> >> Please, advise where I can look into
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5966799
>> >> Sent from the ActiveMQ - User forum at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> >
>> > James
>> > -------
>> > http://radio.weblogs.com/0112098/
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5980491
>> Sent from the ActiveMQ - User forum at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a6134911
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by James Strachan <ja...@gmail.com>.
The only thing I can think of is you're hitting the Nagler part of TCP
- the TCP_NODELAY setting.
I've just patched SVN HEAD to allow you to explicitly set/unset the
Socket.setTcpNoDelay() option. If you grab the latest source code
http://incubator.apache.org/activemq/source.html
then building it
http://incubator.apache.org/activemq/building.html
you should be able to try linux with these 2 URLs to see if it makes a
difference...
tcp://localhost:61616?tcpNoDelay=true
tcp://localhost:61616?tcpNoDelay=false
Let us know if thats it.
On 8/25/06, kaipa <al...@mtu-net.ru> wrote:
>
> Hi James,
>
> Thank you for reply. I still can not get it solved.
>
> 1. Yes, I send a number of messages to the queue first and then start to
> receive
> 2. If I disable persistence Linux is still that bad.
> 3. However, if I disable transactions -- everything works fine and fast both
> with/without persistence. So the problem is in how transactions are handled,
> I believe.
> 4. Broker is running on local file system.
>
> I have tried simpler test scenario: send-commit-receive-commit. On the same
> connection and session.
>
> Windows:
> Testing QService.sendMessage() followed by QService.receiveMessage()
> 1000 ops 1578ms
> Average speed: 633 ops/s
>
> Linux:
> Testing QService.sendMessage() followed by QService.receiveMessage()
> 1000 ops 202094ms
> Average speed: 4 ops/s
>
> I am absolutely lost!
>
> I have also tried to split message consumption into several threads and
> sessions -- that improves message consumption proportionally but this is not
> the root cause.
>
>
>
>
> James.Strachan wrote:
> >
> > BTW for the consumer tests, are there pleny of messages on the queue
> > before you start trying to consume? How does the performance differ
> > between linux and windows if you disable persistence?
> >
> > The only real difference between the two - assuming things are not
> > CPU, network or IO bound is the performance of the disk writes.
> > Persistent sending/consuming of messages by default requires blocking
> > the client until the broker does a sync-to-disk of the journal.
> >
> > Is the broker running on a local file system or a shared network drive?
> >
> > On 8/24/06, kaipa <al...@mtu-net.ru> wrote:
> >> Hi,
> >>
> >> I've read through archives and found that some people experience the same
> >> problem but there were no solution. Tests work great fast on Windows
> >> Laptop
> >> but unbarely slow on Linux server.
> >>
> >> - I have the standard configuration from Latest AMQ release (4.*). The
> >> only
> >> difference that I have tried Kaha persistence that works much faster than
> >> JDBCJournal. I tried JDBCJournal as well -- perfomance difference is the
> >> same.
> >> - I use tcp transport
> >> - Queue is persistent
> >> - Session is transactional
> >>
> >> First pair of tests commit session after every message. Second pair
> >> commit
> >> after all messages are sent or received. I am especially concerned about
> >> first pair that is too slow.
> >>
> >> Windows:
> >>
> >> Testing QService.sendMessage()
> >> 500 ops 453ms
> >> Average speed: 1103 ops/s
> >>
> >> Testing QService.consumeMessage()
> >> 500 ops 484ms
> >> Average speed: 1033 ops/s
> >>
> >> Testing QService.sendMessageNoCommit()
> >> 500 ops 188ms
> >> Average speed: 2659 ops/s
> >>
> >> Testing QService.consumeMessageNoCommit()
> >> 500 ops 125ms
> >> Average speed: 4000 ops/s
> >>
> >> Linux:
> >>
> >> [java] Testing QService.sendMessage()
> >> [java] 500 ops 22290ms
> >> [java] Average speed: 22 ops/s
> >>
> >> [java] Testing QService.consumeMessage()
> >> [java] 500 ops 40366ms
> >> [java] Average speed: 12 ops/s
> >>
> >> [java] Testing QService.sendMessageNoCommit()
> >> [java] 500 ops 926ms
> >> [java] Average speed: 539 ops/s
> >>
> >> [java] Testing QService.consumeMessageNoCommit()
> >> [java] 500 ops 1017ms
> >> [java] Average speed: 491 ops/s
> >>
> >> I suspect it is something with activemq io libraries. Java web services
> >> work
> >> fine on the same server, the problem is only with activemq.
> >>
> >> Please, advise where I can look into
> >> --
> >> View this message in context:
> >> http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5966799
> >> Sent from the ActiveMQ - User forum at Nabble.com.
> >>
> >>
> >
> >
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
> >
>
> --
> View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5980491
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Bad AMQ Linux performance
Posted by kaipa <al...@mtu-net.ru>.
Hi James,
Thank you for reply. I still can not get it solved.
1. Yes, I send a number of messages to the queue first and then start to
receive
2. If I disable persistence Linux is still that bad.
3. However, if I disable transactions -- everything works fine and fast both
with/without persistence. So the problem is in how transactions are handled,
I believe.
4. Broker is running on local file system.
I have tried simpler test scenario: send-commit-receive-commit. On the same
connection and session.
Windows:
Testing QService.sendMessage() followed by QService.receiveMessage()
1000 ops 1578ms
Average speed: 633 ops/s
Linux:
Testing QService.sendMessage() followed by QService.receiveMessage()
1000 ops 202094ms
Average speed: 4 ops/s
I am absolutely lost!
I have also tried to split message consumption into several threads and
sessions -- that improves message consumption proportionally but this is not
the root cause.
James.Strachan wrote:
>
> BTW for the consumer tests, are there pleny of messages on the queue
> before you start trying to consume? How does the performance differ
> between linux and windows if you disable persistence?
>
> The only real difference between the two - assuming things are not
> CPU, network or IO bound is the performance of the disk writes.
> Persistent sending/consuming of messages by default requires blocking
> the client until the broker does a sync-to-disk of the journal.
>
> Is the broker running on a local file system or a shared network drive?
>
> On 8/24/06, kaipa <al...@mtu-net.ru> wrote:
>> Hi,
>>
>> I've read through archives and found that some people experience the same
>> problem but there were no solution. Tests work great fast on Windows
>> Laptop
>> but unbarely slow on Linux server.
>>
>> - I have the standard configuration from Latest AMQ release (4.*). The
>> only
>> difference that I have tried Kaha persistence that works much faster than
>> JDBCJournal. I tried JDBCJournal as well -- perfomance difference is the
>> same.
>> - I use tcp transport
>> - Queue is persistent
>> - Session is transactional
>>
>> First pair of tests commit session after every message. Second pair
>> commit
>> after all messages are sent or received. I am especially concerned about
>> first pair that is too slow.
>>
>> Windows:
>>
>> Testing QService.sendMessage()
>> 500 ops 453ms
>> Average speed: 1103 ops/s
>>
>> Testing QService.consumeMessage()
>> 500 ops 484ms
>> Average speed: 1033 ops/s
>>
>> Testing QService.sendMessageNoCommit()
>> 500 ops 188ms
>> Average speed: 2659 ops/s
>>
>> Testing QService.consumeMessageNoCommit()
>> 500 ops 125ms
>> Average speed: 4000 ops/s
>>
>> Linux:
>>
>> [java] Testing QService.sendMessage()
>> [java] 500 ops 22290ms
>> [java] Average speed: 22 ops/s
>>
>> [java] Testing QService.consumeMessage()
>> [java] 500 ops 40366ms
>> [java] Average speed: 12 ops/s
>>
>> [java] Testing QService.sendMessageNoCommit()
>> [java] 500 ops 926ms
>> [java] Average speed: 539 ops/s
>>
>> [java] Testing QService.consumeMessageNoCommit()
>> [java] 500 ops 1017ms
>> [java] Average speed: 491 ops/s
>>
>> I suspect it is something with activemq io libraries. Java web services
>> work
>> fine on the same server, the problem is only with activemq.
>>
>> Please, advise where I can look into
>> --
>> View this message in context:
>> http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5966799
>> Sent from the ActiveMQ - User forum at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5980491
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by James Strachan <ja...@gmail.com>.
BTW for the consumer tests, are there pleny of messages on the queue
before you start trying to consume? How does the performance differ
between linux and windows if you disable persistence?
The only real difference between the two - assuming things are not
CPU, network or IO bound is the performance of the disk writes.
Persistent sending/consuming of messages by default requires blocking
the client until the broker does a sync-to-disk of the journal.
Is the broker running on a local file system or a shared network drive?
On 8/24/06, kaipa <al...@mtu-net.ru> wrote:
> Hi,
>
> I've read through archives and found that some people experience the same
> problem but there were no solution. Tests work great fast on Windows Laptop
> but unbarely slow on Linux server.
>
> - I have the standard configuration from Latest AMQ release (4.*). The only
> difference that I have tried Kaha persistence that works much faster than
> JDBCJournal. I tried JDBCJournal as well -- perfomance difference is the
> same.
> - I use tcp transport
> - Queue is persistent
> - Session is transactional
>
> First pair of tests commit session after every message. Second pair commit
> after all messages are sent or received. I am especially concerned about
> first pair that is too slow.
>
> Windows:
>
> Testing QService.sendMessage()
> 500 ops 453ms
> Average speed: 1103 ops/s
>
> Testing QService.consumeMessage()
> 500 ops 484ms
> Average speed: 1033 ops/s
>
> Testing QService.sendMessageNoCommit()
> 500 ops 188ms
> Average speed: 2659 ops/s
>
> Testing QService.consumeMessageNoCommit()
> 500 ops 125ms
> Average speed: 4000 ops/s
>
> Linux:
>
> [java] Testing QService.sendMessage()
> [java] 500 ops 22290ms
> [java] Average speed: 22 ops/s
>
> [java] Testing QService.consumeMessage()
> [java] 500 ops 40366ms
> [java] Average speed: 12 ops/s
>
> [java] Testing QService.sendMessageNoCommit()
> [java] 500 ops 926ms
> [java] Average speed: 539 ops/s
>
> [java] Testing QService.consumeMessageNoCommit()
> [java] 500 ops 1017ms
> [java] Average speed: 491 ops/s
>
> I suspect it is something with activemq io libraries. Java web services work
> fine on the same server, the problem is only with activemq.
>
> Please, advise where I can look into
> --
> View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a5966799
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Bad AMQ Linux performance
Posted by garima015 <ga...@rediffmail.com>.
I am facing the same problem.Is aybody having ny solution for this
kaipa wrote:
>
> Thanks, but it does not help either. I have tried on old Linux 2.6.5 as
> well -- same slow.
>
>
>
> Rickard wrote:
>>
>> This might be related to a change in the linux kernel 2.6.15+.
>>
>> Without going into details, you could try:
>>
>> sudo sysctl -w net.ipv4.tcp_abc=0
>> Which turns of the new feature.
>>
>>
>> kaipa wrote:
>>>
>>> Hi,
>>>
>>> I've read through archives and found that some people experience the
>>> same problem but there were no solution. Tests work great fast on
>>> Windows Laptop but unbarely slow on Linux server.
>>>
>>> - I have the standard configuration from Latest AMQ release (4.*). The
>>> only difference that I have tried Kaha persistence that works much
>>> faster than JDBCJournal. I tried JDBCJournal as well -- perfomance
>>> difference is the same.
>>> - I use tcp transport
>>> - Queue is persistent
>>> - Session is transactional
>>>
>>> First pair of tests commit session after every message. Second pair
>>> commit after all messages are sent or received. I am especially
>>> concerned about first pair that is too slow.
>>>
>>> Windows:
>>>
>>> Testing QService.sendMessage()
>>> 500 ops 453ms
>>> Average speed: 1103 ops/s
>>>
>>> Testing QService.consumeMessage()
>>> 500 ops 484ms
>>> Average speed: 1033 ops/s
>>>
>>> Testing QService.sendMessageNoCommit()
>>> 500 ops 188ms
>>> Average speed: 2659 ops/s
>>>
>>> Testing QService.consumeMessageNoCommit()
>>> 500 ops 125ms
>>> Average speed: 4000 ops/s
>>>
>>> Linux:
>>>
>>> [java] Testing QService.sendMessage()
>>> [java] 500 ops 22290ms
>>> [java] Average speed: 22 ops/s
>>>
>>> [java] Testing QService.consumeMessage()
>>> [java] 500 ops 40366ms
>>> [java] Average speed: 12 ops/s
>>>
>>> [java] Testing QService.sendMessageNoCommit()
>>> [java] 500 ops 926ms
>>> [java] Average speed: 539 ops/s
>>>
>>> [java] Testing QService.consumeMessageNoCommit()
>>> [java] 500 ops 1017ms
>>> [java] Average speed: 491 ops/s
>>>
>>> I suspect it is something with activemq io libraries. Java web services
>>> work fine on the same server, the problem is only with activemq.
>>>
>>> Please, advise where I can look into
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a7935963
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Bad AMQ Linux performance
Posted by kaipa <al...@mtu-net.ru>.
Thanks, but it does not help either. I have tried on old Linux 2.6.5 as well
-- same slow.
Rickard wrote:
>
> This might be related to a change in the linux kernel 2.6.15+.
>
> Without going into details, you could try:
>
> sudo sysctl -w net.ipv4.tcp_abc=0
> Which turns of the new feature.
>
>
> kaipa wrote:
>>
>> Hi,
>>
>> I've read through archives and found that some people experience the same
>> problem but there were no solution. Tests work great fast on Windows
>> Laptop but unbarely slow on Linux server.
>>
>> - I have the standard configuration from Latest AMQ release (4.*). The
>> only difference that I have tried Kaha persistence that works much faster
>> than JDBCJournal. I tried JDBCJournal as well -- perfomance difference is
>> the same.
>> - I use tcp transport
>> - Queue is persistent
>> - Session is transactional
>>
>> First pair of tests commit session after every message. Second pair
>> commit after all messages are sent or received. I am especially concerned
>> about first pair that is too slow.
>>
>> Windows:
>>
>> Testing QService.sendMessage()
>> 500 ops 453ms
>> Average speed: 1103 ops/s
>>
>> Testing QService.consumeMessage()
>> 500 ops 484ms
>> Average speed: 1033 ops/s
>>
>> Testing QService.sendMessageNoCommit()
>> 500 ops 188ms
>> Average speed: 2659 ops/s
>>
>> Testing QService.consumeMessageNoCommit()
>> 500 ops 125ms
>> Average speed: 4000 ops/s
>>
>> Linux:
>>
>> [java] Testing QService.sendMessage()
>> [java] 500 ops 22290ms
>> [java] Average speed: 22 ops/s
>>
>> [java] Testing QService.consumeMessage()
>> [java] 500 ops 40366ms
>> [java] Average speed: 12 ops/s
>>
>> [java] Testing QService.sendMessageNoCommit()
>> [java] 500 ops 926ms
>> [java] Average speed: 539 ops/s
>>
>> [java] Testing QService.consumeMessageNoCommit()
>> [java] 500 ops 1017ms
>> [java] Average speed: 491 ops/s
>>
>> I suspect it is something with activemq io libraries. Java web services
>> work fine on the same server, the problem is only with activemq.
>>
>> Please, advise where I can look into
>>
>
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a6016375
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Bad AMQ Linux performance
Posted by Rickard <ec...@joshua.haninge.kth.se>.
This might be related to a change in the linux kernel 2.6.15+.
Without going into details, you could try:
sudo sysctl -w net.ipv4.tcp_abc=0
Which turns of the new feature.
kaipa wrote:
>
> Hi,
>
> I've read through archives and found that some people experience the same
> problem but there were no solution. Tests work great fast on Windows
> Laptop but unbarely slow on Linux server.
>
> - I have the standard configuration from Latest AMQ release (4.*). The
> only difference that I have tried Kaha persistence that works much faster
> than JDBCJournal. I tried JDBCJournal as well -- perfomance difference is
> the same.
> - I use tcp transport
> - Queue is persistent
> - Session is transactional
>
> First pair of tests commit session after every message. Second pair commit
> after all messages are sent or received. I am especially concerned about
> first pair that is too slow.
>
> Windows:
>
> Testing QService.sendMessage()
> 500 ops 453ms
> Average speed: 1103 ops/s
>
> Testing QService.consumeMessage()
> 500 ops 484ms
> Average speed: 1033 ops/s
>
> Testing QService.sendMessageNoCommit()
> 500 ops 188ms
> Average speed: 2659 ops/s
>
> Testing QService.consumeMessageNoCommit()
> 500 ops 125ms
> Average speed: 4000 ops/s
>
> Linux:
>
> [java] Testing QService.sendMessage()
> [java] 500 ops 22290ms
> [java] Average speed: 22 ops/s
>
> [java] Testing QService.consumeMessage()
> [java] 500 ops 40366ms
> [java] Average speed: 12 ops/s
>
> [java] Testing QService.sendMessageNoCommit()
> [java] 500 ops 926ms
> [java] Average speed: 539 ops/s
>
> [java] Testing QService.consumeMessageNoCommit()
> [java] 500 ops 1017ms
> [java] Average speed: 491 ops/s
>
> I suspect it is something with activemq io libraries. Java web services
> work fine on the same server, the problem is only with activemq.
>
> Please, advise where I can look into
>
--
View this message in context: http://www.nabble.com/Bad-AMQ-Linux-performance-tf2159490.html#a6015825
Sent from the ActiveMQ - User forum at Nabble.com.