Active MQ - Performance with CLIENT Ack

[akpuvvada <ak...@gmail.com>]

I am trying to test Active MQ for Performance
I have installed it in Unix in 'Shared File System' Fault Tolerant pair.
Max Heap Size is 3 GB
4GB Persistence Store Size
2Gb Temporary Store Size

I am trying to publish messages with 'Non_Persistent' Delivery mode with
Payload of size 1 MB without any delay in a loop - 10K messages.
I am trying to consume the messages with Clinet ACK and acknowledging after
logging the receipt of message.

Prouder has published 3k+ messages and slowed down as Temp Store size is
only 2GB.

However, Subscriber is consuming the messages at a very slow rate and not
acknowledging or Active MQ is not processing the acknowledgements.

Please advise what I am doing wrong or what should be changed to improve the
performance?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Active-MQ-Performance-with-CLIENT-Ack-tp4728551.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ - Performance with CLIENT Ack

[Tim Bain <tb...@alumni.duke.edu>]

Luckily, you don't need access to the code TIBCO BW generates in order to
test this. You can set a breakpoint on the
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.activemq/activemq-all/5.10.0/org/apache/activemq/ActiveMQMessageConsumer.java#ActiveMQMessageConsumer.acknowledge%28%29
method and attach a debugger to the JVM that's running your TIBCO code, and
you should be able to tell whether the method is being called even if you
don't have access to the generated source code. Another place you could set
a breakpoint is in ActiveMQSession.sendAck():
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.activemq/activemq-all/5.10.0/org/apache/activemq/ActiveMQSession.java#ActiveMQSession.sendAck%28org.apache.activemq.command.MessageAck%29

One other option for investigating this would be to attach a debugger to
the broker and set a breakpoint on TransportConnection.processMessageAck()
-
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.activemq/activemq-all/5.10.0/org/apache/activemq/broker/TransportConnection.java#TransportConnection.processMessageAck%28org.apache.activemq.command.MessageAck%29.
That should let you tell whether the ack is making it to the broker, which
tells us which side of the broker-client divide we should be focusing on.

And although I agree that log lines saying an acknowledgement is sent are
promising, I wouldn't consider that to be confirmation that it's actually
happening, which is why I recommend the debugger for a conclusive result.

You're correct that CLIENT_ACKNOWLEDGE means "acknowledge this message and
all previous ones." It's not something I've used personally and I've seen
very few posts to the mailing list about it, but I don't believe that the
broker would fail to process the ack if it's being sent to the broker. The
only situation I can think of would be one where the producer and consumer
were sharing a single connection and the producer started well before the
consumer and completely filled the broker before the consumer even got
started, and then you had also configured the broker so it couldn't do
per-client producer flow control and it had to block the full TCP
connection that they both share. That scenario is possible, but requires
several not-very-likely things to be true at the same time, so it strikes
me as unlikely.

Also, your suggestion about ActiveMQ waiting because it's busy sounds like
the description of a performance problem, which would be expected to result
in acks eventually (though slowly) being processed, not of a correctness
problem where 100% failure occurs, so if I've correctly understood what you
suggested, then I think it's not likely to be what's happening.

BTW, is producer flow control kicking in when this occurs? And when temp
fills, is it eventually clearing, or does it fill and stay full forever
(even if you stop the producers)?

Tim

On Tue, Jul 18, 2017 at 4:00 AM, akpuvvada <ak...@gmail.com> wrote:

> TestActiveMQ.zip
> <http://activemq.2283324.n4.nabble.com/file/n4728623/TestActiveMQ.zip>
>
> Attached the TIBCO BW Project I am using.
>
> It is not a Java code, TIBCO BW is a GUI (Zero Coding) Integration tool
> that
> stored the configuration in XML and converts the XMLs to Java code at
> Run-time and execute. So I have no idea how this actually works and when
> the
> acknowledge() gets executed.
> However, I can confirm from the logs that Acknowledgement is sent by the
> code.
>
> Active MQ is not accepting or not processing the acknowledgement.
> I read that Client acknowledgement is actually "Acknowledge All Previous".
> Would Active MQ wait to process Acks if it is too busy with other things.
> Producer is producing at a very fast rate and temp space has filled-up fast
> when this is observed. Could that be an issue?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Performance-with-CLIENT-Ack-tp4728551p4728623.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Active MQ - Performance with CLIENT Ack

[akpuvvada <ak...@gmail.com>]

TestActiveMQ.zip
<http://activemq.2283324.n4.nabble.com/file/n4728623/TestActiveMQ.zip>  

Attached the TIBCO BW Project I am using.

It is not a Java code, TIBCO BW is a GUI (Zero Coding) Integration tool that
stored the configuration in XML and converts the XMLs to Java code at
Run-time and execute. So I have no idea how this actually works and when the
acknowledge() gets executed.
However, I can confirm from the logs that Acknowledgement is sent by the
code.

Active MQ is not accepting or not processing the acknowledgement.
I read that Client acknowledgement is actually "Acknowledge All Previous".
Would Active MQ wait to process Acks if it is too busy with other things.
Producer is producing at a very fast rate and temp space has filled-up fast
when this is observed. Could that be an issue?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Active-MQ-Performance-with-CLIENT-Ack-tp4728551p4728623.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ - Performance with CLIENT Ack

[Tim Bain <tb...@alumni.duke.edu>]

If the dequeue count is staying at 0, then this isn't a performance
problem, and you can ignore that portion of my earlier email in this
thread.

It sounds like something's not right, either with your code or with ours.
Can you please post the consumer code you're using, so we can see if
there's anything obvious about it that might need to change? Also, can you
please attach a debugger to your consumer process and set a breakpoint on
the call to acknowledge() and confirm that it's actually being called? (If
some branch in the code caused you to miss that call, that would explain
everything.)

Tim

On Jul 15, 2017 8:02 PM, "akpuvvada" <ak...@gmail.com> wrote:

> When I tried multithreaded (processing parallel) the subscriber is not
> logging any receipt. That made me affectively conclude it is not processing
> any messages. However, when I tried single thread (serial processing) I can
> see the subscriber is updating the log; so I know it is receiving the
> messages and sending back acknowledgement . However, in the Active MQ UI, I
> can see that all messages are in pending state and dequeued is 0.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Performance-with-CLIENT-Ack-tp4728551p4728578.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Active MQ - Performance with CLIENT Ack

[akpuvvada <ak...@gmail.com>]

When I tried multithreaded (processing parallel) the subscriber is not
logging any receipt. That made me affectively conclude it is not processing
any messages. However, when I tried single thread (serial processing) I can
see the subscriber is updating the log; so I know it is receiving the
messages and sending back acknowledgement . However, in the Active MQ UI, I
can see that all messages are in pending state and dequeued is 0.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Active-MQ-Performance-with-CLIENT-Ack-tp4728551p4728578.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ - Performance with CLIENT Ack

[Tim Bain <tb...@alumni.duke.edu>]

Is your workflow process-ack-process-ack-process-ack (one message at a time
immediately followed by acking that one message, in a single thread)? If
so, I'd expect your processing to be slower than if you only acknowledge
every Nth message, though I'm not sure how significant of a performance
impact I'd expect that to be.

Can you describe in more detail what you're actually observing? Are you
slowly but correctly consuming messages (i.e. this is a performance problem
but not a correctness problem)? Or is the problem that messages are not
being acknowledged at all (none, zip, zero), which makes this a correctness
problem? You've described both types of problem in your description, but
any problem is by definition only one or the other, and knowing what the
actual problem is will help us focus on what's actually wrong.

Tim

On Fri, Jul 14, 2017 at 6:54 PM, akpuvvada <ak...@gmail.com> wrote:

> We are testing it from Tibco Business Works. We configured the JMS
> Connection
> using JNDI.
>
> With auto acknowledged we did not see any issues. Everything worked fine.
>
> We are testing queues. Topics are also in scope for the testing, have not
> started yet.
>
> I did not configure prefetch, using the default which as per documentation
> is 1000.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Performance-with-CLIENT-Ack-tp4728551p4728566.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Active MQ - Performance with CLIENT Ack

[akpuvvada <ak...@gmail.com>]

We are testing it from Tibco Business Works. We configured the JMS Connection
using JNDI.

With auto acknowledged we did not see any issues. Everything worked fine.

We are testing queues. Topics are also in scope for the testing, have not
started yet.

I did not configure prefetch, using the default which as per documentation
is 1000.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Active-MQ-Performance-with-CLIENT-Ack-tp4728551p4728566.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ - Performance with CLIENT Ack

[Tim Bain <tb...@alumni.duke.edu>]

1MB payloads are pretty big. I'm not sure that is affecting the behavior
you're seeing, but it stands out as unusual.

Since you say the messages are not being acked, can you show us the code
that attempts to send those acks? And are we talking about queues or
topics?

Are the performance characteristics normal/expected if you use auto-ack?

What prefetch buffer size are you using for your clients?

Tim

On Jul 14, 2017 5:18 AM, "akpuvvada" <ak...@gmail.com> wrote:

> I am trying to test Active MQ for Performance
> I have installed it in Unix in 'Shared File System' Fault Tolerant pair.
> Max Heap Size is 3 GB
> 4GB Persistence Store Size
> 2Gb Temporary Store Size
>
> I am trying to publish messages with 'Non_Persistent' Delivery mode with
> Payload of size 1 MB without any delay in a loop - 10K messages.
> I am trying to consume the messages with Clinet ACK and acknowledging after
> logging the receipt of message.
>
> Prouder has published 3k+ messages and slowed down as Temp Store size is
> only 2GB.
>
> However, Subscriber is consuming the messages at a very slow rate and not
> acknowledging or Active MQ is not processing the acknowledgements.
>
> Please advise what I am doing wrong or what should be changed to improve
> the
> performance?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Performance-with-CLIENT-Ack-tp4728551.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>