You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Benoit Fortin <be...@gmail.com> on 2019/09/03 17:03:54 UTC
JMS pub/sub (using Camel) performance issue with MQ in XA transactions
We have a java application doing JMS subscriptions that is using Camel as
its JMS provider.
The application is subscribing to a topic using XA. It consumes 1 message
in the queue, and then closes the XA transaction (each message is part of
an XA transaction). Then the application re-attaches itself to the topic to
start over the same process for each message.
When there are no messages to process, the client waits for 45 seconds
(which is the XA transaction timeout) before closing the request and the XA
transaction and starting a new iteration.
I have analyzed how this process is actually being done using a tcpdump,
and here is what I found:
——
MQ Client --[XA_START]--> MQ Server
MQ Client <--[XA_START_REPLY]-- MQ Server
MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
MQ Client <--[SPI REPLY]-- MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client <--[NOTIFICATION]-- MQ Server
MQ Client --[MQCLOSE]--> MQ Server
MQ Client <--[MQCLOSE_REPLY]-- MQ Server
MQ Client --[XA_END]--> MQ Server
MQ Client <--[XA_END_REPLY]-- MQ Server
MQ Client --[XA_COMMIT]--> MQ Server
MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
——
This is process is on a durable subscription, so each of the iterations
(even when there is no message to consume) ends up generating I/O on the MQ
server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where
MQ keeps trace of its subscribers).
Having a somewhat high number of subscribers doing this process, the MQ
server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
outside of business hours, when there are no messages to consume). This
process also consumes ~3 CPU outside of business hours.
I am a bit puzzled that using XA on pub/sub scales so bad, and I am
wondering if there is any way to implement this solution without doing so
much subscribe operations.
Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
Posted by Benoit Fortin <be...@gmail.com>.
Hi,
We do need XA as the client application is doing 2 phase commit in (1) the
subscription and (2) a database.
On Tue, Sep 3, 2019 at 14:49 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi,
>
> What about using pure ACK mode ?
>
> You can see an example with queue (same can be applied to topic) here:
> http://blog.nanthrax.net/?p=820
>
> My point is: do you need XA as AFAIU you are using only JMS as resource
> (XA is required when you want to use the same transaction with different
> backends like JMS and database for instance) ?
>
> Regards
> JB
>
> On 03/09/2019 19:03, Benoit Fortin wrote:
> > We have a java application doing JMS subscriptions that is using Camel as
> > its JMS provider.
> >
> > The application is subscribing to a topic using XA. It consumes 1 message
> > in the queue, and then closes the XA transaction (each message is part of
> > an XA transaction). Then the application re-attaches itself to the topic
> to
> > start over the same process for each message.
> >
> > When there are no messages to process, the client waits for 45 seconds
> > (which is the XA transaction timeout) before closing the request and the
> XA
> > transaction and starting a new iteration.
> >
> > I have analyzed how this process is actually being done using a tcpdump,
> > and here is what I found:
> >
> > ——
> >
> > MQ Client --[XA_START]--> MQ Server
> >
> > MQ Client <--[XA_START_REPLY]-- MQ Server
> >
> >
> > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
> >
> > MQ Client <--[SPI REPLY]-- MQ Server
> >
> >
> > MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> > MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> > MQ Client <--[NOTIFICATION]-- MQ Server
> >
> >
> > MQ Client --[MQCLOSE]--> MQ Server
> >
> > MQ Client <--[MQCLOSE_REPLY]-- MQ Server
> >
> >
> > MQ Client --[XA_END]--> MQ Server
> >
> > MQ Client <--[XA_END_REPLY]-- MQ Server
> >
> >
> > MQ Client --[XA_COMMIT]--> MQ Server
> >
> > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
> >
> > ——
> >
> > This is process is on a durable subscription, so each of the iterations
> > (even when there is no message to consume) ends up generating I/O on the
> MQ
> > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue,
> where
> > MQ keeps trace of its subscribers).
> >
> > Having a somewhat high number of subscribers doing this process, the MQ
> > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> > outside of business hours, when there are no messages to consume). This
> > process also consumes ~3 CPU outside of business hours.
> >
> > I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> > wondering if there is any way to implement this solution without doing so
> > much subscribe operations.
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: JMS pub/sub (using Camel) performance issue with MQ in XA
transactions
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
What about using pure ACK mode ?
You can see an example with queue (same can be applied to topic) here:
http://blog.nanthrax.net/?p=820
My point is: do you need XA as AFAIU you are using only JMS as resource
(XA is required when you want to use the same transaction with different
backends like JMS and database for instance) ?
Regards
JB
On 03/09/2019 19:03, Benoit Fortin wrote:
> We have a java application doing JMS subscriptions that is using Camel as
> its JMS provider.
>
> The application is subscribing to a topic using XA. It consumes 1 message
> in the queue, and then closes the XA transaction (each message is part of
> an XA transaction). Then the application re-attaches itself to the topic to
> start over the same process for each message.
>
> When there are no messages to process, the client waits for 45 seconds
> (which is the XA transaction timeout) before closing the request and the XA
> transaction and starting a new iteration.
>
> I have analyzed how this process is actually being done using a tcpdump,
> and here is what I found:
>
> ——
>
> MQ Client --[XA_START]--> MQ Server
>
> MQ Client <--[XA_START_REPLY]-- MQ Server
>
>
> MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
>
> MQ Client <--[SPI REPLY]-- MQ Server
>
>
> MQ Client --[REQUEST_MSGS]--> MQ Server
>
> MQ Client --[REQUEST_MSGS]--> MQ Server
>
> MQ Client <--[NOTIFICATION]-- MQ Server
>
>
> MQ Client --[MQCLOSE]--> MQ Server
>
> MQ Client <--[MQCLOSE_REPLY]-- MQ Server
>
>
> MQ Client --[XA_END]--> MQ Server
>
> MQ Client <--[XA_END_REPLY]-- MQ Server
>
>
> MQ Client --[XA_COMMIT]--> MQ Server
>
> MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
>
> ——
>
> This is process is on a durable subscription, so each of the iterations
> (even when there is no message to consume) ends up generating I/O on the MQ
> server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where
> MQ keeps trace of its subscribers).
>
> Having a somewhat high number of subscribers doing this process, the MQ
> server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> outside of business hours, when there are no messages to consume). This
> process also consumes ~3 CPU outside of business hours.
>
> I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> wondering if there is any way to implement this solution without doing so
> much subscribe operations.
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
Posted by Benoit Fortin <be...@gmail.com>.
The cache shouldn’t be enabled, as we are using XA (as per Camel’s
recommendation). We have also tuned the polling interval.
That being said, what hurts the I/O subsystem on the MQ server is the
Subscribe operation itself (not the polling for messages).
I am wondering is if there would be any way (inside or outside of Camel) to
have the XA transaction “scoped” on the REQUEST_MSGS requests instead of
the SUBSCRIBE operation? So that each polling doesn’t do a subscribe
operation every time and instead stays attached to an already opened
subscription?
As a reminder, I am attaching below how the XA transaction and subscription
operations are being done (according to a tcpdump I analyzed):
MQ Client --[XA_START]--> MQ Server
MQ Client <--[XA_START_REPLY]-- MQ Server
MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
MQ Client <--[SPI REPLY]-- MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client <--[NOTIFICATION]-- MQ Server
MQ Client --[MQCLOSE]--> MQ Server
MQ Client <--[MQCLOSE_REPLY]-- MQ Server
MQ Client --[XA_END]--> MQ Server
MQ Client <--[XA_END_REPLY]-- MQ Server
MQ Client --[XA_COMMIT]--> MQ Server
MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
On Wed, Sep 4, 2019 at 05:46 Claus Ibsen <cl...@gmail.com> wrote:
> camel-jms uses spring jms, so there are many users with this combo
> also with IBM MQ and XA.
> Look at the many options spring jms has to tune its polling, and also
> cache levels you can tweak.
> There may be some more idle options you can set to make it "sleep"
> longer when there are no messages.
> But you can't avoid that spring jms is designed to keep polling for
> new messages also when the queue is empty.
>
>
>
> On Tue, Sep 3, 2019 at 7:04 PM Benoit Fortin <be...@gmail.com>
> wrote:
> >
> > We have a java application doing JMS subscriptions that is using Camel as
> > its JMS provider.
> >
> > The application is subscribing to a topic using XA. It consumes 1 message
> > in the queue, and then closes the XA transaction (each message is part of
> > an XA transaction). Then the application re-attaches itself to the topic
> to
> > start over the same process for each message.
> >
> > When there are no messages to process, the client waits for 45 seconds
> > (which is the XA transaction timeout) before closing the request and the
> XA
> > transaction and starting a new iteration.
> >
> > I have analyzed how this process is actually being done using a tcpdump,
> > and here is what I found:
> >
> > ——
> >
> > MQ Client --[XA_START]--> MQ Server
> >
> > MQ Client <--[XA_START_REPLY]-- MQ Server
> >
> >
> > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
> >
> > MQ Client <--[SPI REPLY]-- MQ Server
> >
> >
> > MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> > MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> > MQ Client <--[NOTIFICATION]-- MQ Server
> >
> >
> > MQ Client --[MQCLOSE]--> MQ Server
> >
> > MQ Client <--[MQCLOSE_REPLY]-- MQ Server
> >
> >
> > MQ Client --[XA_END]--> MQ Server
> >
> > MQ Client <--[XA_END_REPLY]-- MQ Server
> >
> >
> > MQ Client --[XA_COMMIT]--> MQ Server
> >
> > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
> >
> > ——
> >
> > This is process is on a durable subscription, so each of the iterations
> > (even when there is no message to consume) ends up generating I/O on the
> MQ
> > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue,
> where
> > MQ keeps trace of its subscribers).
> >
> > Having a somewhat high number of subscribers doing this process, the MQ
> > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> > outside of business hours, when there are no messages to consume). This
> > process also consumes ~3 CPU outside of business hours.
> >
> > I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> > wondering if there is any way to implement this solution without doing so
> > much subscribe operations.
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
Posted by Claus Ibsen <cl...@gmail.com>.
camel-jms uses spring jms, so there are many users with this combo
also with IBM MQ and XA.
Look at the many options spring jms has to tune its polling, and also
cache levels you can tweak.
There may be some more idle options you can set to make it "sleep"
longer when there are no messages.
But you can't avoid that spring jms is designed to keep polling for
new messages also when the queue is empty.
On Tue, Sep 3, 2019 at 7:04 PM Benoit Fortin <be...@gmail.com> wrote:
>
> We have a java application doing JMS subscriptions that is using Camel as
> its JMS provider.
>
> The application is subscribing to a topic using XA. It consumes 1 message
> in the queue, and then closes the XA transaction (each message is part of
> an XA transaction). Then the application re-attaches itself to the topic to
> start over the same process for each message.
>
> When there are no messages to process, the client waits for 45 seconds
> (which is the XA transaction timeout) before closing the request and the XA
> transaction and starting a new iteration.
>
> I have analyzed how this process is actually being done using a tcpdump,
> and here is what I found:
>
> ——
>
> MQ Client --[XA_START]--> MQ Server
>
> MQ Client <--[XA_START_REPLY]-- MQ Server
>
>
> MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
>
> MQ Client <--[SPI REPLY]-- MQ Server
>
>
> MQ Client --[REQUEST_MSGS]--> MQ Server
>
> MQ Client --[REQUEST_MSGS]--> MQ Server
>
> MQ Client <--[NOTIFICATION]-- MQ Server
>
>
> MQ Client --[MQCLOSE]--> MQ Server
>
> MQ Client <--[MQCLOSE_REPLY]-- MQ Server
>
>
> MQ Client --[XA_END]--> MQ Server
>
> MQ Client <--[XA_END_REPLY]-- MQ Server
>
>
> MQ Client --[XA_COMMIT]--> MQ Server
>
> MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
>
> ——
>
> This is process is on a durable subscription, so each of the iterations
> (even when there is no message to consume) ends up generating I/O on the MQ
> server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where
> MQ keeps trace of its subscribers).
>
> Having a somewhat high number of subscribers doing this process, the MQ
> server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> outside of business hours, when there are no messages to consume). This
> process also consumes ~3 CPU outside of business hours.
>
> I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> wondering if there is any way to implement this solution without doing so
> much subscribe operations.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2