You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by dumlebert <kp...@kth.se> on 2013/10/04 16:58:42 UTC
Deadlock when using XA and PFC
Hi,
I have an issue that I've battled with for some time now and I'm running out
of ideas. We're using ActiveMQ from MuleESB. The flow in this case reads a
message from one queue, splits it into several messages and puts them on
another queue. All this is done within an XA transaction. The messages are
up to a few MB each.
When the producer (our flow) has put say 3/5 messages on the queue, it hangs
when trying to publish the 4th message. This gets into a deadlock, as the
consumer can't consume the messages until the producer commits, and the
producer is not allowed to publish the messages it needs to commit the XA
transaction.
It's logical to think that PFC together with XA could create this kind of
situation, but I'm unable to verify that it actually is PFC that is stopping
the producer from publishing all messages. I've tried to:
- disable PFC (verified by viewing the attributes through jmx)
- set the sendFailIfNoSpace = "true"
- increase the memoryUsage, storeUsage and tempUsage to high values
Nothing changes the behaviour.
If I cut the size of the messages to be produced by limiting the amount of
data they include, all messages a published and the XA commits just fine. So
the hangup does seem to be related to size.
But then, why is the problem the same when disabling PFC. I must be missing
something here...
I want to be dead certain on exactly what is the problem that cause the send
to queue to hang. I'm really thankful for any ideas on how I can figure out
what is going on.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Deadlock-when-using-XA-and-PFC-tp4672276.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Deadlock when using XA and PFC
Posted by dumlebert <kp...@kth.se>.
Thanks for the response.
I haven't yet checked the logs, but I have found the cause of the problem.
When increasing the overall heap of the vm, the producer is allowed to
publish all messages. I would have expected to get an OutOfMemoryException
if the producer tries to write to the queue when the broker is unable to
allocate memory. But perhaps it is a feature of PFC that it avoids such
situation by checking available memory before writing to the queue, and
hangs instead. That does make some sense I guess.
I will check the logs as you suggested and see if I can spot anything in
there.
Best,
Pontus
--
View this message in context: http://activemq.2283324.n4.nabble.com/Deadlock-when-using-XA-and-PFC-tp4672276p4672373.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Deadlock when using XA and PFC
Posted by Gary Tully <ga...@gmail.com>.
a thread dump of the broker will show you exactly where a thread is
blocked. But if pfc is the limit, there will be an indication in the
log - slf4j
On 4 October 2013 15:58, dumlebert <kp...@kth.se> wrote:
> Hi,
> I have an issue that I've battled with for some time now and I'm running out
> of ideas. We're using ActiveMQ from MuleESB. The flow in this case reads a
> message from one queue, splits it into several messages and puts them on
> another queue. All this is done within an XA transaction. The messages are
> up to a few MB each.
>
> When the producer (our flow) has put say 3/5 messages on the queue, it hangs
> when trying to publish the 4th message. This gets into a deadlock, as the
> consumer can't consume the messages until the producer commits, and the
> producer is not allowed to publish the messages it needs to commit the XA
> transaction.
>
> It's logical to think that PFC together with XA could create this kind of
> situation, but I'm unable to verify that it actually is PFC that is stopping
> the producer from publishing all messages. I've tried to:
> - disable PFC (verified by viewing the attributes through jmx)
> - set the sendFailIfNoSpace = "true"
> - increase the memoryUsage, storeUsage and tempUsage to high values
>
> Nothing changes the behaviour.
>
> If I cut the size of the messages to be produced by limiting the amount of
> data they include, all messages a published and the XA commits just fine. So
> the hangup does seem to be related to size.
>
> But then, why is the problem the same when disabling PFC. I must be missing
> something here...
>
> I want to be dead certain on exactly what is the problem that cause the send
> to queue to hang. I'm really thankful for any ideas on how I can figure out
> what is going on.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Deadlock-when-using-XA-and-PFC-tp4672276.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://redhat.com
http://blog.garytully.com