You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@activemq.apache.org by "yang.yang.zz" <ya...@outlook.com> on 2016/03/06 15:50:21 UTC

ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Hi：

We ran into this dead lock since upgrade to 5.11.1. In short, this dead lock
happens on network connector when "duplex" is set to "true". For details of
replicating this defect, see my earlier post  here
<http://activemq.2283324.n4.nabble.com/5-13-1-message-blocked-td4708592.html>  

I also found an unresolved defect  AMQ-5260
<https://issues.apache.org/jira/browse/AMQ-5260>   which has the same
symptom. It looks like this issue could also be found in 5.9.x and 5.10.x.

I'm wondering, is there a way to workaround this dead lock without modifying
the source code? For example, is there an alternative 2-way communication
configuration instead of using duplex? 

Thanks,
-Yang





--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Posted by "yang.yang.zz" <ya...@outlook.com>.

I'm still investigating the cause of the dead lock. Here's a piece of thread
dump I found from the producer broker. * Note producer broker configures the
NetworkConnector..


"ActiveMQ Transport: tcp://lod-uimlongda/10.130.156.161:61616@45650" #139
prio=5 os_prio=0 tid=0x00007fddc0007800 nid=0x1a1a waiting on condition
[0x00007fdd91ede000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000cf606228> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	*at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)*
	at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1432)
	at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:946)
	at
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:902)
	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:190)
	at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:124)
	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:162)
	at
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:154)
	- locked <0x00000000cf6064d8> (a java.util.concurrent.atomic.AtomicBoolean)
	at
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:131)
	*at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)*
	at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
	at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
	at
org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:922)
	at
org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1197)
	at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:776)
	- locked <0x00000000d5428cd8> (a java.net.URI)
	at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:623)
	at
org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:225)
	at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:124)
	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
	at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
	at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
	at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
	at java.lang.Thread.run(Thread.java:745)

I see the MutexTransport.oneway is called twice this is stacktrace, and it
finally got locked. Is this a self-lock? I checked other threads in the
thread dump, but didn't see other thread could potentially hold the 
writeLock...


    @Override
    public void oneway(Object command) throws IOException {
        *writeLock.lock();*
        try {
            next.oneway(command);
        } finally {
            writeLock.unlock();
        }
    }

Here's the full thread dump.
2016.jstack
<http://activemq.2283324.n4.nabble.com/file/n4709271/2016.jstack>  



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952p4709271.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Posted by "yang.yang.zz" <ya...@outlook.com>.

Hi Art and Tim

Thanks very much for your replies. It's a good point you mentioned this
design could be an anti-pattern.

Just to give more background for our use case. The main purpose of our
product is to collect data from LAN or WAN network devices. And the
collectors (producers) will push collected data to the remote consumer.
However, due to the complexity/variety of the environment, the connection
between producer and consumer can be temporarily unavailable, or the
consumer may not be able to consume the data fast enough, or the consumer is
just simply down. Then it requires each collector to be able to keep
collecting data and store the collected data individually. (this is why we
use local broker on the producer side and broker tempStore). Then once the
consumer is backup on line, the producer broker will send the data stored in
tempStore to the consumer.

Yes, we could consider using one broker at the consumer side. Then we may
face the following challenges:

1) Our current scale is the producers will generate 21MB~42MB data per
second in total. If the we use one broker only at the consumer side. Then if
the consumer restarts or goes down during the weekend, the consumer broker
will need 3.6TB~7.2TB disk space to store the collected data. Which is too
much requirement as a single node. However, if we distributed these space
cross 20 consumers, then it's 180GB~360GB, which is more acceptable to our
customers.

2) Besides, what if the consumer-producer network is down, for example due
to some network jam or firewall block, then if the producer doesn't have a
local broker, then it has nowhere to store the data or it needs extra
implementation to store the data. The continuity of collected data is
critical for our business. Then a local broker will help us to address this
issue.

So based on this info, welcome any advice or suggestions!

Regards,
-Yang

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952p4709270.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Posted by Tim Bain <tb...@alumni.duke.edu>.

As Art said, the typical configuration is to have one standalone broker
that all clients (producers and consumers) connect to, that is available
even when the clients are not.  This won't work if the clients frequently
are up but don't have network connectivity, or if no one is willing to host
and manage a central broker that's not colocated with the clients, but for
most use cases, this is the preferred configuration.  Is there a reason why
this isn't the configuration you're using?
On Mar 12, 2016 10:53 PM, "artnaseef" <ar...@artnaseef.com> wrote:

> When you say TempStore, are you talking about storing non-persistent
> messages
> specifically?
>
> To answer your question - YES, the hub broker, and all brokers in the
> network can hold messages during short periods of disconnects between
> clients.  That's kinda the main point of JMS and messaging middleware, of
> which AMQ is one.
>
> In fact, there's nothing special to do to get that functionality --
> standing
> up a single broker and having many producers and consumers use the same
> broker is feasible.  If there's a need for high availability (i.e.
> immediate
> recovery from any single-point-of-failure), or a need to avoid message loss
> across a message broker restart, ActiveMQ has features to handle that as
> well.
>
> There's actually no need to put brokers on the producer and consumer boxes.
> In fact, except in rare cases, I consider doing so an anti-pattern.
>
> So, here's the question in my mind: will a single broker serving multiple
> producers and consumers meet the needs?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952p4709238.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Posted by artnaseef <ar...@artnaseef.com>.

When you say TempStore, are you talking about storing non-persistent messages
specifically?

To answer your question - YES, the hub broker, and all brokers in the
network can hold messages during short periods of disconnects between
clients.  That's kinda the main point of JMS and messaging middleware, of
which AMQ is one.

In fact, there's nothing special to do to get that functionality -- standing
up a single broker and having many producers and consumers use the same
broker is feasible.  If there's a need for high availability (i.e. immediate
recovery from any single-point-of-failure), or a need to avoid message loss
across a message broker restart, ActiveMQ has features to handle that as
well.

There's actually no need to put brokers on the producer and consumer boxes. 
In fact, except in rare cases, I consider doing so an anti-pattern.

So, here's the question in my mind: will a single broker serving multiple
producers and consumers meet the needs?



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952p4709238.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Posted by "yang.yang.zz" <ya...@outlook.com>.

Hi Tim

The main reason we put a broker on the producer is we want to leverage the
broker's TempStore feature. Which is if the Consumer is offline, all the
producer's broker can temporarily hold the produced data, and once the
consumer is back online, it will catch up.

This is the design since we used ActiveMQ 5.5.1. If the client can do the
TempStore same as the broker, then we may consider to make a change.

So then, is there a way to make the client to do this?

-Yang

You have to do what you described, and attempt to connect to all the client
brokers from the hub, which means you have to list them all out up front
and then live with the log messages when the hub can't connect to a spoke
broker because the spoke is offline. That's the more-complicated
configuration I was referencing.

With that being said, I haven't heard you describe why you're using the
client brokers rather than having each producer connect directly to the hub
broker, which would eliminate the whole problem. Do you actually need your
producer (spoke) brokers?

Tim
Thanks for the response Tim!

I would not mind to go with more complicated config as long as it works.
Just my experience on AMQ config is very basic. I've seen some examples of
using two non-duplex network connectors for 1-to-1 brokers. But I didn't
find out similar examples for 1-to-many brokers. (It's 1 consumer broker and
many producer brokers in our case)

For producers, it is easy. They just need to create a network connector with
the consumer broker hostname. But how to config the consumer? Do I have to
hard-code list of consumer hostnames into the consumer broker config? Since
in our case, the producers can come and go, so it would be very unideal to
update consumers' broker config when there's a change on the producers. Is
there a better way to do it? If so can you give a simple example?

Thanks and regards,
-Yang

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952p4708967.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Posted by Tim Bain <tb...@alumni.duke.edu>.

Tim
Thanks for the response Tim!

Thanks and regards,
-Yang

--
View this message in context:
http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952p4708956.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Posted by "yang.yang.zz" <ya...@outlook.com>.

Thanks for the response Tim!

I would not mind to go with more complicated config as long as it works.
Just my experience on AMQ config is very basic. I've seen some examples of
using two non-duplex network connectors for 1-to-1 brokers. But I didn't
find out similar examples for 1-to-many brokers. (It's 1 consumer broker and
many producer brokers in our case)

For producers, it is easy. They just need to create a network connector with
the consumer broker hostname. But how to config the consumer? Do I have to
hard-code list of consumer hostnames into the consumer broker config? Since
in our case, the producers can come and go, so it would be very unideal to
update consumers' broker config when there's a change on the producers. Is
there a better way to do it? If so can you give a simple example? 

Thanks and regards,
-Yang



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952p4708956.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

Posted by Tim Bain <tb...@alumni.duke.edu>.

You can use two non-duplex connections, one from each broker.  This is
advantageous in some network configurations (e.g a pair of brokers) because
it lets you reuse the same configuration on multiple brokers (with a little
placeholder substitution) and disadvantageous in others (e.g. a
hub-and-spoke topology) because it forces both sides to be aware of the
connection between them.

But even if it complicates your config slightly, it'll still be better than
what you're hitting now.

Tim
On Mar 6, 2016 6:44 PM, "yang.yang.zz" <ya...@outlook.com> wrote:

> Hi：
>
> We ran into this dead lock since upgrade to 5.11.1. In short, this dead
> lock
> happens on network connector when "duplex" is set to "true". For details of
> replicating this defect, see my earlier post  here
> <
> http://activemq.2283324.n4.nabble.com/5-13-1-message-blocked-td4708592.html
> >
>
> I also found an unresolved defect  AMQ-5260
> <https://issues.apache.org/jira/browse/AMQ-5260>   which has the same
> symptom. It looks like this issue could also be found in 5.9.x and 5.10.x.
>
> I'm wondering, is there a way to workaround this dead lock without
> modifying
> the source code? For example, is there an alternative 2-way communication
> configuration instead of using duplex?
>
> Thanks,
> -Yang
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708952.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>