You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Pauli Kaila <pa...@napa.fi> on 2010/11/26 14:04:36 UTC
Network of brokers: message forwarding between brokers stops
Hi,
We have a setup where we have a server with it's own internal broker and
then multiple clients that also have their own internal brokers which are
connecting to the server broker.
The server is consuming messages from a queue "server.in" via a vm
-connection to the internal broker. The client is sending messages to the
"server.in" queue via its own vm -connection to its internal broker. The
messages are then forwarded by the client broker to the server broker as
expected. Until they are not.
When testing with just one client (the same thing happens with multiple
clients) I constantly end up in a situation where the client broker stops
forwarding messages to the server broker. The queue just starts growing
inside the client vm (as seen by using JMX) while the queue in the server vm
stays empty. I have enabled the lowest level of logging for ActiveMQ but
have not been able to notice any log messages related to this situation. It
only takes couple of hundred messages to reproduce this situation. Both the
server and the client are running on the same machine.
The only thing that has made this problem disappear is removal of the client
broker and having only one broker. But because our clients and server might
be on different sides of a firewall, we would like to use the network of
brokers feature with the duplex connection to be able to successfully
communicate through a firewall.
I have tried to create a test case to reproduce this situation, but for now
I've been unable to do that. I am not sure whether this is a bug in ActiveMQ
or if there is something funky in the way we use it. I've played around with
different kind of configurations but nothing seems to help.
Here is our configuration:
Server broker configuration:
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="Server" useJmx="true" persistent="true">
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<transportConnectors>
<transportConnector uri="tcp://0.0.0.0:27182" />
</transportConnectors>
</broker>
Client broker configuration:
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="Client" useJmx="true" persistent="true">
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<networkConnectors>
<networkConnector uri="static:(tcp://${server_address}:27182)"
duplex="true" conduitSubscriptions="true"/>
</networkConnectors>
</broker>
We have one queue for the server and one uniquely named queue per each
client. The server also broadcasts changes to data using a topic. In
addition to these we are using QueueRequestors, which internally uses
temporary queues, to do synchronous requests from the clients to the server.
The client also uses message selectors in its queue to direct messages to
different listeners.
I hope someone reading this list could point me to a solution for our
problem.
Best regards,
-Pauli Kaila
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-message-forwarding-between-brokers-stops-tp3060260p3060260.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers: message forwarding between brokers stops
Posted by Pauli Kaila <pa...@napa.fi>.
Hi,
I have been busy with other stuff so I have not been able to focus on this
problem until now.
For now I have not been able to create a test case that could reproduce this
problem. However I have continued my investigations. I once again made
ActiveMQ log everything and found something that could perhaps tell a bit
more about the issue at hand.
I was looking at log entries created during successful communication and how
things change when the communication stops.
During successful communication I can see that the VMTransport thread in the
Server broker logs a MessageAck in PrefetchSubscription.acknowledge and
after that the ActiveMQ Transport thread in the Client broker logs a
MessageAck with the same MessageId (again in
PrefetchSubscription.acknowledge).
However when the communication stalls I can see the Server broker still
logging acknowledgements to some messages but the Client broker does not do
that and at this point no more messages reach the Server broker from the
Client broker (at least for that queue).
The strange thing is, that I was able to make the communication going again
by temporarily cutting the network connections between the Client and Server
(I did this by disabling the Client machines network interface).
I'll continue my investigation of this subject but at the same time hope
that you could be able to perhaps come up with something based on my
findings.
Br,
-Pauli Kaila
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-message-forwarding-between-brokers-stops-tp3060260p3090850.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers: message forwarding between brokers stops
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi Pauli,
any chance you can create a test case that demonstrates this?
Cheers
--
Dejan Bosanac
-----------------
FuseSource - The experts in open source integration and messaging.
Email: dejanb@fusesource.com
Web: http://fusesource.com
Twitter: http://twitter.com/dejanb
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Mon, Nov 29, 2010 at 9:28 AM, Pauli Kaila <pa...@napa.fi> wrote:
>
> Hi,
>
> Thanks for your answer. I had already tried adding
> <dynamicallyIncludedDestination> and now tried the
> <staticallyIncludedDestination> but neither of them help.
>
> I just noticed that when this problem occurs the
> "MessageCountAwaitingAcknowledge" of the queue subscription for the
> Server.in queue named NC_Server_inbound_Client in the client gets high and
> stays there. I assume that this subscription belongs to the Network
> Connector and is the one forwarding the messages to the Server. So it seems
> that the Server broker is not acknowledging those messages.
>
> I tried modifying our code to see whether the problem is in the message
> acknowledgement there. We were using auto acknowledge before, but I also
> tried with client acknowledge. However this did not change the situation in
> any way.
>
> Br,
> -Pauli
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-message-forwarding-between-brokers-stops-tp3060260p3063198.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Network of brokers: message forwarding between brokers stops
Posted by Brett Wooldridge <br...@gmail.com>.
Pauli,
Just so you know, I'm another user very interested in the outcome of this
because we just started building a solution based on a network of brokers.
I appreciate your debugging efforts.
Brett
On Mon, Nov 29, 2010 at 5:28 PM, Pauli Kaila <pa...@napa.fi> wrote:
>
> Hi,
>
> Thanks for your answer. I had already tried adding
> <dynamicallyIncludedDestination> and now tried the
> <staticallyIncludedDestination> but neither of them help.
>
> I just noticed that when this problem occurs the
> "MessageCountAwaitingAcknowledge" of the queue subscription for the
> Server.in queue named NC_Server_inbound_Client in the client gets high and
> stays there. I assume that this subscription belongs to the Network
> Connector and is the one forwarding the messages to the Server. So it seems
> that the Server broker is not acknowledging those messages.
>
> I tried modifying our code to see whether the problem is in the message
> acknowledgement there. We were using auto acknowledge before, but I also
> tried with client acknowledge. However this did not change the situation in
> any way.
>
> Br,
> -Pauli
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Network-of-brokers-message-forwarding-between-brokers-stops-tp3060260p3063198.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Network of brokers: message forwarding between brokers stops
Posted by Pauli Kaila <pa...@napa.fi>.
Hi,
Thanks for your answer. I had already tried adding
<dynamicallyIncludedDestination> and now tried the
<staticallyIncludedDestination> but neither of them help.
I just noticed that when this problem occurs the
"MessageCountAwaitingAcknowledge" of the queue subscription for the
Server.in queue named NC_Server_inbound_Client in the client gets high and
stays there. I assume that this subscription belongs to the Network
Connector and is the one forwarding the messages to the Server. So it seems
that the Server broker is not acknowledging those messages.
I tried modifying our code to see whether the problem is in the message
acknowledgement there. We were using auto acknowledge before, but I also
tried with client acknowledge. However this did not change the situation in
any way.
Br,
-Pauli
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-message-forwarding-between-brokers-stops-tp3060260p3063198.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers: message forwarding between brokers stops
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi,
you can try adding <staticallyIncludedDestination> in you client
connector which should ensure that messages on that destination is
always passed over the network.
Take a look at http://activemq.apache.org/networks-of-brokers.html to
see how to achieve this.
Cheers
--
Dejan Bosanac
-----------------
FuseSource - The experts in open source integration and messaging.
Email: dejanb@fusesource.com
Web: http://fusesource.com
Twitter: http://twitter.com/dejanb
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Fri, Nov 26, 2010 at 2:04 PM, Pauli Kaila <pa...@napa.fi> wrote:
>
> Hi,
>
> We have a setup where we have a server with it's own internal broker and
> then multiple clients that also have their own internal brokers which are
> connecting to the server broker.
>
> The server is consuming messages from a queue "server.in" via a vm
> -connection to the internal broker. The client is sending messages to the
> "server.in" queue via its own vm -connection to its internal broker. The
> messages are then forwarded by the client broker to the server broker as
> expected. Until they are not.
>
> When testing with just one client (the same thing happens with multiple
> clients) I constantly end up in a situation where the client broker stops
> forwarding messages to the server broker. The queue just starts growing
> inside the client vm (as seen by using JMX) while the queue in the server vm
> stays empty. I have enabled the lowest level of logging for ActiveMQ but
> have not been able to notice any log messages related to this situation. It
> only takes couple of hundred messages to reproduce this situation. Both the
> server and the client are running on the same machine.
>
> The only thing that has made this problem disappear is removal of the client
> broker and having only one broker. But because our clients and server might
> be on different sides of a firewall, we would like to use the network of
> brokers feature with the duplex connection to be able to successfully
> communicate through a firewall.
>
> I have tried to create a test case to reproduce this situation, but for now
> I've been unable to do that. I am not sure whether this is a bug in ActiveMQ
> or if there is something funky in the way we use it. I've played around with
> different kind of configurations but nothing seems to help.
>
> Here is our configuration:
>
> Server broker configuration:
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="Server" useJmx="true" persistent="true">
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
> <transportConnectors>
> <transportConnector uri="tcp://0.0.0.0:27182" />
> </transportConnectors>
> </broker>
>
> Client broker configuration:
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="Client" useJmx="true" persistent="true">
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
> <networkConnectors>
> <networkConnector uri="static:(tcp://${server_address}:27182)"
> duplex="true" conduitSubscriptions="true"/>
> </networkConnectors>
> </broker>
>
> We have one queue for the server and one uniquely named queue per each
> client. The server also broadcasts changes to data using a topic. In
> addition to these we are using QueueRequestors, which internally uses
> temporary queues, to do synchronous requests from the clients to the server.
> The client also uses message selectors in its queue to direct messages to
> different listeners.
>
> I hope someone reading this list could point me to a solution for our
> problem.
>
> Best regards,
> -Pauli Kaila
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-message-forwarding-between-brokers-stops-tp3060260p3060260.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>