You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by rockies <la...@bd.com> on 2017/09/14 23:46:14 UTC
Stuck messages
Hi,
We have setup network of brokers with just two ActiveMQ nodes in version
5.15.0
Config is as follows:
<networkConnectors>
<networkConnector conduitSubscriptions="false"
decreaseNetworkConsumerPriority="true" duplex="true" dynamicOnly="true"
name="broker1Tobroker02" networkTTL="3" userName="testadmin"
password="password" uri="static:tcp://xxxx.test.com:61616">
</networkConnector>
</networkConnectors>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
rebalanceClusterClients="true" updateClusterClients="true"
updateClusterClientsOnRemove="true"
uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" rebalanceClusterClients="true"
updateClusterClients="true" updateClusterClientsOnRemove="true"
uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp"
uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt"
uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws"
uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
Data is persisted in SQL Server. We use Mule ESB cluster, where
subscribers from both nodes listen to Queues in ActiveMQ with a failover URI
such as :
failover://(tcp://broker1.test.com:61616,tcp://broker2.test.com:61616)?randomize=false&priorityBackup=true&initialReconnectDelay=100
Since this updgrade from 5.11.1 to 5.14.5 to 5.15.0 we are seem to have
more stuck messages in our queues, which usually ends up in re-starting JMS
brokers and ESB servers. There are atleast close to 200 queues. Message size
rarely gets to 25MB and number of messages per second could be close to 50.
So why is it that we seem to have more issues with stuck messages ? Can you
please let us know what whould be the optimal configuration for this kind of
clustered environment?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stuck messages
Posted by rockies <la...@bd.com>.
Thank you Tim, I will make these changes appreciate it ! Yes only some of our
producers are setting persistent property. I will also make that required
for all our producer apps to make sure we don't lose messages from our
queues on a broker re-start. Thank you !
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stuck messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
1) I've always found far more value in the information exposed via JMX than
in the content shown on the web console. The web console is great for basic
system monitoring, but anytime you want to understand what's actually going
on inside the broker, I've always found that JMX was far more useful. Since
you're in the latter category, you can probably guess what my first
suggestion is.
2) I don't remember what seeing consumers without a session indicates. I've
noticed (on versions in the 5.8 to 5.10 range, which are the only versions
I've ever run) that the JMX information always seemed to include a number
of consumers that had previously disconnected, though I've never tried to
figure out exactly why they weren't removed as soon as they were
disconnected. But it's possible that that's all they are.
3) Either way, it's not going to hurt anything to use
replayWhenNoConsumers, so there's no reason for you not to use it. In the
worst case, it doesn't prevent messages from getting stuck and we have to
figure out why, but that's where we are already so that's not a reason to
not use it.
4) I'd need more information about precisely where you're applying that
durable=true attribute, but my initial thought is that it's not doing what
you think it's doing. Durability only applies to consumers, and only to
topic consumers, so it's irrelevant for queues. And even for topics, it's
not sufficient to ensure that messages survive a broker restart. Durability
ensures that the consumer is still deemed to be subscribed to the topic
even if the consumer disconnects (say, because the broker restarts), which
means that topic messages that have not yet been consumed by that consumer
are not deemed to be fully consumed and so the broker can't immediately
discard those messages upon restarting. However, for any of that to matter,
you have to have written each message to disk in the first place; if you
didn't, it's just in RAM, and it's lost when the broker is restarted. That
is called persistence, and it's controlled by the message producer, so if
your producers aren't setting that attribute (or if you didn't configure a
plugin to force all messages to be persistent), then that's why you're
losing messages when you restart. Details on how to set the flag when
sending messages are on the wiki.
Tim
On Sep 19, 2017 8:46 AM, "rockies" <la...@bd.com> wrote:
> Thank you Tim and Raffi,
>
> I made the change and re-started brokers.
>
> Tim,
>
> I do see consumers that have no session attached in ActiveMQ Web Console.
> Can I still set replayWhenNoConsumers ? One other thing we have to deal
> with
> is that whenever brokers are re-started most of our queues disappear.
> Doesn't the property durable="true" on JMS leave those queues in place
> after
> a re-start ?
>
>
> Appreciate your time !
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: Stuck messages
Posted by rockies <la...@bd.com>.
Thank you Tim and Raffi,
I made the change and re-started brokers.
Tim,
I do see consumers that have no session attached in ActiveMQ Web Console.
Can I still set replayWhenNoConsumers ? One other thing we have to deal with
is that whenever brokers are re-started most of our queues disappear.
Doesn't the property durable="true" on JMS leave those queues in place after
a re-start ?
Appreciate your time !
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stuck messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
In addition, read and follow the Stuck Messages section of the Network of
Brokers page I linked to earlier to avoid stranding messages.
On Sep 18, 2017 12:42 PM, "Raffi" <ra...@gmail.com> wrote:
> Remove "rebalanceClusterClients=true", it distributes connections across
> brokers, likely exacerbating your problem. Is "duplex="true" enabled from
> both ends of the connector? If so, enable it from one end only, or, remove
> it entirely and use static connections between the brokers.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: Stuck messages
Posted by Raffi <ra...@gmail.com>.
Remove "rebalanceClusterClients=true", it distributes connections across
brokers, likely exacerbating your problem. Is "duplex="true" enabled from
both ends of the connector? If so, enable it from one end only, or, remove
it entirely and use static connections between the brokers.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stuck messages
Posted by rockies <la...@bd.com>.
Yes our Consumers are always up and running and there were actually two
subscribers at the time of this issue we have seen with stuck messages. This
is more seems to happen very frequently with 5.15.0 version causing panic
withing the team on whether its good to set up network of brokers or not. I
used " rebalanceClusterClients" after reading a blog post and did not
realize that priorityBackup negates it. I removed rebalanceClusterClients
from our config now.
What other parameters can I watch to see why this happens frequently on
different subscribers ?
Also we seem to have constant errors on user not authenticated, even though
same user is able to process messages once we re-start the JMS brokers.
Thank you for your time !
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stuck messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
When a message is stuck on a given queue, have you confirmed that there is
at least one consumer for that queue connected to the broker on which the
message resides? A JMX viewer such as JConsole is ideal for determining
this.
I haven't used the rebalanceClusterClients option before, but I expect that
it's not able to guarantee that there is at least one consumer on each
broker for each queue, so I could envision a situation where small numbers
of messages get stuck on a given broker. If that turns out to be what's
going on, you should probably use the replayWhenNoConsumers option
described in the Stuck Messages section of
http://activemq.apache.org/networks-of-brokers.html instead of (or in
addition to, if you want) the rebalanceClusterClients option.
Also, I don't understand why you have both rebalanceClusterClients=true and
priorityBackup=true; those two options work at cross-purposes. What are you
trying to accomplish with the combination of the two?
Tim
On Thu, Sep 14, 2017 at 5:46 PM, rockies <la...@bd.com> wrote:
> Hi,
>
> We have setup network of brokers with just two ActiveMQ nodes in version
> 5.15.0
> Config is as follows:
> <networkConnectors>
> <networkConnector conduitSubscriptions="false"
> decreaseNetworkConsumerPriority="true" duplex="true" dynamicOnly="true"
> name="broker1Tobroker02" networkTTL="3" userName="testadmin"
> password="password" uri="static:tcp://xxxx.test.com:61616">
> </networkConnector>
> </networkConnectors>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="70" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
>
> <transportConnectors>
>
> <transportConnector name="openwire"
> rebalanceClusterClients="true" updateClusterClients="true"
> updateClusterClientsOnRemove="true"
> uri="tcp://0.0.0.0:61616?maximumConnections=1000&
> wireFormat.maxFrameSize=104857600"/>
> <transportConnector name="amqp" rebalanceClusterClients="true"
> updateClusterClients="true" updateClusterClientsOnRemove="true"
> uri="amqp://0.0.0.0:5672?maximumConnections=1000&
> wireFormat.maxFrameSize=104857600"/>
> <transportConnector name="stomp"
> uri="stomp://0.0.0.0:61613?maximumConnections=1000&
> wireFormat.maxFrameSize=104857600"/>
> <transportConnector name="mqtt"
> uri="mqtt://0.0.0.0:1883?maximumConnections=1000&
> wireFormat.maxFrameSize=104857600"/>
> <transportConnector name="ws"
> uri="ws://0.0.0.0:61614?maximumConnections=1000&
> wireFormat.maxFrameSize=104857600"/>
> </transportConnectors>
>
>
> Data is persisted in SQL Server. We use Mule ESB cluster,
> where
> subscribers from both nodes listen to Queues in ActiveMQ with a failover
> URI
> such as :
> failover://(tcp://broker1.test.com:61616,tcp://broker2.test.com:61616
> )?randomize=false&priorityBackup=true&initialReconnectDelay=100
>
> Since this updgrade from 5.11.1 to 5.14.5 to 5.15.0 we are
> seem to have
> more stuck messages in our queues, which usually ends up in re-starting JMS
> brokers and ESB servers. There are atleast close to 200 queues. Message
> size
> rarely gets to 25MB and number of messages per second could be close to 50.
> So why is it that we seem to have more issues with stuck messages ? Can you
> please let us know what whould be the optimal configuration for this kind
> of
> clustered environment?
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>