Re: Queue hanging with n to n network of brokers

[Lucas Dias <lu...@ceabs.com.br>]

Hello,

After many testing, it seems to work fine when reseting the broker. I've
included a reboot step in my recipe to run when a broker comes up or down,
indeed I've seen the network work again.

It looks like the runtime plugin does not fully restores the broker network
when reloading the file.

Answering your question: When looking at the network consumers I see their
number gets increased when there are messages being produced, however
there's no consumption.

The prefetch was just experimental, as you can read at the documentation: 

"If you have very few messages and each message takes a very long time to
process you might want to set the prefetch value to 1 so that a consumer is
given one message at a time."

I was experimenting some values to find any evidence, my hipothesis was the
network connections were holding messages as I've seen too many broker to
broker consumers.

The evidence seems towards the network is still not prepared for dynamic
configurations. 

I will try a few more testing, and eventually come up with more evidences. 

Thank you



--
View this message in context: http://activemq.2283324.n4.nabble.com/Queue-hanging-with-n-to-n-network-of-brokers-tp4707499p4709039.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Queue hanging with n to n network of brokers

[Lucas Dias <lu...@ceabs.com.br>]

Hello again,

I've done exhaustive testing and I ended up finding this URL:

https://issues.apache.org/jira/browse/AMQ-3446?jql=status%20in%20(Open%2C%20%22In%20Progress%22)%20AND%20text%20~%20%22network%20of%20broker%22

The case is the same one as the one I've set up. I would like to focus the
following line from that post:

"
(...)
This BUG does not occur when networkTTL is 1 because subscription route in a
network will be not be greater then 2
"
So indeed, after setting the networkTTL of 1 it works for me (I was using
10).

Hope it helps anyone interested.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Queue-hanging-with-n-to-n-network-of-brokers-tp4707499p4709583.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Queue hanging with n to n network of brokers

[Lucas Dias <lu...@ceabs.com.br>]

I've removed the failover and used only tcp, as in:
static:tcp://myip.sample:61616. I see the same problem.

I've tried also to follow the stuck messages session, even before I started
this thread. I've followed the instructions and included the configs from
that post (by setting a networkBridgeFilterFactory and
replayWhenNoConsumers=true).

The problem stills on. 

I've done two tests: 

1) 100.000 messages sent with a random time reconnecting consumer. There's a
load balancer (Elastic Load Balancer) that distributes traffic on port
61616. Every 10 seconds there's a chance of reconnection of 50% for that
consumer.

2) 100.000 messages sent with a persistent consumer. The consumer stays
connected. I also use the same Load Balancer and setup from test 1.

Both show the same symptons. I've used only one consumer and on a few
occasions I've used more than one, but less than the number of broker
instances, in order to ensure the network would forward the messages. 

Test 1 on performs better, mainly because it varies from broker to broker
and eventually consumes the messages.

The curious is I see above 20 consumers from the bridge, but they don't
forward the messages and their Enqueues/Dequeues/Dispatched counters do not
increase.

>From what I see I would still be able to read from the message properties
it's TTL but I see messages of no TTL get locked.

About the plugin: it is indeed used. When an instance shuts down, AWS
Opsworks triggers a chef recipe that changes the activemq.xml file on the
remaining instances. The code removes the broker that shuts down. By reading
the logs and comparing such logs with the source code (From the latest
branch) I see the plugin works.

I've also tried the latest version of AMQ from the master branch. It
presents the same behavior.

Perhaps I am not using AMQ Networks of brokers the way it should be used.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Queue-hanging-with-n-to-n-network-of-brokers-tp4707499p4709066.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Queue hanging with n to n network of brokers

[Tim Bain <tb...@alumni.duke.edu>]

Actually, maybe not, because I just realized that you're not actually
changing your config.  To confirm that, please disable the config refresh
plugin and see if you still get the problem.

I think your problem is that you specified a networkConnector using
failover without specifying maxReconnectAttempts=0.  Fix that and see if
your problem disappears.

If not, is there any chance you're hitting the problem described in the
"Stuck Messages" section of
http://activemq.apache.org/networks-of-brokers.html?  Please describe which
broker each producer and each consumer are connected to before, during, and
after you restart the broker that causes the problem.

Tim
On Mar 9, 2016 7:37 AM, "Tim Bain" <tb...@alumni.duke.edu> wrote:

> It sounds like you've found a bug.  Please submit a bug report in JIRA,
> including config files and test steps required to reproduce the problem.
> Please reduce the config files to the smallest form that still reproduces
> the problem; eliminate anything that's not required, to make it easier for
> whoever investigates the issue.
>
> Even better would be to create a unit test that demonstrates the problem
> and attach that to the JIRA bug report.
> On Mar 8, 2016 5:43 AM, "Lucas Dias" <lu...@ceabs.com.br> wrote:
>
>> Hello,
>>
>> After many testing, it seems to work fine when reseting the broker. I've
>> included a reboot step in my recipe to run when a broker comes up or down,
>> indeed I've seen the network work again.
>>
>> It looks like the runtime plugin does not fully restores the broker
>> network
>> when reloading the file.
>>
>> Answering your question: When looking at the network consumers I see their
>> number gets increased when there are messages being produced, however
>> there's no consumption.
>>
>> The prefetch was just experimental, as you can read at the documentation:
>>
>> "If you have very few messages and each message takes a very long time to
>> process you might want to set the prefetch value to 1 so that a consumer
>> is
>> given one message at a time."
>>
>> I was experimenting some values to find any evidence, my hipothesis was
>> the
>> network connections were holding messages as I've seen too many broker to
>> broker consumers.
>>
>> The evidence seems towards the network is still not prepared for dynamic
>> configurations.
>>
>> I will try a few more testing, and eventually come up with more evidences.
>>
>> Thank you
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Queue-hanging-with-n-to-n-network-of-brokers-tp4707499p4709039.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>

Re: Queue hanging with n to n network of brokers

[Tim Bain <tb...@alumni.duke.edu>]

It sounds like you've found a bug.  Please submit a bug report in JIRA,
including config files and test steps required to reproduce the problem.
Please reduce the config files to the smallest form that still reproduces
the problem; eliminate anything that's not required, to make it easier for
whoever investigates the issue.

Even better would be to create a unit test that demonstrates the problem
and attach that to the JIRA bug report.
On Mar 8, 2016 5:43 AM, "Lucas Dias" <lu...@ceabs.com.br> wrote:

> Hello,
>
> After many testing, it seems to work fine when reseting the broker. I've
> included a reboot step in my recipe to run when a broker comes up or down,
> indeed I've seen the network work again.
>
> It looks like the runtime plugin does not fully restores the broker network
> when reloading the file.
>
> Answering your question: When looking at the network consumers I see their
> number gets increased when there are messages being produced, however
> there's no consumption.
>
> The prefetch was just experimental, as you can read at the documentation:
>
> "If you have very few messages and each message takes a very long time to
> process you might want to set the prefetch value to 1 so that a consumer is
> given one message at a time."
>
> I was experimenting some values to find any evidence, my hipothesis was the
> network connections were holding messages as I've seen too many broker to
> broker consumers.
>
> The evidence seems towards the network is still not prepared for dynamic
> configurations.
>
> I will try a few more testing, and eventually come up with more evidences.
>
> Thank you
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Queue-hanging-with-n-to-n-network-of-brokers-tp4707499p4709039.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>