You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by someword <de...@jivesoftware.com> on 2013/09/18 20:29:57 UTC

Did I turn off producer flow control correctly?

Hello.
We are running activemq 5.8, on centos 6.X with OpenJDK 64-Bit Server VM
(build 20.0-b12, mixed mode).

I have a situation where I think I have turned off flow control but I am
seeing seeing behavior that makes me think flow control is still on.

The activemq broker configurations are here
https://github.com/someword/activemq-configs/tree/master/nabble-configs
(minus auth bits).

In my test scenario I have nodes i'll call node-client and node-server.  
Node-server connects to an activemq spoke and subscribes to tive.nodes
waiting for messages to come in to reply to.   Node-client connects to an
activemq hub and publishes messages to tive.nodes which and include a
temporary reply-to queue in the message (eg: global.reply.spammy_<PID>).

The publishing application on node-client happily publishes messages for a
second or so and then the client hangs as it's blocked on a write.  At this
point it seems as if activemq flow control on the hub broker has kicked in
and stopped accepting messages from node-client.    

When looking at the queue counters I see that all of the messages published
by node-client reached node-server and got a response in the reply-to queue.   
The reply to queue usually has 500 - 900 messages in the reply-to queue
marked as 'dequeued'.    The problem is that I don't want to consume any
messages until I have finished publishing everything I have to publish.

So I took the steps that I felt were required to disable producer flow
control.   Here are the highlights from the referenced configs which I
thought would turn of off flow control.   Any ideas what I'm missing?

       <policyEntry topic=">" producerFlowControl="false"/>
              <policyEntry queue=">" gcInactiveDestinations="true"
inactiveTimoutBeforeGC="300000" producerFlowControl="false"
queuePrefetch="1">
              
                <deadLetterStrategy>
                  <sharedDeadLetterStrategy processExpired="false" />
                </deadLetterStrategy>
                <pendingQueuePolicy>
                  <vmQueueCursor />
                </pendingQueuePolicy>


TIA.  Derek.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Did I turn off producer flow control correctly?

Posted by someword <de...@jivesoftware.com>.
Thanks!
The updated test runs to completion 

I do see this which happened last time to (just this one occurrence during
the test run)

making progress 0
You had an issue sending back messages!
making progress 1

The test run summary looks clean though
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.081 sec

At this point is it fair to say that the symptoms we are seeing are not
related to activemq blocking the producer but one or all of the ruby stomp
gem, ssl and or ruby itself?








--
View this message in context: http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671918.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Did I turn off producer flow control correctly?

Posted by someword <de...@jivesoftware.com>.
I have taken some thread dumps on the hub and broker.
The first thread dump occurs before the execution of the process which
exhibits the blocking behavior.   The remaining thread dumps are taken while
the application is still running but blocked.

The hub thread dumps are at http://pastebin.com/shkg2WEy
The spoke thread dumps are at http://pastebin.com/AXFhPMWU

Thanks.  Derek.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671921.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Did I turn off producer flow control correctly?

Posted by Christian Posta <ch...@gmail.com>.
Well, not sure. I would review the test case to make sure it's set up
exactly the way you guys have it.
The goal is to get the test case to lock up somehow so that it doesn't
reach "making progress 99"

Or you could take some time-series (3-4 snapshots) of the broker's threads
(both brokers) so that we can see the progression of the threads over time
to see what they're doing.




On Fri, Sep 27, 2013 at 11:28 AM, someword <de...@jivesoftware.com>wrote:

> Blarhg!  My prior post was out of order.
>
> Thanks!
> The updated test runs to completion
>
> I do see this which happened last time to (just this one occurrence during
> the test run)
>
> making progress 0
> You had an issue sending back messages!
> making progress 1
>
> The test run summary looks clean though
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.081 sec
>
> At this point is it fair to say that the symptoms we are seeing are not
> related to activemq blocking the producer but one or all of the ruby stomp
> gem, ssl and or ruby itself?
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671919.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Re: Did I turn off producer flow control correctly?

Posted by someword <de...@jivesoftware.com>.
Blarhg!  My prior post was out of order.

Thanks! 
The updated test runs to completion 

I do see this which happened last time to (just this one occurrence during
the test run) 

making progress 0 
You had an issue sending back messages! 
making progress 1 

The test run summary looks clean though 
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.081 sec 

At this point is it fair to say that the symptoms we are seeing are not
related to activemq blocking the producer but one or all of the ruby stomp
gem, ssl and or ruby itself? 





--
View this message in context: http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671919.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Did I turn off producer flow control correctly?

Posted by Christian Posta <ch...@gmail.com>.
Fixed, take a look now.




On Fri, Sep 27, 2013 at 9:45 AM, someword <de...@jivesoftware.com>wrote:

> The mvn clean install -Dtest=false -DfailIfNoTests=false failed with this
> error http://pastebin.com/mHbPfuKL .  However it appears that this didn't
> hamper the test suite from running?
>
> Running mvn test
> -Dtest=org.apache.activemq.usecases.TwoBrokerReqeustReplyStompTempDestTest
>
> produced these results http://pastebin.com/VMyQtz9i
>
> It looks like out of the gate one problem occurred and then it ran fine
> until OOM.
>
> Should I increase the heap the or should the test suite run with heap
> defined in the test suite?
> Thanks again for your help.
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671912.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Re: Did I turn off producer flow control correctly?

Posted by someword <de...@jivesoftware.com>.
The mvn clean install -Dtest=false -DfailIfNoTests=false failed with this
error http://pastebin.com/mHbPfuKL .  However it appears that this didn't
hamper the test suite from running?

Running mvn test 
-Dtest=org.apache.activemq.usecases.TwoBrokerReqeustReplyStompTempDestTest 

produced these results http://pastebin.com/VMyQtz9i

It looks like out of the gate one problem occurred and then it ran fine
until OOM.

Should I increase the heap the or should the test suite run with heap
defined in the test suite?
Thanks again for your help.






--
View this message in context: http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671912.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Did I turn off producer flow control correctly?

Posted by Christian Posta <ch...@gmail.com>.
Yah, mvn clean install -Dtest=false will build everything.
Actually, i pulled out the test into my test sandbox here:

https://github.com/christian-posta/camel-sandbox/blob/master/activemq-5.8-issues/src/test/java/org/apache/activemq/usecases/TwoBrokerReqeustReplyStompTempDestTest.java

You can clone the proj and build w/

$> mvn clean install -Dtest=false -DfailIfNoTests=false

Then navigate to the activemq-5.8-issues dir and can run test w/

$> mvn test
-Dtest=org.apache.activemq.usecases.TwoBrokerReqeustReplyStompTempDestTest




On Thu, Sep 26, 2013 at 11:19 AM, someword <de...@jivesoftware.com>wrote:

> ceposta wrote
> > "i've gone ahead and put together the unit test here:
> >
> >
> https://github.com/christian-posta/activemq/blob/someword-from-mailing-list/activemq-unit-tests/src/test/java/org/apache/activemq/usecases/TwoBrokerReqeustReplyStompTempDestTest.java
> "
> >
> > ceposta
> > Thanks for putting the test together.   I've cloned the repo but i'm not
> > entirely clear on what to do.  I think I am supposed to 'build' the test
> > on one of our brokers.  Something like this
> > mvn install -Dtest=TwoBrokerReqeustReplyStompTempDestTest
> >
> > Then if that works I should adjust some of the test to try to get it to
> > fail.
> > Is that pretty accurate?
> >
> > Thanks. Derek.
> >
> >
> >
> > Do you have any links you could point me to which describe the process to
> > build this test?   I
> > It's a branch of the activemq code base, and the test is in there. I
> would
> > recommend running the test to see that it passes, then tweak it to get it
> > just right (like the environment you guys have) to get it to fail.
> >
> > I realize might be a lot going on there in the test, but if you and your
> > team can recreate the issue there, it would be much easier to solve :)
> >
> >
> > On Wed, Sep 25, 2013 at 3:01 PM, someword &lt;
>
> > derek.olsen@
>
> > &gt;wrote:
> >
> >>
> >> "Maybe you can post more about your clients?
> >> So your flow is Client -> Hub -> Spoke -> Server at which point the
> >> server
> >> responds to a temporary destination which flows messages Server -> Spoke
> >> ->
> >> Hub -> Client right?"
> >>
> >>
> >> That is correct.
> >>
> >> "How many messages does the server respond w/ over the temporary
> >> destination
> >> for each message the client sends in?"
> >>
> >> The server will respond with just one response to each message it gets.
> >> The server responds just fine for every single request it gets.
>  What's
> >> interesting is that during the course of about 1-2 seconds wall clock
> >> time
> >> 500 - 800 messages get to the server and responses get back to the
> >> reply-to
> >> queue before the blocking occurs.
> >>
> >>
> >> The underlying application we are using is called MCollective
> >> http://puppetlabs.com/mcollective.   I will post the code we are using
> >> which
> >> uses mcollective and stomp to recreate the problem to assist in
> >> troubleshooting.
> >>
> >> The code which runs on the server is here http://pastebin.com/7UgTpT32
> >> The code which runs on the client is here http://pastebin.com/kDYXYndv
> >>
> >> Please let me know what else I can provide.
> >> Thanks. Derek.
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
> >>
> >> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link
> >> to
> >> a
> >> > thread dump from the broker where the blocking is occurring -
> >> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
> >> strace/gdb
> >> > output of the ruby client in case that's helpful -
> >> > http://pastebin.com/FTKfczVd.
> >> >
> >> > Any ideas?
> >> > Thanks.  Derek.
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >>
> >>
> >>
> >> --
> >> *Christian Posta*
> >> http://www.christianposta.com/blog
> >> twitter: @christianposta
> >>
> >>
> >> Maybe you can post more about your clients?
> >> So your flow is Client -> Hub -> Spoke -> Server at which point the
> >> server
> >> responds to a temporary destination which flows messages Server -> Spoke
> >> ->
> >> Hub -> Client right?
> >>
> >> How many messages does the server respond w/ over the temporary
> >> destination
> >> for each message the client sends in?
> >>
> >>
> >> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
> >>
> >> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link
> >> to
> >> a
> >> > thread dump from the broker where the blocking is occurring -
> >> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
> >> strace/gdb
> >> > output of the ruby client in case that's helpful -
> >> > http://pastebin.com/FTKfczVd.
> >> >
> >> > Any ideas?
> >> > Thanks.  Derek.
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >>
> >>
> >>
> >> --
> >> *Christian Posta*
> >> http://www.christianposta.com/blog
> >> twitter: @christianposta
> >>
> >>
> >> Maybe you can post more about your clients?
> >> So your flow is Client -> Hub -> Spoke -> Server at which point the
> >> server
> >> responds to a temporary destination which flows messages Server -> Spoke
> >> ->
> >> Hub -> Client right?
> >>
> >> How many messages does the server respond w/ over the temporary
> >> destination
> >> for each message the client sends in?
> >>
> >>
> >> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
> >>
> >> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link
> >> to
> >> a
> >> > thread dump from the broker where the blocking is occurring -
> >> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
> >> strace/gdb
> >> > output of the ruby client in case that's helpful -
> >> > http://pastebin.com/FTKfczVd.
> >> >
> >> > Any ideas?
> >> > Thanks.  Derek.
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >>
> >>
> >>
> >> --
> >> *Christian Posta*
> >> http://www.christianposta.com/blog
> >> twitter: @christianposta
> >>
> >>
> >> Maybe you can post more about your clients?
> >> So your flow is Client -> Hub -> Spoke -> Server at which point the
> >> server
> >> responds to a temporary destination which flows messages Server -> Spoke
> >> ->
> >> Hub -> Client right?
> >>
> >> How many messages does the server respond w/ over the temporary
> >> destination
> >> for each message the client sends in?
> >>
> >>
> >> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
> >>
> >> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link
> >> to
> >> a
> >> > thread dump from the broker where the blocking is occurring -
> >> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
> >> strace/gdb
> >> > output of the ruby client in case that's helpful -
> >> > http://pastebin.com/FTKfczVd.
> >> >
> >> > Any ideas?
> >> > Thanks.  Derek.
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >>
> >>
> >>
> >> --
> >> *Christian Posta*
> >> http://www.christianposta.com/blog
> >> twitter: @christianposta
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671839.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671862.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Re: Did I turn off producer flow control correctly?

Posted by someword <de...@jivesoftware.com>.
ceposta wrote
> "i've gone ahead and put together the unit test here:
> 
> https://github.com/christian-posta/activemq/blob/someword-from-mailing-list/activemq-unit-tests/src/test/java/org/apache/activemq/usecases/TwoBrokerReqeustReplyStompTempDestTest.java"
> 
> ceposta
> Thanks for putting the test together.   I've cloned the repo but i'm not
> entirely clear on what to do.  I think I am supposed to 'build' the test
> on one of our brokers.  Something like this
> mvn install -Dtest=TwoBrokerReqeustReplyStompTempDestTest
> 
> Then if that works I should adjust some of the test to try to get it to
> fail.
> Is that pretty accurate?
> 
> Thanks. Derek.
> 
> 
> 
> Do you have any links you could point me to which describe the process to
> build this test?   I
> It's a branch of the activemq code base, and the test is in there. I would
> recommend running the test to see that it passes, then tweak it to get it
> just right (like the environment you guys have) to get it to fail.
> 
> I realize might be a lot going on there in the test, but if you and your
> team can recreate the issue there, it would be much easier to solve :)
> 
> 
> On Wed, Sep 25, 2013 at 3:01 PM, someword &lt;

> derek.olsen@

> &gt;wrote:
> 
>>
>> "Maybe you can post more about your clients?
>> So your flow is Client -> Hub -> Spoke -> Server at which point the
>> server
>> responds to a temporary destination which flows messages Server -> Spoke
>> ->
>> Hub -> Client right?"
>>
>>
>> That is correct.
>>
>> "How many messages does the server respond w/ over the temporary
>> destination
>> for each message the client sends in?"
>>
>> The server will respond with just one response to each message it gets.
>> The server responds just fine for every single request it gets.    What's
>> interesting is that during the course of about 1-2 seconds wall clock
>> time
>> 500 - 800 messages get to the server and responses get back to the
>> reply-to
>> queue before the blocking occurs.
>>
>>
>> The underlying application we are using is called MCollective
>> http://puppetlabs.com/mcollective.   I will post the code we are using
>> which
>> uses mcollective and stomp to recreate the problem to assist in
>> troubleshooting.
>>
>> The code which runs on the server is here http://pastebin.com/7UgTpT32
>> The code which runs on the client is here http://pastebin.com/kDYXYndv
>>
>> Please let me know what else I can provide.
>> Thanks. Derek.
>>
>>
>>
>>
>>
>>
>> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
>>
>> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link
>> to
>> a
>> > thread dump from the broker where the blocking is occurring -
>> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
>> strace/gdb
>> > output of the ruby client in case that's helpful -
>> > http://pastebin.com/FTKfczVd.
>> >
>> > Any ideas?
>> > Thanks.  Derek.
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta
>>
>>
>> Maybe you can post more about your clients?
>> So your flow is Client -> Hub -> Spoke -> Server at which point the
>> server
>> responds to a temporary destination which flows messages Server -> Spoke
>> ->
>> Hub -> Client right?
>>
>> How many messages does the server respond w/ over the temporary
>> destination
>> for each message the client sends in?
>>
>>
>> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
>>
>> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link
>> to
>> a
>> > thread dump from the broker where the blocking is occurring -
>> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
>> strace/gdb
>> > output of the ruby client in case that's helpful -
>> > http://pastebin.com/FTKfczVd.
>> >
>> > Any ideas?
>> > Thanks.  Derek.
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta
>>
>>
>> Maybe you can post more about your clients?
>> So your flow is Client -> Hub -> Spoke -> Server at which point the
>> server
>> responds to a temporary destination which flows messages Server -> Spoke
>> ->
>> Hub -> Client right?
>>
>> How many messages does the server respond w/ over the temporary
>> destination
>> for each message the client sends in?
>>
>>
>> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
>>
>> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link
>> to
>> a
>> > thread dump from the broker where the blocking is occurring -
>> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
>> strace/gdb
>> > output of the ruby client in case that's helpful -
>> > http://pastebin.com/FTKfczVd.
>> >
>> > Any ideas?
>> > Thanks.  Derek.
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta
>>
>>
>> Maybe you can post more about your clients?
>> So your flow is Client -> Hub -> Spoke -> Server at which point the
>> server
>> responds to a temporary destination which flows messages Server -> Spoke
>> ->
>> Hub -> Client right?
>>
>> How many messages does the server respond w/ over the temporary
>> destination
>> for each message the client sends in?
>>
>>
>> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
>>
>> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link
>> to
>> a
>> > thread dump from the broker where the blocking is occurring -
>> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
>> strace/gdb
>> > output of the ruby client in case that's helpful -
>> > http://pastebin.com/FTKfczVd.
>> >
>> > Any ideas?
>> > Thanks.  Derek.
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671839.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 
> -- 
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta





--
View this message in context: http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671862.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Did I turn off producer flow control correctly?

Posted by Christian Posta <ch...@gmail.com>.
@someword:

might be best to take stack traces of both brokers at 10s intervals for
about 4-5 periods... i saw some suspicious threads, but would need to see
what kind of progress they made. and curious about what the network bridge
was doing on the spoke.

in any event, it would be best to try and reproduce this in a unit test..
especially if you can recreate this on the latest 5.9 snapshots.

i've gone ahead and put together the unit test here:

https://github.com/christian-posta/activemq/blob/someword-from-mailing-list/activemq-unit-tests/src/test/java/org/apache/activemq/usecases/TwoBrokerReqeustReplyStompTempDestTest.java

It's a branch of the activemq code base, and the test is in there. I would
recommend running the test to see that it passes, then tweak it to get it
just right (like the environment you guys have) to get it to fail.

I realize might be a lot going on there in the test, but if you and your
team can recreate the issue there, it would be much easier to solve :)


On Wed, Sep 25, 2013 at 3:01 PM, someword <de...@jivesoftware.com>wrote:

>
> "Maybe you can post more about your clients?
> So your flow is Client -> Hub -> Spoke -> Server at which point the server
> responds to a temporary destination which flows messages Server -> Spoke ->
> Hub -> Client right?"
>
>
> That is correct.
>
> "How many messages does the server respond w/ over the temporary
> destination
> for each message the client sends in?"
>
> The server will respond with just one response to each message it gets.
> The server responds just fine for every single request it gets.    What's
> interesting is that during the course of about 1-2 seconds wall clock time
> 500 - 800 messages get to the server and responses get back to the reply-to
> queue before the blocking occurs.
>
>
> The underlying application we are using is called MCollective
> http://puppetlabs.com/mcollective.   I will post the code we are using
> which
> uses mcollective and stomp to recreate the problem to assist in
> troubleshooting.
>
> The code which runs on the server is here http://pastebin.com/7UgTpT32
> The code which runs on the client is here http://pastebin.com/kDYXYndv
>
> Please let me know what else I can provide.
> Thanks. Derek.
>
>
>
>
>
>
> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
>
> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to
> a
> > thread dump from the broker where the blocking is occurring -
> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
> strace/gdb
> > output of the ruby client in case that's helpful -
> > http://pastebin.com/FTKfczVd.
> >
> > Any ideas?
> > Thanks.  Derek.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>
> Maybe you can post more about your clients?
> So your flow is Client -> Hub -> Spoke -> Server at which point the server
> responds to a temporary destination which flows messages Server -> Spoke ->
> Hub -> Client right?
>
> How many messages does the server respond w/ over the temporary destination
> for each message the client sends in?
>
>
> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
>
> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to
> a
> > thread dump from the broker where the blocking is occurring -
> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
> strace/gdb
> > output of the ruby client in case that's helpful -
> > http://pastebin.com/FTKfczVd.
> >
> > Any ideas?
> > Thanks.  Derek.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>
> Maybe you can post more about your clients?
> So your flow is Client -> Hub -> Spoke -> Server at which point the server
> responds to a temporary destination which flows messages Server -> Spoke ->
> Hub -> Client right?
>
> How many messages does the server respond w/ over the temporary destination
> for each message the client sends in?
>
>
> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
>
> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to
> a
> > thread dump from the broker where the blocking is occurring -
> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
> strace/gdb
> > output of the ruby client in case that's helpful -
> > http://pastebin.com/FTKfczVd.
> >
> > Any ideas?
> > Thanks.  Derek.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>
> Maybe you can post more about your clients?
> So your flow is Client -> Hub -> Spoke -> Server at which point the server
> responds to a temporary destination which flows messages Server -> Spoke ->
> Hub -> Client right?
>
> How many messages does the server respond w/ over the temporary destination
> for each message the client sends in?
>
>
> On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:
>
> > I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to
> a
> > thread dump from the broker where the blocking is occurring -
> > http://pastebin.com/ew0cL6Kd.  Additionally I've captured some
> strace/gdb
> > output of the ruby client in case that's helpful -
> > http://pastebin.com/FTKfczVd.
> >
> > Any ideas?
> > Thanks.  Derek.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671839.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Re: Did I turn off producer flow control correctly?

Posted by someword <de...@jivesoftware.com>.
"Maybe you can post more about your clients?
So your flow is Client -> Hub -> Spoke -> Server at which point the server
responds to a temporary destination which flows messages Server -> Spoke ->
Hub -> Client right?"


That is correct.   

"How many messages does the server respond w/ over the temporary destination
for each message the client sends in?"

The server will respond with just one response to each message it gets.   
The server responds just fine for every single request it gets.    What's
interesting is that during the course of about 1-2 seconds wall clock time
500 - 800 messages get to the server and responses get back to the reply-to
queue before the blocking occurs.


The underlying application we are using is called MCollective
http://puppetlabs.com/mcollective.   I will post the code we are using which
uses mcollective and stomp to recreate the problem to assist in
troubleshooting.

The code which runs on the server is here http://pastebin.com/7UgTpT32
The code which runs on the client is here http://pastebin.com/kDYXYndv

Please let me know what else I can provide.    
Thanks. Derek.






On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:

> I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to a
> thread dump from the broker where the blocking is occurring -
> http://pastebin.com/ew0cL6Kd.  Additionally I've captured some strace/gdb
> output of the ruby client in case that's helpful -
> http://pastebin.com/FTKfczVd.
>
> Any ideas?
> Thanks.  Derek.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta


Maybe you can post more about your clients?
So your flow is Client -> Hub -> Spoke -> Server at which point the server
responds to a temporary destination which flows messages Server -> Spoke ->
Hub -> Client right?

How many messages does the server respond w/ over the temporary destination
for each message the client sends in?


On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:

> I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to a
> thread dump from the broker where the blocking is occurring -
> http://pastebin.com/ew0cL6Kd.  Additionally I've captured some strace/gdb
> output of the ruby client in case that's helpful -
> http://pastebin.com/FTKfczVd.
>
> Any ideas?
> Thanks.  Derek.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta


Maybe you can post more about your clients?
So your flow is Client -> Hub -> Spoke -> Server at which point the server
responds to a temporary destination which flows messages Server -> Spoke ->
Hub -> Client right?

How many messages does the server respond w/ over the temporary destination
for each message the client sends in?


On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:

> I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to a
> thread dump from the broker where the blocking is occurring -
> http://pastebin.com/ew0cL6Kd.  Additionally I've captured some strace/gdb
> output of the ruby client in case that's helpful -
> http://pastebin.com/FTKfczVd.
>
> Any ideas?
> Thanks.  Derek.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta


Maybe you can post more about your clients?
So your flow is Client -> Hub -> Spoke -> Server at which point the server
responds to a temporary destination which flows messages Server -> Spoke ->
Hub -> Client right?

How many messages does the server respond w/ over the temporary destination
for each message the client sends in?


On Wed, Sep 25, 2013 at 2:02 PM, someword &lt;derek.olsen@&gt;wrote:

> I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to a
> thread dump from the broker where the blocking is occurring -
> http://pastebin.com/ew0cL6Kd.  Additionally I've captured some strace/gdb
> output of the ruby client in case that's helpful -
> http://pastebin.com/FTKfczVd.
>
> Any ideas?
> Thanks.  Derek.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta




--
View this message in context: http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671839.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Did I turn off producer flow control correctly?

Posted by Christian Posta <ch...@gmail.com>.
Maybe you can post more about your clients?
So your flow is Client -> Hub -> Spoke -> Server at which point the server
responds to a temporary destination which flows messages Server -> Spoke ->
Hub -> Client right?

How many messages does the server respond w/ over the temporary destination
for each message the client sends in?


On Wed, Sep 25, 2013 at 2:02 PM, someword <de...@jivesoftware.com>wrote:

> I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to a
> thread dump from the broker where the blocking is occurring -
> http://pastebin.com/ew0cL6Kd.  Additionally I've captured some strace/gdb
> output of the ruby client in case that's helpful -
> http://pastebin.com/FTKfczVd.
>
> Any ideas?
> Thanks.  Derek.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Re: Did I turn off producer flow control correctly?

Posted by someword <de...@jivesoftware.com>.
I tried using 5.9 SNAPSHOT and get the same behavior.  Here is a link to a
thread dump from the broker where the blocking is occurring -
http://pastebin.com/ew0cL6Kd.  Additionally I've captured some strace/gdb
output of the ruby client in case that's helpful -
http://pastebin.com/FTKfczVd.

Any ideas?
Thanks.  Derek.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.