You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by JRR <jr...@cisco.com> on 2012/07/07 00:56:31 UTC

ActiveMQ-CPP Producer Flow Control Question

Hello,

I'm having problems with Producer Flow Control. I have a C++ producer that
interacts with a Java application and we are using AMQ to communicate
between
the two. The AMQ Broker is managed by our Java team, and I've been working
with
the C++ producer client.

The java team recently changed the broker's configuration so that it won't
consume all available disk space. They've added the following lines to the
activemq.xml configuration file.

---------------------------------------------------------------------------
		<systemUsage>
		    <systemUsage sendFailIfNoSpace="true">
    			<memoryUsage>
      				<memoryUsage limit="5 mb" />
    			</memoryUsage>
    			<storeUsage>
      				<storeUsage limit="4 gb" />
    			</storeUsage>
    			<tempUsage>
      				<tempUsage limit="1 gb" />
    			</tempUsage>
  			</systemUsage>
		</systemUsage>

---------------------------------------------------------------------------

After this change we had some long running tests where the consumer was
taken
off line while the C++ producer was continuing to send messages. Eventually
the C++ producer locked up (was flow controlled).

I used GDB to attach to the producer process and found the producer was
blocked
on mutexs, etc. deep in the bowels of the 3rd party activemq-cpp code.

I googled sendFailIfNoSpace and landed on the Producer Flow Control page at
     http://activemq.apache.org/producer-flow-control.html
http://activemq.apache.org/producer-flow-control.html 

I've read through this several times, and that's where I learned the overall
name of this "lockup up", and that it's actually expected behavior.


What we want is for the broker to limit its disk usage as specified above.
And
if it reaches the limit the producer gets informed of this, so it can note
the
fact that the broker is full -- but doesn't lock up.

In reading the producer-flow-control.html page it appears that this is
possible
if we configure client side exceptions -- which the broker does using
sendFailIfNoSpace.

However, the second half of this appears to be making the client catch the
javax.jms.ResourceAllocationException. This is were I get lost. This appears
to
be java code -- which won't work for me as I am C++ based. How do I handle
this
exception? I suspect they don't mean simply adding try/catch logic because I
would have then expected my C++ producer to crash with an unhandled
exception.

I'm assuming that I can do this with the C++ API. How would I do this?


We are using:
    AMQ Broker    5.5.0
    activemq-cpp  3.4.0

We are using:
    1. queues,
    2. they are not durable,
    3. the messages are not persistent, and
    4. we use openwire.


The high level flow for the producer client's AMQ exchange is based upon the
Producer from the CMS API Overview Page
(http://activemq.apache.org/cms/cms-api-overview.html)


o. Initialize the Active MQ CPP library (once for the process)

o. Create a connection factory using just the broker URI
   (tcp://localhost:61616)

o. Create a connection from the factory.

o. Start the connection

o. Create a session from the connection using the ack mode of auto
acknowledge.

o. Create a destination queue using the session

o. Create a producer using the destination.

o. Change the producer's delivery mode to NON_PERSISTENT

o. Create the message to send using the session

o. Use the producer to send the message.


We have multiple threads sending in the producer. They all use the same
underlying connection, and create sessions on top of the connection for
sending
their messages.

Thanks in advance for your help,

-=John

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ-CPP Producer Flow Control Question

Posted by JRR <jr...@cisco.com>.
Logged: https://issues.apache.org/jira/browse/AMQCPP-413 (Producer connection
that causes broker to reach its memory/disk limits doesn't get the 'all
full' exception even though the broker is configured to send it for Producer
Flow Control.)

Thank you for your help.

-=John

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781p4653833.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ-CPP Producer Flow Control Question

Posted by JRR <jr...@cisco.com>.
Bleh. I looked more closely. The exception isn't a flow control exception,
but rather a timeout exception:

  what():  Error while sending message[No valid response received for
command: Message { commandId = 8879, responseRequired = true, ProducerId =
ID:psbu-jrr-lnx-44893-1341874696684-0:0:1775:0, Destination =
queue://c.c.p.v.ms.events, TransactionId = NULL, OriginalDestination = NULL,
MessageId = ID:psbu-jrr-lnx-44893-1341874696684-0:0:1775:0:0:0,
OriginalTransactionId = NULL, GroupID = , GroupSequence = 0, CorrelationId =
, Persistent = false, Expiration = 0, Priority = 4, ReplyTo = NULL,
Timestamp = 1341874701001, Type = TEST_TYPE, Content = [size=1045],
MarshalledProperties = NULL, DataStructure = NULL, TargetConsumerId = NULL,
Compressed = false, RedeliveryCounter = 0, BrokerPath = NULL, Arrival = 0,
UserID = , RecievedByDFBridge = false, Droppable = false, Cluster = NULL,
BrokerInTime = 0, BrokerOutTime = 0 }Text = 1024 characters folowed by a
message.xxxxxxxx... world! 1776, check broker.]


So my permanently flow controlled send() timed out and raised that as an
exception rather than the broker full exception.

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781p4653815.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ-CPP Producer Flow Control Question

Posted by JRR <jr...@cisco.com>.
Hi Tim,

Before I do this. I think I found the culprit. And while I feel strongly
this is a defect, I could also see it argued as working as designed.

I revisited the configuration page
(http://activemq.apache.org/cms/configuring.html) after browsing the GDB
output and noticed the following configuration option.


http://activemq.apache.org/cms/configuring.html wrote
> 
>  http://activemq.2283324.n4.nabble.com/file/n4653814/amq_connection.jpg 
> 

So I changed my test to included '&connection.sendTimeout=500' (500 ms if I
recall correctly).

I repeated the test and this time it does not lock up. Instead it pauses
then throws an exception.

I would argue that this is a defect because the Producer Flow Control page
(http://activemq.apache.org/producer-flow-control.html) indicates the
following:

http://activemq.apache.org/producer-flow-control.html wrote
> 
> 
> http://activemq.2283324.n4.nabble.com/file/n4653814/client-side-exception.jpg 
> 

The text above makes me believe that we would simply add this configuration
on the broker and an exception would be raised on the client, regardless as
to the client's sendTimeout setting.  Otherwise the send could 'fail' with
either a timeout or a flow control issue. Where I'm only interested in a
flow control issue issue.

Moreover, I argue that the client shouldn't wait for the timeout to declare
the flow control exception. It should do so immediately. If I create a new
connection and invoked send, it will throw the flow control exception
immediately.

Although I suppose it could be argued that I should also be concerned with
timeouts too otherwise I could get an indefinite lockup if the broker never
responds. But I'd say that's beside the point. We shouldn't lock up the send
side, if we've followed the broker configuration rules to prevent permanent
throttling.

What are you thoughts? Should I still open this using Jira (I ask up front
since I see you are the lead developer for these changes in the CPP client).

Thank you for all your time,

-=John

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781p4653814.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ-CPP Producer Flow Control Question

Posted by Timothy Bish <ta...@gmail.com>.
On Mon, 2012-07-09 at 15:49 -0700, JRR wrote: 
> Hello Tim,
> 
> I tried with the latest CPP version, AMQ-CPP-3.4.4. The problem persists.
> 
> I've trimmed it down to a test case, and I've gathered some GDB information
> for the lock up, etc.
> 
> How would you like me to proceed. Do I post the source, test
> information/results, etc. here?
> 
> Thanks
> 
> -=John

You should open a new Jira issue and post the relevant info there.
https://issues.apache.org/jira/browse/AMQCPP


> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781p4653812.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

-- 
Tim Bish
Sr Software Engineer | FuseSource Corp
tim.bish@fusesource.com | www.fusesource.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/


Re: ActiveMQ-CPP Producer Flow Control Question

Posted by JRR <jr...@cisco.com>.
Hello Tim,

I tried with the latest CPP version, AMQ-CPP-3.4.4. The problem persists.

I've trimmed it down to a test case, and I've gathered some GDB information
for the lock up, etc.

How would you like me to proceed. Do I post the source, test
information/results, etc. here?

Thanks

-=John

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781p4653812.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ-CPP Producer Flow Control Question

Posted by Timothy Bish <ta...@gmail.com>.
On Mon, 2012-07-09 at 10:32 -0700, JRR wrote: 
> Hello Tim,
> 
> Thank you for the suggestion.  I made this change, but it doesn't quite
> work. I did some investigation and it's an interesting scenario....
> 
> 
> I extracted all of the AMQ code from the product into a small test program
> so I
> could more quickly and easily play with the settings. The test program
> creates
> a connection once, and then reuses the connection, building the session,
> destination, message, producer, etc. on top of the connection for each
> message.
> 
> 
> When I change the URI to add '?connection.alwaysSyncSend=true', and then
> cause
> the application to run, it sends approximately 1400 message and then locks
> up
> (throttles indefinitely).
> 
> 
> However, what I found is that if I run /another /instance of the application
> in
> another xterm, That new application */will/* get an exception when it
> reaches the
> producer.send() method -- as expected.
> 
> 
> psbu-jrr-lnx[SUSE10.1]:162>  env NM_MSGS=100 simple_producer.exe
> =====================================================
> Starting the example:
> -----------------------------------------------------
> NM_MSGS set to 100
> Starting Producer Thread 0x8072358
> Creating connection for tcp://127.0.0.1:61616?connection.alwaysSyncSend=true
> terminate called after throwing an instance of 'ee_except'
>   what():  Error while sending message[*** BEGIN SERVER-SIDE STACK TRACE ***
> Message: Usage Manager Temp Store is Full (01002026116f 1048576). Stopping
> producer (ID:psbu-jrr-lnx-36396-1341854427427-0:0:0:0) to prevent flooding
> queue://c.c.p.v.ms.events. See
> http://activemq.apache.org/producer-flow-control.html for more info
> 
> 
> 
> 
> Moreover, if I kill the original locked up process. All subsequent
> invocations
> of the process will now fail with the exception -- as expected.
> 
> 
> My initial lockup problem remains. Can you provide any insights as to how to
> avoid the lockup?
> 
> 
> Also, I was able to add '?connection.alwaysSyncSend=true' to the URI, but I
> would rather invoke the method
> activemq::core::ActiveMQConnectionFactory::setAlwaysSyncSend(). However,
> this
> method doesn't exist for a cms::ConnectionFactory, which is what is returned
> by
> the static method 'createCMSConnectionFActory'. Is there a way to convert
> from
> a CMS connection factory to an AMQ connection factor? Or would this just be
> a
> matter of using the AMQ Connection Factory constructor instead?
> 
> Thanks in advance,
> 
> -=John

You can use dynamic_cast to cast the returned ConnectionFactory instance
to an ActiveMQConnectionFactory if you want to configure the factory in
code.

For the other problem you'd need to create a test case that reproduces
the issue so it can be investigated.

Also recommend you upgrade to the latest release to ensure you have all
the latest fixes. 


> 
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781p4653805.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

-- 
Tim Bish
Sr Software Engineer | FuseSource Corp
tim.bish@fusesource.com | www.fusesource.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/


Re: ActiveMQ-CPP Producer Flow Control Question

Posted by JRR <jr...@cisco.com>.
Hello Tim,

Thank you for the suggestion.  I made this change, but it doesn't quite
work. I did some investigation and it's an interesting scenario....


I extracted all of the AMQ code from the product into a small test program
so I
could more quickly and easily play with the settings. The test program
creates
a connection once, and then reuses the connection, building the session,
destination, message, producer, etc. on top of the connection for each
message.


When I change the URI to add '?connection.alwaysSyncSend=true', and then
cause
the application to run, it sends approximately 1400 message and then locks
up
(throttles indefinitely).


However, what I found is that if I run /another /instance of the application
in
another xterm, That new application */will/* get an exception when it
reaches the
producer.send() method -- as expected.


psbu-jrr-lnx[SUSE10.1]:162>  env NM_MSGS=100 simple_producer.exe
=====================================================
Starting the example:
-----------------------------------------------------
NM_MSGS set to 100
Starting Producer Thread 0x8072358
Creating connection for tcp://127.0.0.1:61616?connection.alwaysSyncSend=true
terminate called after throwing an instance of 'ee_except'
  what():  Error while sending message[*** BEGIN SERVER-SIDE STACK TRACE ***
Message: Usage Manager Temp Store is Full (01002026116f 1048576). Stopping
producer (ID:psbu-jrr-lnx-36396-1341854427427-0:0:0:0) to prevent flooding
queue://c.c.p.v.ms.events. See
http://activemq.apache.org/producer-flow-control.html for more info




Moreover, if I kill the original locked up process. All subsequent
invocations
of the process will now fail with the exception -- as expected.


My initial lockup problem remains. Can you provide any insights as to how to
avoid the lockup?


Also, I was able to add '?connection.alwaysSyncSend=true' to the URI, but I
would rather invoke the method
activemq::core::ActiveMQConnectionFactory::setAlwaysSyncSend(). However,
this
method doesn't exist for a cms::ConnectionFactory, which is what is returned
by
the static method 'createCMSConnectionFActory'. Is there a way to convert
from
a CMS connection factory to an AMQ connection factor? Or would this just be
a
matter of using the AMQ Connection Factory constructor instead?

Thanks in advance,

-=John


--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781p4653805.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ-CPP Producer Flow Control Question

Posted by Timothy Bish <ta...@gmail.com>.
On Fri, 2012-07-06 at 15:56 -0700, JRR wrote: 
> Hello,
> 
> I'm having problems with Producer Flow Control. I have a C++ producer that
> interacts with a Java application and we are using AMQ to communicate
> between
> the two. The AMQ Broker is managed by our Java team, and I've been working
> with
> the C++ producer client.
> 
> The java team recently changed the broker's configuration so that it won't
> consume all available disk space. They've added the following lines to the
> activemq.xml configuration file.
> 
> ---------------------------------------------------------------------------
> 		<systemUsage>
> 		    <systemUsage sendFailIfNoSpace="true">
>     			<memoryUsage>
>       				<memoryUsage limit="5 mb" />
>     			</memoryUsage>
>     			<storeUsage>
>       				<storeUsage limit="4 gb" />
>     			</storeUsage>
>     			<tempUsage>
>       				<tempUsage limit="1 gb" />
>     			</tempUsage>
>   			</systemUsage>
> 		</systemUsage>
> 
> ---------------------------------------------------------------------------


> After this change we had some long running tests where the consumer was
> taken
> off line while the C++ producer was continuing to send messages. Eventually
> the C++ producer locked up (was flow controlled).
> 
> I used GDB to attach to the producer process and found the producer was
> blocked
> on mutexs, etc. deep in the bowels of the 3rd party activemq-cpp code.
> 
> I googled sendFailIfNoSpace and landed on the Producer Flow Control page at
>      http://activemq.apache.org/producer-flow-control.html
> http://activemq.apache.org/producer-flow-control.html 
> 
> I've read through this several times, and that's where I learned the overall
> name of this "lockup up", and that it's actually expected behavior.
> 
> 
> What we want is for the broker to limit its disk usage as specified above.
> And
> if it reaches the limit the producer gets informed of this, so it can note
> the
> fact that the broker is full -- but doesn't lock up.
> 
> In reading the producer-flow-control.html page it appears that this is
> possible
> if we configure client side exceptions -- which the broker does using
> sendFailIfNoSpace.
> 
> However, the second half of this appears to be making the client catch the
> javax.jms.ResourceAllocationException. This is were I get lost. This appears
> to
> be java code -- which won't work for me as I am C++ based. How do I handle
> this
> exception? I suspect they don't mean simply adding try/catch logic because I
> would have then expected my C++ producer to crash with an unhandled
> exception.
> 
> I'm assuming that I can do this with the C++ API. How would I do this?
> 
> 
> We are using:
>     AMQ Broker    5.5.0
>     activemq-cpp  3.4.0
> 
> We are using:
>     1. queues,
>     2. they are not durable,
>     3. the messages are not persistent, and
>     4. we use openwire.
> 
> 
> The high level flow for the producer client's AMQ exchange is based upon the
> Producer from the CMS API Overview Page
> (http://activemq.apache.org/cms/cms-api-overview.html)
> 
> 
> o. Initialize the Active MQ CPP library (once for the process)
> 
> o. Create a connection factory using just the broker URI
>    (tcp://localhost:61616)
> 
> o. Create a connection from the factory.
> 
> o. Start the connection
> 
> o. Create a session from the connection using the ack mode of auto
> acknowledge.
> 
> o. Create a destination queue using the session
> 
> o. Create a producer using the destination.
> 
> o. Change the producer's delivery mode to NON_PERSISTENT
> 
> o. Create the message to send using the session
> 
> o. Use the producer to send the message.
> 
> 
> We have multiple threads sending in the producer. They all use the same
> underlying connection, and create sessions on top of the connection for
> sending
> their messages.
> 
> Thanks in advance for your help,
> 
> -=John
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-CPP-Producer-Flow-Control-Question-tp4653781.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

If you want the CMS client to throw a CMSException for a producer send
operation when you have sendFailIfNoSpace="true" then you need to ensure
that the client uses synchronous send operations for all producer sends
by setting the connection.alwaysSyncSend="true" option on the URI or use
the setter in the connection factory then your client's will not hang
but instead receive the exception. 

-- 
Tim Bish
Sr Software Engineer | FuseSource Corp
tim.bish@fusesource.com | www.fusesource.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/