You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by BRoach <br...@returnpath.net> on 2008/08/17 10:23:41 UTC
activeMQ 5.1 + activemq-cpp-2.2 openwire transaction problem
There seems to be a rather nasty bug having to do with transactions in the
c++ client and openwire.
If you send a large number of messages inside a transaction, and ActiveMQ
runs out of memory, cms::MessageProducer::send() throws an exception (as it
should).
The exception text is: No valid response received for command
The problem is ... the last message sent (the one that caused the exception
to be thrown) gets added to the queue while the others are rolled back. This
is a large problem, as the whole point of transactions is to avoid this sort
of thing.
To summarize:
- Start sending messages with the session set to SESSION_TRANSACTED
- Fill ActiveMQ memory
- Receive exception
- The last message sent will be in the queue. All other messages will be
rolled back.
Here's what the debug log looks like:
DEBUG Usage - Memory usage change. from: 2, to: 1
DEBUG AMQMessageStore - Transacted message add rollback for:
68445488-66bc-c258-0d30-4a1faede93a5:0:1:109, at: offset = 4527877, file =
6, size = 47444, type = 1
DEBUG Usage - Memory usage change. from: 1, to: 0
DEBUG AMQMessageStore - Transacted message add rollback for:
68445488-66bc-c258-0d30-4a1faede93a5:0:1:110, at: offset = 4575321, file =
6, size = 47439, type = 1
DEBUG TransportConnection - Connection Stopped: /127.0.0.1:44382
DEBUG AMQMessageStore - Journalled message add for:
68445488-66bc-c258-0d30-4a1faede93a5:0:1:111, at: offset = 4622760, file =
6, size = 47442, type = 1
As you can see, it adds a message to the queue after the rollback and after
stopping the connection!
Trying to use Stomp rather than openwire creates a whole other set of
problems (at least using the activemq-cpp API). It blocks when you fill the
memory (ok, not so bad) ... but if you kill your client because it's
blocking forever ... activeMQ somehow doesn't lose the socket and you're
completely hosed (you have to restart activeMQ).
I wrote a quick test C client using libstomp rather than activemq-cpp and
can code around this problem using a timeout and some sanity checking ...
but it's a bit of work and I really don't want to have to re-write the
application I've already completed using activemq-cpp.
Thoughts?
Thanks in advance,
Brian Roach
--
View this message in context: http://www.nabble.com/activeMQ-5.1-%2B-activemq-cpp-2.2-openwire-transaction-problem-tp19018190p19018190.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: activeMQ 5.1 + activemq-cpp-2.2 openwire transaction problem
Posted by BRoach <br...@returnpath.net>.
Here's a bit of a followup:
I dusted off my Java hat and modified the test Producer that comes with
activeMQ to simulate the same condition. It will block forever once memory
fills. Killing the client when in this state causes a rollback to occur and
no messages are left in the queue. While this is still not ideal (you're in
a transaction, so no consumer can possibly relieve the situation, leaving
you blocked forever), it's better than the cpp API. A timeout for the send
would be a happy thing here. Or I think I was reading where you're going to
add a topic for resource usage messages?
I also thought to try the latest cpp from trunk. It now exhibits a slightly
different behavior from 2.2.
The trunk (2.3?) now blocks when the memory fill occurs. However, if you
kill the client, the same problem occurs - the last message winds up in the
queue outside of the transaction. And, unfortunately, wrapping the
Producer::send() in an alarm and destroying the session on a timeout doesn't
solve the problem - still get that last message sent.
I tried wading through the cpp code for a couple hours but I got lost in the
abstraction when the send() was getting passed back down through the objects
as request() back to the Connector class.
I also found that the increasing the
transport.ResponseCorrelator.maxResponseWaitTime setting in cpp 2.2 shows
that the producer is blocking on send() until that timeout occurs. Once that
timeout happens, the exception gets thrown and all bets are off. That
doesn't really matter though as it wouldn't be a problem if the errant
message wasn't winding up in the queue.
I'll cut and paste a cpp test case together from my application code and
open a ticket.
Thanks for the input.
Brian Roach
nmittler wrote:
>
> Definitely sounds like a bug - would you mind capturing this in a JIRA
> issue? If you have a code snippet, you can attach that as well, and
> we'll take a look.
>
> Thanks,
> Nate
>
> On Aug 17, 2008, at 1:23 AM, BRoach wrote:
>
>>
>> There seems to be a rather nasty bug having to do with transactions
>> in the
>> c++ client and openwire.
>>
>> If you send a large number of messages inside a transaction, and
>> ActiveMQ
>> runs out of memory, cms::MessageProducer::send() throws an exception
>> (as it
>> should).
>>
>> The exception text is: No valid response received for command
>>
>> The problem is ... the last message sent (the one that caused the
>> exception
>> to be thrown) gets added to the queue while the others are rolled
>> back. This
>> is a large problem, as the whole point of transactions is to avoid
>> this sort
>> of thing.
>>
>> To summarize:
>> - Start sending messages with the session set to SESSION_TRANSACTED
>> - Fill ActiveMQ memory
>> - Receive exception
>> - The last message sent will be in the queue. All other messages
>> will be
>> rolled back.
>>
>> Here's what the debug log looks like:
>> DEBUG Usage - Memory usage change. from:
>> 2, to: 1
>> DEBUG AMQMessageStore - Transacted message add
>> rollback for:
>> 68445488-66bc-c258-0d30-4a1faede93a5:0:1:109, at: offset = 4527877,
>> file =
>> 6, size = 47444, type = 1
>> DEBUG Usage - Memory usage change. from:
>> 1, to: 0
>> DEBUG AMQMessageStore - Transacted message add
>> rollback for:
>> 68445488-66bc-c258-0d30-4a1faede93a5:0:1:110, at: offset = 4575321,
>> file =
>> 6, size = 47439, type = 1
>> DEBUG TransportConnection - Connection Stopped: /
>> 127.0.0.1:44382
>> DEBUG AMQMessageStore - Journalled message add for:
>> 68445488-66bc-c258-0d30-4a1faede93a5:0:1:111, at: offset = 4622760,
>> file =
>> 6, size = 47442, type = 1
>>
>> As you can see, it adds a message to the queue after the rollback
>> and after
>> stopping the connection!
>>
>> Trying to use Stomp rather than openwire creates a whole other set of
>> problems (at least using the activemq-cpp API). It blocks when you
>> fill the
>> memory (ok, not so bad) ... but if you kill your client because it's
>> blocking forever ... activeMQ somehow doesn't lose the socket and
>> you're
>> completely hosed (you have to restart activeMQ).
>>
>> I wrote a quick test C client using libstomp rather than activemq-
>> cpp and
>> can code around this problem using a timeout and some sanity
>> checking ...
>> but it's a bit of work and I really don't want to have to re-write the
>> application I've already completed using activemq-cpp.
>>
>> Thoughts?
>>
>> Thanks in advance,
>> Brian Roach
>>
>> --
>> View this message in context:
>> http://www.nabble.com/activeMQ-5.1-%2B-activemq-cpp-2.2-openwire-transaction-problem-tp19018190p19018190.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/activeMQ-5.1-%2B-activemq-cpp-2.2-openwire-transaction-problem-tp19018190p19025170.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: activeMQ 5.1 + activemq-cpp-2.2 openwire transaction problem
Posted by Nathan Mittler <na...@gmail.com>.
Definitely sounds like a bug - would you mind capturing this in a JIRA
issue? If you have a code snippet, you can attach that as well, and
we'll take a look.
Thanks,
Nate
On Aug 17, 2008, at 1:23 AM, BRoach wrote:
>
> There seems to be a rather nasty bug having to do with transactions
> in the
> c++ client and openwire.
>
> If you send a large number of messages inside a transaction, and
> ActiveMQ
> runs out of memory, cms::MessageProducer::send() throws an exception
> (as it
> should).
>
> The exception text is: No valid response received for command
>
> The problem is ... the last message sent (the one that caused the
> exception
> to be thrown) gets added to the queue while the others are rolled
> back. This
> is a large problem, as the whole point of transactions is to avoid
> this sort
> of thing.
>
> To summarize:
> - Start sending messages with the session set to SESSION_TRANSACTED
> - Fill ActiveMQ memory
> - Receive exception
> - The last message sent will be in the queue. All other messages
> will be
> rolled back.
>
> Here's what the debug log looks like:
> DEBUG Usage - Memory usage change. from:
> 2, to: 1
> DEBUG AMQMessageStore - Transacted message add
> rollback for:
> 68445488-66bc-c258-0d30-4a1faede93a5:0:1:109, at: offset = 4527877,
> file =
> 6, size = 47444, type = 1
> DEBUG Usage - Memory usage change. from:
> 1, to: 0
> DEBUG AMQMessageStore - Transacted message add
> rollback for:
> 68445488-66bc-c258-0d30-4a1faede93a5:0:1:110, at: offset = 4575321,
> file =
> 6, size = 47439, type = 1
> DEBUG TransportConnection - Connection Stopped: /
> 127.0.0.1:44382
> DEBUG AMQMessageStore - Journalled message add for:
> 68445488-66bc-c258-0d30-4a1faede93a5:0:1:111, at: offset = 4622760,
> file =
> 6, size = 47442, type = 1
>
> As you can see, it adds a message to the queue after the rollback
> and after
> stopping the connection!
>
> Trying to use Stomp rather than openwire creates a whole other set of
> problems (at least using the activemq-cpp API). It blocks when you
> fill the
> memory (ok, not so bad) ... but if you kill your client because it's
> blocking forever ... activeMQ somehow doesn't lose the socket and
> you're
> completely hosed (you have to restart activeMQ).
>
> I wrote a quick test C client using libstomp rather than activemq-
> cpp and
> can code around this problem using a timeout and some sanity
> checking ...
> but it's a bit of work and I really don't want to have to re-write the
> application I've already completed using activemq-cpp.
>
> Thoughts?
>
> Thanks in advance,
> Brian Roach
>
> --
> View this message in context: http://www.nabble.com/activeMQ-5.1-%2B-activemq-cpp-2.2-openwire-transaction-problem-tp19018190p19018190.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>