You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "joakim@mnemonic.no" <jo...@mnemonic.no> on 2013/08/28 21:38:44 UTC
Clients not receiving ResourceAllocationException in transacted
sessions
Hi all.
I am fairly new to ActiveMQ, having used SwiftMQ for a long time.
However, I am stuck with a problem which I think may be a bug, unless I have
seriously misread some documentation... so any suggestions are welcome!
Short environment description:
Producers are sending messages to a topic, and a set of Durable Consumers
are subscribing to these messages. Both Producer and Consumer sessions are
transacted, and messages are persistent.
When a Consumer falls behind, or becomes offline, the producer should STOP
(when configured resource limits have been reached).
To configure this in ActiveMQ, I have configured systemUsage
sendFailIfNoSpace=true:
However, when the storeUsage has been reached, ActiveMQ log tells me that
the store is full, and the producer is being stopped, but the client does
not get any hint of this!
>From ActiveMQ log:
javax.jms.ResourceAllocationException: Persistent store is Full, 100% of
209715200. Stopping producer (ID:Calvin.local-52167-1377716876253-5:1:1:1)
to prevent flooding topic://nidstopic. See
http://activemq.apache.org/producer-flow-control.html for more info
However, the client just keeps sending messages, which are being discarded.
I have debugged the client code into ActiveMQ classes, and the client is
happily both sending the message, and commiting the transaction, without any
exception being thrown.
Changing to non-transacted session causes the producer to receive
ResourceAllocationException when the store is full.
I am running ActiveMQ 5.8.0 on Java 1.7 on Linux.
I have tried both with LevelDB and KahaDB, just in case.
Any suggestions?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Clients not receiving ResourceAllocationException in transacted
sessions
Posted by jvbrandis <jo...@mnemonic.no>.
In my opinion, it would be good form for the producer.send() to throw exception when the exception notification reaches the client (understanding that since this is async operation, the client may send several messages before the exception reaches the client).
However, as long as the exception is thrown calling session.commit(), then the client is able to recover correctly.
So yes, that would work.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4671054.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Clients not receiving ResourceAllocationException in transacted
sessions
Posted by HellKnight <he...@foxmail.com>.
use alwaysSyncSend has huge impacts on performance, but async sends could not
receive exceptions in the same thread you call producer.send(). Could the
client side receive jms.resourceAllocationException while calling
session.commit() ? Then async sends and receive exception in the same thread
could live together
--
View this message in context: http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4671050.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Clients not receiving ResourceAllocationException in transacted
sessions
Posted by jvbrandis <jo...@mnemonic.no>.
Yes, that does work, thanks. :)
However, without alwaysSyncSend=true, commit() is not performed synchronous (or at least, the call to commit() is completed without errors), so this is still a strange transactional contract.
Setting alwaysSyncSend=true seems to have had a huge performance impact on the application, but I'll see if tuning the configuration can solve this.
> No, use alwasySyncSend==true.
>
> Transactions are sent asynchronously as Tim mentioned, however the commit
> is always synchronous. You can force all sends within the TX to by sync
> with the above property. This would get you the sendFailIfNoSpace exception
> on the same thread.
>
>
> On Thu, Aug 29, 2013 at 5:50 AM, jvbrandis <[hidden email]> wrote:
>
> > >>
> > >> What's happening is that inside a transaction the normal behavior is to
> > >> send messages without waiting for a response from the broker (async)
> > >> which is why the only way the exception gets noticed is by an exception
> > >> listener. You can however configure this to work by using the option
> > >> 'setUseAsyncSend' on the ActiveMQConnectionFactory to force all send to
> > >> wait for a response which will then cause the exception to be thrown
> > >> from the producer send.
> > >>
> > >> You can set this on the connection URI via:
> > "tcp:xxx:x?useAsyncSend=false"
> > >
> > > Thanks!
> > >
> > > I still think that it is bad form that commit() is completed without
> > error when any (or all!) my messages are discarded.
> > > Even if the message send()-operation is async, the commit() should be
> > sync….
> > >
> > > Anyway, thank you for the configuration tip!
> >
> > A little update, setting jms.useAsyncSend=false did not help.
> > According to documentation, jms.useAsyncSend=false is the default.
> > Also, when not using sendFailIfNoSpace(AfterTimeout), the client is
> > blocked until space is available (as expected).
> > So this is a feature/bug only appearing when using sendFail…
> >
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670905.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
> http://www.christianposta.com/blog
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670907.html
> To unsubscribe from Clients not receiving ResourceAllocationException in transacted sessions, click here.
> NAML
--
View this message in context: http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670913.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Clients not receiving ResourceAllocationException in transacted sessions
Posted by Christian Posta <ch...@gmail.com>.
No, use alwasySyncSend==true.
Transactions are sent asynchronously as Tim mentioned, however the commit
is always synchronous. You can force all sends within the TX to by sync
with the above property. This would get you the sendFailIfNoSpace exception
on the same thread.
On Thu, Aug 29, 2013 at 5:50 AM, jvbrandis <jo...@mnemonic.no> wrote:
> >>
> >> What's happening is that inside a transaction the normal behavior is to
> >> send messages without waiting for a response from the broker (async)
> >> which is why the only way the exception gets noticed is by an exception
> >> listener. You can however configure this to work by using the option
> >> 'setUseAsyncSend' on the ActiveMQConnectionFactory to force all send to
> >> wait for a response which will then cause the exception to be thrown
> >> from the producer send.
> >>
> >> You can set this on the connection URI via:
> "tcp:xxx:x?useAsyncSend=false"
> >
> > Thanks!
> >
> > I still think that it is bad form that commit() is completed without
> error when any (or all!) my messages are discarded.
> > Even if the message send()-operation is async, the commit() should be
> sync….
> >
> > Anyway, thank you for the configuration tip!
>
> A little update, setting jms.useAsyncSend=false did not help.
> According to documentation, jms.useAsyncSend=false is the default.
> Also, when not using sendFailIfNoSpace(AfterTimeout), the client is
> blocked until space is available (as expected).
> So this is a feature/bug only appearing when using sendFail…
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670905.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Clients not receiving ResourceAllocationException in transacted
sessions
Posted by jvbrandis <jo...@mnemonic.no>.
>>
>> What's happening is that inside a transaction the normal behavior is to
>> send messages without waiting for a response from the broker (async)
>> which is why the only way the exception gets noticed is by an exception
>> listener. You can however configure this to work by using the option
>> 'setUseAsyncSend' on the ActiveMQConnectionFactory to force all send to
>> wait for a response which will then cause the exception to be thrown
>> from the producer send.
>>
>> You can set this on the connection URI via: "tcp:xxx:x?useAsyncSend=false"
>
> Thanks!
>
> I still think that it is bad form that commit() is completed without error when any (or all!) my messages are discarded.
> Even if the message send()-operation is async, the commit() should be sync….
>
> Anyway, thank you for the configuration tip!
A little update, setting jms.useAsyncSend=false did not help.
According to documentation, jms.useAsyncSend=false is the default.
Also, when not using sendFailIfNoSpace(AfterTimeout), the client is blocked until space is available (as expected).
So this is a feature/bug only appearing when using sendFail…
--
View this message in context: http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670905.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Clients not receiving ResourceAllocationException in transacted
sessions
Posted by jvbrandis <jo...@mnemonic.no>.
>
> What's happening is that inside a transaction the normal behavior is to
> send messages without waiting for a response from the broker (async)
> which is why the only way the exception gets noticed is by an exception
> listener. You can however configure this to work by using the option
> 'setUseAsyncSend' on the ActiveMQConnectionFactory to force all send to
> wait for a response which will then cause the exception to be thrown
> from the producer send.
>
> You can set this on the connection URI via: "tcp:xxx:x?useAsyncSend=false"
Thanks!
I still think that it is bad form that commit() is completed without error when any (or all!) my messages are discarded.
Even if the message send()-operation is async, the commit() should be sync….
Anyway, thank you for the configuration tip!
--
View this message in context: http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670904.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Clients not receiving ResourceAllocationException in transacted
sessions
Posted by Timothy Bish <ta...@gmail.com>.
On 08/29/2013 05:54 AM, jvbrandis wrote:
> Good suggestion!! :)
> Yes, an ExceptionListener on the Connection does get the exception when ResourceAllocationException is thrown.
>
> Of course, I could hack my way around it, notifying the client from the connection when the connection receives the error, but that seems backwards to me…. the error is sent in another thread, so I have no guarantee that the client is notified in time to avoid committing after an error occurs.
>
> (and honestly, returning from commit() without errors when all my messages have been discarded seems to be a strange transactional contract…. even if error notification is sent in another channel)
>
> I could not find an existing issue that describes this problem, so I'm going to post a bug report of the problem.
What's happening is that inside a transaction the normal behavior is to
send messages without waiting for a response from the broker (async)
which is why the only way the exception gets noticed is by an exception
listener. You can however configure this to work by using the option
'setUseAsyncSend' on the ActiveMQConnectionFactory to force all send to
wait for a response which will then cause the exception to be thrown
from the producer send.
You can set this on the connection URI via: "tcp:xxx:x?useAsyncSend=false"
> On 28. Aug2013, at 22:26, "ceposta [via ActiveMQ]" <ml...@n4.nabble.com> wrote:
>
>> Maybe set an exception listener on the connection and see if you get it
>> there?
>>
>> http://activemq.apache.org/maven/apidocs/org/apache/activemq/ActiveMQConnection.html#setExceptionListener(javax.jms.ExceptionListener)
>>
>>
>> On Wed, Aug 28, 2013 at 12:38 PM, [hidden email] <[hidden email]>wrote:
>>
>>> Hi all.
>>>
>>> I am fairly new to ActiveMQ, having used SwiftMQ for a long time.
>>> However, I am stuck with a problem which I think may be a bug, unless I
>>> have
>>> seriously misread some documentation... so any suggestions are welcome!
>>>
>>> Short environment description:
>>> Producers are sending messages to a topic, and a set of Durable Consumers
>>> are subscribing to these messages. Both Producer and Consumer sessions are
>>> transacted, and messages are persistent.
>>> When a Consumer falls behind, or becomes offline, the producer should STOP
>>> (when configured resource limits have been reached).
>>>
>>> To configure this in ActiveMQ, I have configured systemUsage
>>> sendFailIfNoSpace=true:
>>>
>>>
>>>
>>> However, when the storeUsage has been reached, ActiveMQ log tells me that
>>> the store is full, and the producer is being stopped, but the client does
>>> not get any hint of this!
>>>
>>> From ActiveMQ log:
>>> javax.jms.ResourceAllocationException: Persistent store is Full, 100% of
>>> 209715200. Stopping producer (ID:Calvin.local-52167-1377716876253-5:1:1:1)
>>> to prevent flooding topic://nidstopic. See
>>> http://activemq.apache.org/producer-flow-control.html for more info
>>>
>>> However, the client just keeps sending messages, which are being discarded.
>>> I have debugged the client code into ActiveMQ classes, and the client is
>>> happily both sending the message, and commiting the transaction, without
>>> any
>>> exception being thrown.
>>>
>>> Changing to non-transacted session causes the producer to receive
>>> ResourceAllocationException when the store is full.
>>>
>>> I am running ActiveMQ 5.8.0 on Java 1.7 on Linux.
>>> I have tried both with LevelDB and KahaDB, just in case.
>>>
>>> Any suggestions?
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>> --
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta
>> http://www.christianposta.com/blog
>>
>>
>> If you reply to this email, your message will be added to the discussion below:
>> http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670864.html
>> To unsubscribe from Clients not receiving ResourceAllocationException in transacted sessions, click here.
>> NAML
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670895.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/
Re: Clients not receiving ResourceAllocationException in transacted
sessions
Posted by jvbrandis <jo...@mnemonic.no>.
Good suggestion!! :)
Yes, an ExceptionListener on the Connection does get the exception when ResourceAllocationException is thrown.
Of course, I could hack my way around it, notifying the client from the connection when the connection receives the error, but that seems backwards to me…. the error is sent in another thread, so I have no guarantee that the client is notified in time to avoid committing after an error occurs.
(and honestly, returning from commit() without errors when all my messages have been discarded seems to be a strange transactional contract…. even if error notification is sent in another channel)
I could not find an existing issue that describes this problem, so I'm going to post a bug report of the problem.
On 28. Aug2013, at 22:26, "ceposta [via ActiveMQ]" <ml...@n4.nabble.com> wrote:
> Maybe set an exception listener on the connection and see if you get it
> there?
>
> http://activemq.apache.org/maven/apidocs/org/apache/activemq/ActiveMQConnection.html#setExceptionListener(javax.jms.ExceptionListener)
>
>
> On Wed, Aug 28, 2013 at 12:38 PM, [hidden email] <[hidden email]>wrote:
>
> > Hi all.
> >
> > I am fairly new to ActiveMQ, having used SwiftMQ for a long time.
> > However, I am stuck with a problem which I think may be a bug, unless I
> > have
> > seriously misread some documentation... so any suggestions are welcome!
> >
> > Short environment description:
> > Producers are sending messages to a topic, and a set of Durable Consumers
> > are subscribing to these messages. Both Producer and Consumer sessions are
> > transacted, and messages are persistent.
> > When a Consumer falls behind, or becomes offline, the producer should STOP
> > (when configured resource limits have been reached).
> >
> > To configure this in ActiveMQ, I have configured systemUsage
> > sendFailIfNoSpace=true:
> >
> >
> >
> > However, when the storeUsage has been reached, ActiveMQ log tells me that
> > the store is full, and the producer is being stopped, but the client does
> > not get any hint of this!
> >
> > From ActiveMQ log:
> > javax.jms.ResourceAllocationException: Persistent store is Full, 100% of
> > 209715200. Stopping producer (ID:Calvin.local-52167-1377716876253-5:1:1:1)
> > to prevent flooding topic://nidstopic. See
> > http://activemq.apache.org/producer-flow-control.html for more info
> >
> > However, the client just keeps sending messages, which are being discarded.
> > I have debugged the client code into ActiveMQ classes, and the client is
> > happily both sending the message, and commiting the transaction, without
> > any
> > exception being thrown.
> >
> > Changing to non-transacted session causes the producer to receive
> > ResourceAllocationException when the store is full.
> >
> > I am running ActiveMQ 5.8.0 on Java 1.7 on Linux.
> > I have tried both with LevelDB and KahaDB, just in case.
> >
> > Any suggestions?
> >
> >
> >
> > --
> > View this message in context:
> > http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
> http://www.christianposta.com/blog
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670864.html
> To unsubscribe from Clients not receiving ResourceAllocationException in transacted sessions, click here.
> NAML
--
View this message in context: http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862p4670895.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Clients not receiving ResourceAllocationException in transacted sessions
Posted by Christian Posta <ch...@gmail.com>.
Maybe set an exception listener on the connection and see if you get it
there?
http://activemq.apache.org/maven/apidocs/org/apache/activemq/ActiveMQConnection.html#setExceptionListener(javax.jms.ExceptionListener)
On Wed, Aug 28, 2013 at 12:38 PM, joakim@mnemonic.no <jo...@mnemonic.no>wrote:
> Hi all.
>
> I am fairly new to ActiveMQ, having used SwiftMQ for a long time.
> However, I am stuck with a problem which I think may be a bug, unless I
> have
> seriously misread some documentation... so any suggestions are welcome!
>
> Short environment description:
> Producers are sending messages to a topic, and a set of Durable Consumers
> are subscribing to these messages. Both Producer and Consumer sessions are
> transacted, and messages are persistent.
> When a Consumer falls behind, or becomes offline, the producer should STOP
> (when configured resource limits have been reached).
>
> To configure this in ActiveMQ, I have configured systemUsage
> sendFailIfNoSpace=true:
>
>
>
> However, when the storeUsage has been reached, ActiveMQ log tells me that
> the store is full, and the producer is being stopped, but the client does
> not get any hint of this!
>
> From ActiveMQ log:
> javax.jms.ResourceAllocationException: Persistent store is Full, 100% of
> 209715200. Stopping producer (ID:Calvin.local-52167-1377716876253-5:1:1:1)
> to prevent flooding topic://nidstopic. See
> http://activemq.apache.org/producer-flow-control.html for more info
>
> However, the client just keeps sending messages, which are being discarded.
> I have debugged the client code into ActiveMQ classes, and the client is
> happily both sending the message, and commiting the transaction, without
> any
> exception being thrown.
>
> Changing to non-transacted session causes the producer to receive
> ResourceAllocationException when the store is full.
>
> I am running ActiveMQ 5.8.0 on Java 1.7 on Linux.
> I have tried both with LevelDB and KahaDB, just in case.
>
> Any suggestions?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Clients-not-receiving-ResourceAllocationException-in-transacted-sessions-tp4670862.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta