Failover on no allocated disk space left?

[Byte Flinger <by...@gmail.com>]

Hi

I am trying to configure 2 AMQ instances so that if there are issues with
one, the client will failover to another one.

I am having an issue with this behaviour when the alloted disk space is out.

>From my tests, amq will block the thread forever when it runs "out of disk"
so I have configured the brokers with sendFailIfNoSpaceAfterTimeout
parameter however this just makes it so that the client returns an error
(even though it is configured with failover).

Is there any way to trigger the failover protocol in such a situation
rather than simply throwing the error back to the client?

Regards
ByteFlinger

Re: Failover on no allocated disk space left?

[Christian Posta <ch...@gmail.com>]

this seems like a good idea.

gtully: i've opened the issue here to be able to track it:
https://issues.apache.org/jira/browse/AMQ-4824


On Wed, Oct 23, 2013 at 2:49 PM, Gary Tully <ga...@gmail.com> wrote:

> it may make sense to have an option to determine the type of exception
> we throw when we are resource constrained.
> ResourceAllocation is a jms exception so it bubbles up to the app. But
> we could have an IO exception in that case
> that would trigger failover.
>
> If the persistence adapter hits a real IOException like a real no
> space error from the filesystem, the default
> IOExceptionHandler on the broker will stop the broker, but that can be
> configured to stop the transport connectors till space becomes
> available.
> In that case, failover will kick in on the clients. Maybe that is an
> option for you, don't use the store limit in this case, leave it to
> the OS limits.
> see: http://activemq.apache.org/configurable-ioexception-handling.html
>
> I think it is worth raising an enhancement for the sendFailIfNoSpace
> feature to allow an IOException to be raised
>
> On 23 October 2013 19:39, ByteFlinger <by...@gmail.com> wrote:
> > Thanks ceposta.
> >
> > I was hoping that there was either an official solution to the issue or
> that
> > it was a confirmed bug.
> >
> > I hope this gets reviewed in the future because failover behaviour
> should be
> > consistent even in such a case.
> >
> >
> >
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/Failover-on-no-allocated-disk-space-left-tp4672138p4673160.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> http://redhat.com
> http://blog.garytully.com
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Re: Failover on no allocated disk space left?

[Gary Tully <ga...@gmail.com>]

it may make sense to have an option to determine the type of exception
we throw when we are resource constrained.
ResourceAllocation is a jms exception so it bubbles up to the app. But
we could have an IO exception in that case
that would trigger failover.

If the persistence adapter hits a real IOException like a real no
space error from the filesystem, the default
IOExceptionHandler on the broker will stop the broker, but that can be
configured to stop the transport connectors till space becomes
available.
In that case, failover will kick in on the clients. Maybe that is an
option for you, don't use the store limit in this case, leave it to
the OS limits.
see: http://activemq.apache.org/configurable-ioexception-handling.html

I think it is worth raising an enhancement for the sendFailIfNoSpace
feature to allow an IOException to be raised

On 23 October 2013 19:39, ByteFlinger <by...@gmail.com> wrote:
> Thanks ceposta.
>
> I was hoping that there was either an official solution to the issue or that
> it was a confirmed bug.
>
> I hope this gets reviewed in the future because failover behaviour should be
> consistent even in such a case.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Failover-on-no-allocated-disk-space-left-tp4672138p4673160.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



-- 
http://redhat.com
http://blog.garytully.com

Re: Failover on no allocated disk space left?

[ByteFlinger <by...@gmail.com>]

Thanks ceposta.

I was hoping that there was either an official solution to the issue or that
it was a confirmed bug.

I hope this gets reviewed in the future because failover behaviour should be
consistent even in such a case.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Failover-on-no-allocated-disk-space-left-tp4672138p4673160.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Failover on no allocated disk space left?

[Christian Posta <ch...@gmail.com>]

You might be able to put together a broker filter/plugin that shuts down
the broker if the store becomes full. That might not be a good idea,
however.. what happens with messages that are already stored on the first
broker?

Maybe you can set an exceptionListener on the connection and for
ResourceAllocationExceptions, you can try to trick the FailoverTransport to
cycle to its next broker?


On Wed, Oct 2, 2013 at 5:05 AM, Byte Flinger <by...@gmail.com> wrote:

> Hi
>
> I am trying to configure 2 AMQ instances so that if there are issues with
> one, the client will failover to another one.
>
> I am having an issue with this behaviour when the alloted disk space is
> out.
>
> From my tests, amq will block the thread forever when it runs "out of disk"
> so I have configured the brokers with sendFailIfNoSpaceAfterTimeout
> parameter however this just makes it so that the client returns an error
> (even though it is configured with failover).
>
> Is there any way to trigger the failover protocol in such a situation
> rather than simply throwing the error back to the client?
>
> Regards
> ByteFlinger
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta