You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Howard Gao <ho...@gmail.com> on 2018/10/16 03:19:15 UTC
[discussion] About blocking producers
Hi guys,
I'm working on this story and I'd like to hear your thoughts.
The Jira: https://issues.apache.org/jira/browse/ARTEMIS-2097
This is to provide a functionality of the broker to block on producers so
that user can consume all the existing messages (for maintaining their
servers for example), regardless of how the address setting's address full
policy is configured. The unblock
functionality should also be provided to complete this feature.
Here is my current implementation:
https://github.com/apache/activemq-artemis/pull/2371
The idea is to keep a list of blocked/unblock addresses set by the user.
Any incoming messages will be checked against the list and if its address
is blocked, it will be rejected and an exception will be thrown to the
clients. It works for all types of clients (core, openwire, jms, mqtt etc).
Any comments are welcome.
Thanks
Howard
Re: [discussion] About blocking producers
Posted by Howard Gao <ho...@gmail.com>.
This sounds like one way to do it. Most users need secured brokers and they
can adopt this method.
Thanks
On Wed, Oct 17, 2018 at 1:59 PM michael.andre.pearce
<mi...@me.com.invalid> wrote:
> Because our brokers are fully secured, if we need to stop incoming
> messages we simply remove the produce authorisation for users using
> security settings.
> This seems to work. Is this maybe an option for you?
>
>
> Sent from my Samsung Galaxy smartphone.
> -------- Original message --------From: Howard Gao <ho...@gmail.com>
> Date: 17/10/2018 04:02 (GMT+01:00) To: dev@activemq.apache.org Subject:
> Re: [discussion] About blocking producers
> Thanks Gary, you are quite right. This seems not a simple problem as I
> think is. The pause/resume is even harder to implement.
>
> And also the credit thing, some client support it (but may or may not use
> it), some are totally without it (like mqtt and stomp).
>
> Howard
>
> On Tue, Oct 16, 2018 at 6:57 PM Gary Tully <ga...@gmail.com> wrote:
>
> > There may be two different use cases here;
> > 1) the pause/resume use case, where it makes some sense to leave
> producers
> > active stop reading/deny credit/buffer etc; the corollary to the pause
> > consumers
> > 2) the take down for maintenance case, where it would be nice to drain
> the
> > broker.
> > In the past, users have tackled the "maintenance" use case with
> > separate acceptors, one for producers and one for consumers. with the
> > ability to disable an acceptor via a management operation. That is not
> > ideal b/c it requires cooperation with the applications to use the
> correct
> > acceptors etc.
> > However it is a coarse grained option that is a step to shutdown, a
> one
> > way step that won't be undone via resume. That can more easily be
> > implemented with a management op that kills off producers and blocks all
> > send operations with some deny interceptor (along the lines of the
> > authorisation interceptors). something like shutdownSenders
> >
> > If the use case is more like shutdownSenders keeping it coarse (on or off
> > for the entire server) will be best.
> >
> > For the pause/resume case it gets tricky;
> > - for example if the same connection is used for producing and
> consuming,
> > any exception can be fatal for the connection, causing retries etc and
> > looping.
> > - blocking will only work for sync send operations and async operations
> > may accumulate. It may require protocol specific smarts.
> > - same for of credit, only where flow control is respected by the
> client.
> >
> > I guess my point is that pause/resume may not be the way to go if the use
> > case is actually shut down for maintenance.
> >
> >
> > On Tue, 16 Oct 2018 at 04:19 Howard Gao <ho...@gmail.com> wrote:
> >
> > > Hi guys,
> > >
> > > I'm working on this story and I'd like to hear your thoughts.
> > >
> > > The Jira: https://issues.apache.org/jira/browse/ARTEMIS-2097
> > >
> > > This is to provide a functionality of the broker to block on producers
> so
> > > that user can consume all the existing messages (for maintaining their
> > > servers for example), regardless of how the address setting's address
> > full
> > > policy is configured. The unblock
> > > functionality should also be provided to complete this feature.
> > >
> > > Here is my current implementation:
> > > https://github.com/apache/activemq-artemis/pull/2371
> > >
> > > The idea is to keep a list of blocked/unblock addresses set by the
> user.
> > > Any incoming messages will be checked against the list and if its
> address
> > > is blocked, it will be rejected and an exception will be thrown to the
> > > clients. It works for all types of clients (core, openwire, jms, mqtt
> > > etc).
> > >
> > > Any comments are welcome.
> > >
> > > Thanks
> > > Howard
> > >
> >
>
Re: [discussion] About blocking producers
Posted by Clebert Suconic <cl...@gmail.com>.
Or simpler pause (block|fail) as an argument.
And resume.
Protocol specific flow control is already an issue and could be treated
differently.
This would be just an extension of what’s already implement and not very
extensive to insolent.
On Wed, Oct 17, 2018 at 9:05 AM Clebert Suconic <cl...@gmail.com>
wrote:
> I propose we add two separate options.
>
>
> - Pause flow control
>
> That is. We should treat the address as full and all credits wouldn’t be
> send until you resume.
>
> - deny producing
>
> The addrsss would be in fail mode until you hit resume.
>
>
> All semantics from fail and block flow control would apply as now.
>
>
>
> This is easy to implement and the user would select the appropriate
> operation as he wishes.
>
> On Wed, Oct 17, 2018 at 1:59 AM michael.andre.pearce
> <mi...@me.com.invalid> wrote:
>
>> Because our brokers are fully secured, if we need to stop incoming
>> messages we simply remove the produce authorisation for users using
>> security settings.
>> This seems to work. Is this maybe an option for you?
>>
>>
>> Sent from my Samsung Galaxy smartphone.
>> -------- Original message --------From: Howard Gao <ho...@gmail.com>
>> Date: 17/10/2018 04:02 (GMT+01:00) To: dev@activemq.apache.org
>> Subject: Re: [discussion] About blocking producers
>> Thanks Gary, you are quite right. This seems not a simple problem as I
>> think is. The pause/resume is even harder to implement.
>>
>> And also the credit thing, some client support it (but may or may not use
>> it), some are totally without it (like mqtt and stomp).
>>
>> Howard
>>
>> On Tue, Oct 16, 2018 at 6:57 PM Gary Tully <ga...@gmail.com> wrote:
>>
>> > There may be two different use cases here;
>> > 1) the pause/resume use case, where it makes some sense to leave
>> producers
>> > active stop reading/deny credit/buffer etc; the corollary to the pause
>> > consumers
>> > 2) the take down for maintenance case, where it would be nice to drain
>> the
>> > broker.
>> > In the past, users have tackled the "maintenance" use case with
>> > separate acceptors, one for producers and one for consumers. with the
>> > ability to disable an acceptor via a management operation. That is not
>> > ideal b/c it requires cooperation with the applications to use the
>> correct
>> > acceptors etc.
>> > However it is a coarse grained option that is a step to shutdown, a
>> one
>> > way step that won't be undone via resume. That can more easily be
>> > implemented with a management op that kills off producers and blocks all
>> > send operations with some deny interceptor (along the lines of the
>> > authorisation interceptors). something like shutdownSenders
>> >
>> > If the use case is more like shutdownSenders keeping it coarse (on or
>> off
>> > for the entire server) will be best.
>> >
>> > For the pause/resume case it gets tricky;
>> > - for example if the same connection is used for producing and
>> consuming,
>> > any exception can be fatal for the connection, causing retries etc and
>> > looping.
>> > - blocking will only work for sync send operations and async operations
>> > may accumulate. It may require protocol specific smarts.
>> > - same for of credit, only where flow control is respected by the
>> client.
>> >
>> > I guess my point is that pause/resume may not be the way to go if the
>> use
>> > case is actually shut down for maintenance.
>> >
>> >
>> > On Tue, 16 Oct 2018 at 04:19 Howard Gao <ho...@gmail.com> wrote:
>> >
>> > > Hi guys,
>> > >
>> > > I'm working on this story and I'd like to hear your thoughts.
>> > >
>> > > The Jira: https://issues.apache.org/jira/browse/ARTEMIS-2097
>> > >
>> > > This is to provide a functionality of the broker to block on
>> producers so
>> > > that user can consume all the existing messages (for maintaining their
>> > > servers for example), regardless of how the address setting's address
>> > full
>> > > policy is configured. The unblock
>> > > functionality should also be provided to complete this feature.
>> > >
>> > > Here is my current implementation:
>> > > https://github.com/apache/activemq-artemis/pull/2371
>> > >
>> > > The idea is to keep a list of blocked/unblock addresses set by the
>> user.
>> > > Any incoming messages will be checked against the list and if its
>> address
>> > > is blocked, it will be rejected and an exception will be thrown to the
>> > > clients. It works for all types of clients (core, openwire, jms, mqtt
>> > > etc).
>> > >
>> > > Any comments are welcome.
>> > >
>> > > Thanks
>> > > Howard
>> > >
>> >
>>
> --
> Clebert Suconic
>
--
Clebert Suconic
Re: [discussion] About blocking producers
Posted by Clebert Suconic <cl...@gmail.com>.
I propose we add two separate options.
- Pause flow control
That is. We should treat the address as full and all credits wouldn’t be
send until you resume.
- deny producing
The addrsss would be in fail mode until you hit resume.
All semantics from fail and block flow control would apply as now.
This is easy to implement and the user would select the appropriate
operation as he wishes.
On Wed, Oct 17, 2018 at 1:59 AM michael.andre.pearce
<mi...@me.com.invalid> wrote:
> Because our brokers are fully secured, if we need to stop incoming
> messages we simply remove the produce authorisation for users using
> security settings.
> This seems to work. Is this maybe an option for you?
>
>
> Sent from my Samsung Galaxy smartphone.
> -------- Original message --------From: Howard Gao <ho...@gmail.com>
> Date: 17/10/2018 04:02 (GMT+01:00) To: dev@activemq.apache.org Subject:
> Re: [discussion] About blocking producers
> Thanks Gary, you are quite right. This seems not a simple problem as I
> think is. The pause/resume is even harder to implement.
>
> And also the credit thing, some client support it (but may or may not use
> it), some are totally without it (like mqtt and stomp).
>
> Howard
>
> On Tue, Oct 16, 2018 at 6:57 PM Gary Tully <ga...@gmail.com> wrote:
>
> > There may be two different use cases here;
> > 1) the pause/resume use case, where it makes some sense to leave
> producers
> > active stop reading/deny credit/buffer etc; the corollary to the pause
> > consumers
> > 2) the take down for maintenance case, where it would be nice to drain
> the
> > broker.
> > In the past, users have tackled the "maintenance" use case with
> > separate acceptors, one for producers and one for consumers. with the
> > ability to disable an acceptor via a management operation. That is not
> > ideal b/c it requires cooperation with the applications to use the
> correct
> > acceptors etc.
> > However it is a coarse grained option that is a step to shutdown, a
> one
> > way step that won't be undone via resume. That can more easily be
> > implemented with a management op that kills off producers and blocks all
> > send operations with some deny interceptor (along the lines of the
> > authorisation interceptors). something like shutdownSenders
> >
> > If the use case is more like shutdownSenders keeping it coarse (on or off
> > for the entire server) will be best.
> >
> > For the pause/resume case it gets tricky;
> > - for example if the same connection is used for producing and
> consuming,
> > any exception can be fatal for the connection, causing retries etc and
> > looping.
> > - blocking will only work for sync send operations and async operations
> > may accumulate. It may require protocol specific smarts.
> > - same for of credit, only where flow control is respected by the
> client.
> >
> > I guess my point is that pause/resume may not be the way to go if the use
> > case is actually shut down for maintenance.
> >
> >
> > On Tue, 16 Oct 2018 at 04:19 Howard Gao <ho...@gmail.com> wrote:
> >
> > > Hi guys,
> > >
> > > I'm working on this story and I'd like to hear your thoughts.
> > >
> > > The Jira: https://issues.apache.org/jira/browse/ARTEMIS-2097
> > >
> > > This is to provide a functionality of the broker to block on producers
> so
> > > that user can consume all the existing messages (for maintaining their
> > > servers for example), regardless of how the address setting's address
> > full
> > > policy is configured. The unblock
> > > functionality should also be provided to complete this feature.
> > >
> > > Here is my current implementation:
> > > https://github.com/apache/activemq-artemis/pull/2371
> > >
> > > The idea is to keep a list of blocked/unblock addresses set by the
> user.
> > > Any incoming messages will be checked against the list and if its
> address
> > > is blocked, it will be rejected and an exception will be thrown to the
> > > clients. It works for all types of clients (core, openwire, jms, mqtt
> > > etc).
> > >
> > > Any comments are welcome.
> > >
> > > Thanks
> > > Howard
> > >
> >
>
--
Clebert Suconic
Re: [discussion] About blocking producers
Posted by Howard Gao <ho...@gmail.com>.
Thanks Gary, you are quite right. This seems not a simple problem as I
think is. The pause/resume is even harder to implement.
And also the credit thing, some client support it (but may or may not use
it), some are totally without it (like mqtt and stomp).
Howard
On Tue, Oct 16, 2018 at 6:57 PM Gary Tully <ga...@gmail.com> wrote:
> There may be two different use cases here;
> 1) the pause/resume use case, where it makes some sense to leave producers
> active stop reading/deny credit/buffer etc; the corollary to the pause
> consumers
> 2) the take down for maintenance case, where it would be nice to drain the
> broker.
> In the past, users have tackled the "maintenance" use case with
> separate acceptors, one for producers and one for consumers. with the
> ability to disable an acceptor via a management operation. That is not
> ideal b/c it requires cooperation with the applications to use the correct
> acceptors etc.
> However it is a coarse grained option that is a step to shutdown, a one
> way step that won't be undone via resume. That can more easily be
> implemented with a management op that kills off producers and blocks all
> send operations with some deny interceptor (along the lines of the
> authorisation interceptors). something like shutdownSenders
>
> If the use case is more like shutdownSenders keeping it coarse (on or off
> for the entire server) will be best.
>
> For the pause/resume case it gets tricky;
> - for example if the same connection is used for producing and consuming,
> any exception can be fatal for the connection, causing retries etc and
> looping.
> - blocking will only work for sync send operations and async operations
> may accumulate. It may require protocol specific smarts.
> - same for of credit, only where flow control is respected by the client.
>
> I guess my point is that pause/resume may not be the way to go if the use
> case is actually shut down for maintenance.
>
>
> On Tue, 16 Oct 2018 at 04:19 Howard Gao <ho...@gmail.com> wrote:
>
> > Hi guys,
> >
> > I'm working on this story and I'd like to hear your thoughts.
> >
> > The Jira: https://issues.apache.org/jira/browse/ARTEMIS-2097
> >
> > This is to provide a functionality of the broker to block on producers so
> > that user can consume all the existing messages (for maintaining their
> > servers for example), regardless of how the address setting's address
> full
> > policy is configured. The unblock
> > functionality should also be provided to complete this feature.
> >
> > Here is my current implementation:
> > https://github.com/apache/activemq-artemis/pull/2371
> >
> > The idea is to keep a list of blocked/unblock addresses set by the user.
> > Any incoming messages will be checked against the list and if its address
> > is blocked, it will be rejected and an exception will be thrown to the
> > clients. It works for all types of clients (core, openwire, jms, mqtt
> > etc).
> >
> > Any comments are welcome.
> >
> > Thanks
> > Howard
> >
>
Re: [discussion] About blocking producers
Posted by Gary Tully <ga...@gmail.com>.
There may be two different use cases here;
1) the pause/resume use case, where it makes some sense to leave producers
active stop reading/deny credit/buffer etc; the corollary to the pause
consumers
2) the take down for maintenance case, where it would be nice to drain the
broker.
In the past, users have tackled the "maintenance" use case with
separate acceptors, one for producers and one for consumers. with the
ability to disable an acceptor via a management operation. That is not
ideal b/c it requires cooperation with the applications to use the correct
acceptors etc.
However it is a coarse grained option that is a step to shutdown, a one
way step that won't be undone via resume. That can more easily be
implemented with a management op that kills off producers and blocks all
send operations with some deny interceptor (along the lines of the
authorisation interceptors). something like shutdownSenders
If the use case is more like shutdownSenders keeping it coarse (on or off
for the entire server) will be best.
For the pause/resume case it gets tricky;
- for example if the same connection is used for producing and consuming,
any exception can be fatal for the connection, causing retries etc and
looping.
- blocking will only work for sync send operations and async operations
may accumulate. It may require protocol specific smarts.
- same for of credit, only where flow control is respected by the client.
I guess my point is that pause/resume may not be the way to go if the use
case is actually shut down for maintenance.
On Tue, 16 Oct 2018 at 04:19 Howard Gao <ho...@gmail.com> wrote:
> Hi guys,
>
> I'm working on this story and I'd like to hear your thoughts.
>
> The Jira: https://issues.apache.org/jira/browse/ARTEMIS-2097
>
> This is to provide a functionality of the broker to block on producers so
> that user can consume all the existing messages (for maintaining their
> servers for example), regardless of how the address setting's address full
> policy is configured. The unblock
> functionality should also be provided to complete this feature.
>
> Here is my current implementation:
> https://github.com/apache/activemq-artemis/pull/2371
>
> The idea is to keep a list of blocked/unblock addresses set by the user.
> Any incoming messages will be checked against the list and if its address
> is blocked, it will be rejected and an exception will be thrown to the
> clients. It works for all types of clients (core, openwire, jms, mqtt
> etc).
>
> Any comments are welcome.
>
> Thanks
> Howard
>