You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Haiting Jiang <ji...@apache.org> on 2022/04/09 08:24:34 UTC
[DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.
Hi Pulsar community,
I created a PIP to add support for subscription level dispatch rate limiter setting.
The proposal can be found: https://github.com/apache/pulsar/issues/15094
----
## Motivation
Currently, for message dispatch rate limiter in a subscription , we have 3 level setting :
- Broker level setting: configured with `dispatchThrottlingRatePerSubscriptionInMsg` and `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf
- Namespace level setting: configured with `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate`
- Topic level setting: configured with `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate`
As we all know, in the pub-sub messaging model, different subscriber of the same topic process the messages for various purpose, and they may have different requirement of message dispatch rate limiter. Here are some use case in my organization:
- On the client side, subscriptions have different max-process-capacity. If the dispatch rate is too large, they may crush their downstream services.
- We are billing base on the max message rate of the subscription. Some are sensitive to budgets and willing to pay less for lower throughput.
## Goal
Support subscription level dispatch rate limiter setting.
## API Changes
1. Add client api in org.apache.pulsar.client.admin.TopicPolicies.
```
void getSubscriptionDispatchRate(String topic, String sub) throws PulsarAdminException;
void getSubscriptionDispatchRate(String topic, String sub, boolean applied) throws PulsarAdminException;
void setSubscriptionDispatchRate(String topic, String sub, DispatchRate dispatchRate) throws PulsarAdminException;
void removeSubscriptionDispatchRate(String topic, String sub) throws PulsarAdminException;
//And the async version of these methods.
```
2. Add new admin API
```
@PUT @DELETE @GET
@Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate")
```
## Implementation
The rate limiter itself is already implemented with each subscription. We only need to update the rate limiter settings if subscription level config is set.
I propose to just add a new field in `org.apache.pulsar.common.policies.data.TopicPolicies` to store the data.
```
private Map<String/*SubName*/, DispatchRateImpl> subscriptionDispatchRateMap;
```
And subscription level rate limiter setting has higher priority than topic level. We need to calculate the applied value when we create the subscription or any level config is changed.
## Reject Alternatives
None yet.
Thanks,
Haiting
Re: [DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.
Posted by Haiting Jiang <ji...@apache.org>.
Hi Joe,
> The rate limits that are currently in place are there to protect the Pulsar
> service, to manage multi-tenancy on the broker. This is not meant as a
> feature to manage demand side throttling.
Yes, this is not only for demand side throttling. Consider the case of catch up consuming with
multiply subscriber. Different subscriber of the same topic can be considered as different
tenancies, and this subscription level dispatch rate limiter setting can server the purpose of
managing multi-tenancy on the broker.
For example, we have a topic with 1TB message data stored in Pulsar and two subscriber.
The resource consumption is quite different if we allow two subscriber throttling both at
1000MB/s or one is 1000MB/s and the other is 10MB/s.
> There is no need to add complexity
For this point, I have to clarify that the complexity is not significant. The subscription level
dispatch throttling is already implemented, only use the same value for all subscriptions of the
same topic.
Thanks,
Haiting
On 2022/04/12 17:58:43 Joe F wrote:
> -1
>
> The rate limits that are currently in place are there to protect the Pulsar
> service, to manage multi-tenancy on the broker. This is not meant as a
> feature to manage demand side throttling.
>
> In my opinion, this is best done as a client side feature. There is no
> need to add complexity to the broker to manage demand side throttling.
> Just throttle demand on the client side.
>
>
>
> -joe
>
> On Mon, Apr 11, 2022 at 7:54 PM Zixuan Liu <no...@gmail.com> wrote:
>
> > +1
> >
> > Zixuan
> >
> > mattison chao <ma...@gmail.com> 于2022年4月12日周二 09:24写道：
> >
> > > +1
> > >
> > > Looks like a very useful feature. Thank you.
> > >
> > > Best,
> > > Mattison
> > >
> > > On Mon, 11 Apr 2022 at 08:55, PengHui Li <pe...@apache.org> wrote:
> > >
> > > > +1
> > > >
> > > > Penghui
> > > >
> > > > On Sat, Apr 9, 2022 at 4:24 PM Haiting Jiang <ji...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Pulsar community,
> > > > >
> > > > > I created a PIP to add support for subscription level dispatch rate
> > > > > limiter setting.
> > > > >
> > > > > The proposal can be found:
> > > https://github.com/apache/pulsar/issues/15094
> > > > >
> > > > > ----
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > Currently, for message dispatch rate limiter in a subscription , we
> > > have
> > > > 3
> > > > > level setting :
> > > > > - Broker level setting: configured with
> > > > > `dispatchThrottlingRatePerSubscriptionInMsg` and
> > > > > `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf
> > > > > - Namespace level setting: configured with
> > > > >
> > `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate`
> > > > > - Topic level setting: configured with
> > > > >
> > > >
> > >
> > `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate`
> > > > >
> > > > > As we all know, in the pub-sub messaging model, different subscriber
> > of
> > > > > the same topic process the messages for various purpose, and they may
> > > > have
> > > > > different requirement of message dispatch rate limiter. Here are some
> > > use
> > > > > case in my organization:
> > > > > - On the client side, subscriptions have different
> > > max-process-capacity.
> > > > > If the dispatch rate is too large, they may crush their downstream
> > > > services.
> > > > > - We are billing base on the max message rate of the subscription.
> > Some
> > > > > are sensitive to budgets and willing to pay less for lower
> > throughput.
> > > > >
> > > > >
> > > > > ## Goal
> > > > >
> > > > > Support subscription level dispatch rate limiter setting.
> > > > >
> > > > > ## API Changes
> > > > >
> > > > >
> > > > > 1. Add client api in org.apache.pulsar.client.admin.TopicPolicies.
> > > > > ```
> > > > > void getSubscriptionDispatchRate(String topic, String sub) throws
> > > > > PulsarAdminException;
> > > > > void getSubscriptionDispatchRate(String topic, String sub, boolean
> > > > > applied) throws PulsarAdminException;
> > > > > void setSubscriptionDispatchRate(String topic, String sub,
> > DispatchRate
> > > > > dispatchRate) throws PulsarAdminException;
> > > > > void removeSubscriptionDispatchRate(String topic, String sub) throws
> > > > > PulsarAdminException;
> > > > >
> > > > > //And the async version of these methods.
> > > > >
> > > > > ```
> > > > >
> > > > > 2. Add new admin API
> > > > >
> > > > > ```
> > > > > @PUT @DELETE @GET
> > > > > @Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate")
> > > > >
> > > > > ```
> > > > >
> > > > > ## Implementation
> > > > >
> > > > > The rate limiter itself is already implemented with each
> > subscription.
> > > We
> > > > > only need to update the rate limiter settings if subscription level
> > > > config
> > > > > is set.
> > > > > I propose to just add a new field in
> > > > > `org.apache.pulsar.common.policies.data.TopicPolicies` to store the
> > > data.
> > > > > ```
> > > > > private Map<String/*SubName*/, DispatchRateImpl>
> > > > > subscriptionDispatchRateMap;
> > > > > ```
> > > > > And subscription level rate limiter setting has higher priority than
> > > > topic
> > > > > level. We need to calculate the applied value when we create the
> > > > > subscription or any level config is changed.
> > > > >
> > > > > ## Reject Alternatives
> > > > > None yet.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Haiting
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.
Posted by Haiting Jiang <ji...@apache.org>.
Hi Michael,
> To clarify, I think this proposal is not seeking to add support for a
> subscription rate limiter. That support was added in 2018 [0]. It is
> instead seeking to make it possible to override the broker's configs
> for dispatch rate limiting per subscription [1].
Yes, but not only broker level setting. As I mentioned in this PIP, we have
three level setting with priority: broker < namespace < topic. What I am
seeking is to add another subscription level setting.
> Out of my own curiosity, will you explain why the existing
> `receiverQueueSize` configuration is not sufficient for protecting
> against the kind of over consumption?
IMO, `receiverQueueSize` is used for improve consuming throughput.
For most online applications/services, the resource usage of mq client
itself is not significant, more is used in processing the message.
> I would have expected that
> config to provide the necessary back pressure to protect applications,
> and I wouldn't expect rate limiting the subscription to be the
> recommended way to protect consumers.
And protecting consumer is more like a side benefit with this feature.
It's more useful for server side protection and subscriber isolation.
Thanks,
Haiting
On 2022/04/12 20:02:23 Michael Marshall wrote:
> To clarify, I think this proposal is not seeking to add support for a
> subscription rate limiter. That support was added in 2018 [0]. It is
> instead seeking to make it possible to override the broker's configs
> for dispatch rate limiting per subscription [1].
>
> > On the client side, subscriptions have different max-process-capacity.
> > If the dispatch rate is too large, they may crush their downstream services.
>
> Out of my own curiosity, will you explain why the existing
> `receiverQueueSize` configuration is not sufficient for protecting
> against the kind of over consumption? I would have expected that
> config to provide the necessary back pressure to protect applications,
> and I wouldn't expect rate limiting the subscription to be the
> recommended way to protect consumers.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/pull/1358
> [1] https://github.com/apache/pulsar/blob/ec3821176612621e24dfbc4345525849a729fb06/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L881-L892
>
> On Tue, Apr 12, 2022 at 12:58 PM Joe F <jo...@apache.org> wrote:
> >
> > -1
> >
> > The rate limits that are currently in place are there to protect the Pulsar
> > service, to manage multi-tenancy on the broker. This is not meant as a
> > feature to manage demand side throttling.
> >
> > In my opinion, this is best done as a client side feature. There is no
> > need to add complexity to the broker to manage demand side throttling.
> > Just throttle demand on the client side.
> >
> >
> >
> > -joe
> >
> > On Mon, Apr 11, 2022 at 7:54 PM Zixuan Liu <no...@gmail.com> wrote:
> >
> > > +1
> > >
> > > Zixuan
> > >
> > > mattison chao <ma...@gmail.com> 于2022年4月12日周二 09:24写道：
> > >
> > > > +1
> > > >
> > > > Looks like a very useful feature. Thank you.
> > > >
> > > > Best,
> > > > Mattison
> > > >
> > > > On Mon, 11 Apr 2022 at 08:55, PengHui Li <pe...@apache.org> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Penghui
> > > > >
> > > > > On Sat, Apr 9, 2022 at 4:24 PM Haiting Jiang <ji...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Pulsar community,
> > > > > >
> > > > > > I created a PIP to add support for subscription level dispatch rate
> > > > > > limiter setting.
> > > > > >
> > > > > > The proposal can be found:
> > > > https://github.com/apache/pulsar/issues/15094
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > Currently, for message dispatch rate limiter in a subscription , we
> > > > have
> > > > > 3
> > > > > > level setting :
> > > > > > - Broker level setting: configured with
> > > > > > `dispatchThrottlingRatePerSubscriptionInMsg` and
> > > > > > `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf
> > > > > > - Namespace level setting: configured with
> > > > > >
> > > `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate`
> > > > > > - Topic level setting: configured with
> > > > > >
> > > > >
> > > >
> > > `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate`
> > > > > >
> > > > > > As we all know, in the pub-sub messaging model, different subscriber
> > > of
> > > > > > the same topic process the messages for various purpose, and they may
> > > > > have
> > > > > > different requirement of message dispatch rate limiter. Here are some
> > > > use
> > > > > > case in my organization:
> > > > > > - On the client side, subscriptions have different
> > > > max-process-capacity.
> > > > > > If the dispatch rate is too large, they may crush their downstream
> > > > > services.
> > > > > > - We are billing base on the max message rate of the subscription.
> > > Some
> > > > > > are sensitive to budgets and willing to pay less for lower
> > > throughput.
> > > > > >
> > > > > >
> > > > > > ## Goal
> > > > > >
> > > > > > Support subscription level dispatch rate limiter setting.
> > > > > >
> > > > > > ## API Changes
> > > > > >
> > > > > >
> > > > > > 1. Add client api in org.apache.pulsar.client.admin.TopicPolicies.
> > > > > > ```
> > > > > > void getSubscriptionDispatchRate(String topic, String sub) throws
> > > > > > PulsarAdminException;
> > > > > > void getSubscriptionDispatchRate(String topic, String sub, boolean
> > > > > > applied) throws PulsarAdminException;
> > > > > > void setSubscriptionDispatchRate(String topic, String sub,
> > > DispatchRate
> > > > > > dispatchRate) throws PulsarAdminException;
> > > > > > void removeSubscriptionDispatchRate(String topic, String sub) throws
> > > > > > PulsarAdminException;
> > > > > >
> > > > > > //And the async version of these methods.
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > 2. Add new admin API
> > > > > >
> > > > > > ```
> > > > > > @PUT @DELETE @GET
> > > > > > @Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate")
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > ## Implementation
> > > > > >
> > > > > > The rate limiter itself is already implemented with each
> > > subscription.
> > > > We
> > > > > > only need to update the rate limiter settings if subscription level
> > > > > config
> > > > > > is set.
> > > > > > I propose to just add a new field in
> > > > > > `org.apache.pulsar.common.policies.data.TopicPolicies` to store the
> > > > data.
> > > > > > ```
> > > > > > private Map<String/*SubName*/, DispatchRateImpl>
> > > > > > subscriptionDispatchRateMap;
> > > > > > ```
> > > > > > And subscription level rate limiter setting has higher priority than
> > > > > topic
> > > > > > level. We need to calculate the applied value when we create the
> > > > > > subscription or any level config is changed.
> > > > > >
> > > > > > ## Reject Alternatives
> > > > > > None yet.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Haiting
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.
Posted by Michael Marshall <mm...@apache.org>.
To clarify, I think this proposal is not seeking to add support for a
subscription rate limiter. That support was added in 2018 [0]. It is
instead seeking to make it possible to override the broker's configs
for dispatch rate limiting per subscription [1].
> On the client side, subscriptions have different max-process-capacity.
> If the dispatch rate is too large, they may crush their downstream services.
Out of my own curiosity, will you explain why the existing
`receiverQueueSize` configuration is not sufficient for protecting
against the kind of over consumption? I would have expected that
config to provide the necessary back pressure to protect applications,
and I wouldn't expect rate limiting the subscription to be the
recommended way to protect consumers.
Thanks,
Michael
[0] https://github.com/apache/pulsar/pull/1358
[1] https://github.com/apache/pulsar/blob/ec3821176612621e24dfbc4345525849a729fb06/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L881-L892
On Tue, Apr 12, 2022 at 12:58 PM Joe F <jo...@apache.org> wrote:
>
> -1
>
> The rate limits that are currently in place are there to protect the Pulsar
> service, to manage multi-tenancy on the broker. This is not meant as a
> feature to manage demand side throttling.
>
> In my opinion, this is best done as a client side feature. There is no
> need to add complexity to the broker to manage demand side throttling.
> Just throttle demand on the client side.
>
>
>
> -joe
>
> On Mon, Apr 11, 2022 at 7:54 PM Zixuan Liu <no...@gmail.com> wrote:
>
> > +1
> >
> > Zixuan
> >
> > mattison chao <ma...@gmail.com> 于2022年4月12日周二 09:24写道：
> >
> > > +1
> > >
> > > Looks like a very useful feature. Thank you.
> > >
> > > Best,
> > > Mattison
> > >
> > > On Mon, 11 Apr 2022 at 08:55, PengHui Li <pe...@apache.org> wrote:
> > >
> > > > +1
> > > >
> > > > Penghui
> > > >
> > > > On Sat, Apr 9, 2022 at 4:24 PM Haiting Jiang <ji...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Pulsar community,
> > > > >
> > > > > I created a PIP to add support for subscription level dispatch rate
> > > > > limiter setting.
> > > > >
> > > > > The proposal can be found:
> > > https://github.com/apache/pulsar/issues/15094
> > > > >
> > > > > ----
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > Currently, for message dispatch rate limiter in a subscription , we
> > > have
> > > > 3
> > > > > level setting :
> > > > > - Broker level setting: configured with
> > > > > `dispatchThrottlingRatePerSubscriptionInMsg` and
> > > > > `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf
> > > > > - Namespace level setting: configured with
> > > > >
> > `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate`
> > > > > - Topic level setting: configured with
> > > > >
> > > >
> > >
> > `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate`
> > > > >
> > > > > As we all know, in the pub-sub messaging model, different subscriber
> > of
> > > > > the same topic process the messages for various purpose, and they may
> > > > have
> > > > > different requirement of message dispatch rate limiter. Here are some
> > > use
> > > > > case in my organization:
> > > > > - On the client side, subscriptions have different
> > > max-process-capacity.
> > > > > If the dispatch rate is too large, they may crush their downstream
> > > > services.
> > > > > - We are billing base on the max message rate of the subscription.
> > Some
> > > > > are sensitive to budgets and willing to pay less for lower
> > throughput.
> > > > >
> > > > >
> > > > > ## Goal
> > > > >
> > > > > Support subscription level dispatch rate limiter setting.
> > > > >
> > > > > ## API Changes
> > > > >
> > > > >
> > > > > 1. Add client api in org.apache.pulsar.client.admin.TopicPolicies.
> > > > > ```
> > > > > void getSubscriptionDispatchRate(String topic, String sub) throws
> > > > > PulsarAdminException;
> > > > > void getSubscriptionDispatchRate(String topic, String sub, boolean
> > > > > applied) throws PulsarAdminException;
> > > > > void setSubscriptionDispatchRate(String topic, String sub,
> > DispatchRate
> > > > > dispatchRate) throws PulsarAdminException;
> > > > > void removeSubscriptionDispatchRate(String topic, String sub) throws
> > > > > PulsarAdminException;
> > > > >
> > > > > //And the async version of these methods.
> > > > >
> > > > > ```
> > > > >
> > > > > 2. Add new admin API
> > > > >
> > > > > ```
> > > > > @PUT @DELETE @GET
> > > > > @Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate")
> > > > >
> > > > > ```
> > > > >
> > > > > ## Implementation
> > > > >
> > > > > The rate limiter itself is already implemented with each
> > subscription.
> > > We
> > > > > only need to update the rate limiter settings if subscription level
> > > > config
> > > > > is set.
> > > > > I propose to just add a new field in
> > > > > `org.apache.pulsar.common.policies.data.TopicPolicies` to store the
> > > data.
> > > > > ```
> > > > > private Map<String/*SubName*/, DispatchRateImpl>
> > > > > subscriptionDispatchRateMap;
> > > > > ```
> > > > > And subscription level rate limiter setting has higher priority than
> > > > topic
> > > > > level. We need to calculate the applied value when we create the
> > > > > subscription or any level config is changed.
> > > > >
> > > > > ## Reject Alternatives
> > > > > None yet.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Haiting
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.
Posted by Joe F <jo...@apache.org>.
-1
The rate limits that are currently in place are there to protect the Pulsar
service, to manage multi-tenancy on the broker. This is not meant as a
feature to manage demand side throttling.
In my opinion, this is best done as a client side feature. There is no
need to add complexity to the broker to manage demand side throttling.
Just throttle demand on the client side.
-joe
On Mon, Apr 11, 2022 at 7:54 PM Zixuan Liu <no...@gmail.com> wrote:
> +1
>
> Zixuan
>
> mattison chao <ma...@gmail.com> 于2022年4月12日周二 09:24写道：
>
> > +1
> >
> > Looks like a very useful feature. Thank you.
> >
> > Best,
> > Mattison
> >
> > On Mon, 11 Apr 2022 at 08:55, PengHui Li <pe...@apache.org> wrote:
> >
> > > +1
> > >
> > > Penghui
> > >
> > > On Sat, Apr 9, 2022 at 4:24 PM Haiting Jiang <ji...@apache.org>
> > > wrote:
> > >
> > > > Hi Pulsar community,
> > > >
> > > > I created a PIP to add support for subscription level dispatch rate
> > > > limiter setting.
> > > >
> > > > The proposal can be found:
> > https://github.com/apache/pulsar/issues/15094
> > > >
> > > > ----
> > > >
> > > > ## Motivation
> > > >
> > > > Currently, for message dispatch rate limiter in a subscription , we
> > have
> > > 3
> > > > level setting :
> > > > - Broker level setting: configured with
> > > > `dispatchThrottlingRatePerSubscriptionInMsg` and
> > > > `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf
> > > > - Namespace level setting: configured with
> > > >
> `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate`
> > > > - Topic level setting: configured with
> > > >
> > >
> >
> `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate`
> > > >
> > > > As we all know, in the pub-sub messaging model, different subscriber
> of
> > > > the same topic process the messages for various purpose, and they may
> > > have
> > > > different requirement of message dispatch rate limiter. Here are some
> > use
> > > > case in my organization:
> > > > - On the client side, subscriptions have different
> > max-process-capacity.
> > > > If the dispatch rate is too large, they may crush their downstream
> > > services.
> > > > - We are billing base on the max message rate of the subscription.
> Some
> > > > are sensitive to budgets and willing to pay less for lower
> throughput.
> > > >
> > > >
> > > > ## Goal
> > > >
> > > > Support subscription level dispatch rate limiter setting.
> > > >
> > > > ## API Changes
> > > >
> > > >
> > > > 1. Add client api in org.apache.pulsar.client.admin.TopicPolicies.
> > > > ```
> > > > void getSubscriptionDispatchRate(String topic, String sub) throws
> > > > PulsarAdminException;
> > > > void getSubscriptionDispatchRate(String topic, String sub, boolean
> > > > applied) throws PulsarAdminException;
> > > > void setSubscriptionDispatchRate(String topic, String sub,
> DispatchRate
> > > > dispatchRate) throws PulsarAdminException;
> > > > void removeSubscriptionDispatchRate(String topic, String sub) throws
> > > > PulsarAdminException;
> > > >
> > > > //And the async version of these methods.
> > > >
> > > > ```
> > > >
> > > > 2. Add new admin API
> > > >
> > > > ```
> > > > @PUT @DELETE @GET
> > > > @Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate")
> > > >
> > > > ```
> > > >
> > > > ## Implementation
> > > >
> > > > The rate limiter itself is already implemented with each
> subscription.
> > We
> > > > only need to update the rate limiter settings if subscription level
> > > config
> > > > is set.
> > > > I propose to just add a new field in
> > > > `org.apache.pulsar.common.policies.data.TopicPolicies` to store the
> > data.
> > > > ```
> > > > private Map<String/*SubName*/, DispatchRateImpl>
> > > > subscriptionDispatchRateMap;
> > > > ```
> > > > And subscription level rate limiter setting has higher priority than
> > > topic
> > > > level. We need to calculate the applied value when we create the
> > > > subscription or any level config is changed.
> > > >
> > > > ## Reject Alternatives
> > > > None yet.
> > > >
> > > >
> > > > Thanks,
> > > > Haiting
> > > >
> > >
> >
>
Re: [DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.
Posted by Zixuan Liu <no...@gmail.com>.
+1
Zixuan
mattison chao <ma...@gmail.com> 于2022年4月12日周二 09:24写道：
> +1
>
> Looks like a very useful feature. Thank you.
>
> Best,
> Mattison
>
> On Mon, 11 Apr 2022 at 08:55, PengHui Li <pe...@apache.org> wrote:
>
> > +1
> >
> > Penghui
> >
> > On Sat, Apr 9, 2022 at 4:24 PM Haiting Jiang <ji...@apache.org>
> > wrote:
> >
> > > Hi Pulsar community,
> > >
> > > I created a PIP to add support for subscription level dispatch rate
> > > limiter setting.
> > >
> > > The proposal can be found:
> https://github.com/apache/pulsar/issues/15094
> > >
> > > ----
> > >
> > > ## Motivation
> > >
> > > Currently, for message dispatch rate limiter in a subscription , we
> have
> > 3
> > > level setting :
> > > - Broker level setting: configured with
> > > `dispatchThrottlingRatePerSubscriptionInMsg` and
> > > `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf
> > > - Namespace level setting: configured with
> > > `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate`
> > > - Topic level setting: configured with
> > >
> >
> `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate`
> > >
> > > As we all know, in the pub-sub messaging model, different subscriber of
> > > the same topic process the messages for various purpose, and they may
> > have
> > > different requirement of message dispatch rate limiter. Here are some
> use
> > > case in my organization:
> > > - On the client side, subscriptions have different
> max-process-capacity.
> > > If the dispatch rate is too large, they may crush their downstream
> > services.
> > > - We are billing base on the max message rate of the subscription. Some
> > > are sensitive to budgets and willing to pay less for lower throughput.
> > >
> > >
> > > ## Goal
> > >
> > > Support subscription level dispatch rate limiter setting.
> > >
> > > ## API Changes
> > >
> > >
> > > 1. Add client api in org.apache.pulsar.client.admin.TopicPolicies.
> > > ```
> > > void getSubscriptionDispatchRate(String topic, String sub) throws
> > > PulsarAdminException;
> > > void getSubscriptionDispatchRate(String topic, String sub, boolean
> > > applied) throws PulsarAdminException;
> > > void setSubscriptionDispatchRate(String topic, String sub, DispatchRate
> > > dispatchRate) throws PulsarAdminException;
> > > void removeSubscriptionDispatchRate(String topic, String sub) throws
> > > PulsarAdminException;
> > >
> > > //And the async version of these methods.
> > >
> > > ```
> > >
> > > 2. Add new admin API
> > >
> > > ```
> > > @PUT @DELETE @GET
> > > @Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate")
> > >
> > > ```
> > >
> > > ## Implementation
> > >
> > > The rate limiter itself is already implemented with each subscription.
> We
> > > only need to update the rate limiter settings if subscription level
> > config
> > > is set.
> > > I propose to just add a new field in
> > > `org.apache.pulsar.common.policies.data.TopicPolicies` to store the
> data.
> > > ```
> > > private Map<String/*SubName*/, DispatchRateImpl>
> > > subscriptionDispatchRateMap;
> > > ```
> > > And subscription level rate limiter setting has higher priority than
> > topic
> > > level. We need to calculate the applied value when we create the
> > > subscription or any level config is changed.
> > >
> > > ## Reject Alternatives
> > > None yet.
> > >
> > >
> > > Thanks,
> > > Haiting
> > >
> >
>
Re: [DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.
Posted by mattison chao <ma...@gmail.com>.
+1
Looks like a very useful feature. Thank you.
Best,
Mattison
On Mon, 11 Apr 2022 at 08:55, PengHui Li <pe...@apache.org> wrote:
> +1
>
> Penghui
>
> On Sat, Apr 9, 2022 at 4:24 PM Haiting Jiang <ji...@apache.org>
> wrote:
>
> > Hi Pulsar community,
> >
> > I created a PIP to add support for subscription level dispatch rate
> > limiter setting.
> >
> > The proposal can be found: https://github.com/apache/pulsar/issues/15094
> >
> > ----
> >
> > ## Motivation
> >
> > Currently, for message dispatch rate limiter in a subscription , we have
> 3
> > level setting :
> > - Broker level setting: configured with
> > `dispatchThrottlingRatePerSubscriptionInMsg` and
> > `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf
> > - Namespace level setting: configured with
> > `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate`
> > - Topic level setting: configured with
> >
> `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate`
> >
> > As we all know, in the pub-sub messaging model, different subscriber of
> > the same topic process the messages for various purpose, and they may
> have
> > different requirement of message dispatch rate limiter. Here are some use
> > case in my organization:
> > - On the client side, subscriptions have different max-process-capacity.
> > If the dispatch rate is too large, they may crush their downstream
> services.
> > - We are billing base on the max message rate of the subscription. Some
> > are sensitive to budgets and willing to pay less for lower throughput.
> >
> >
> > ## Goal
> >
> > Support subscription level dispatch rate limiter setting.
> >
> > ## API Changes
> >
> >
> > 1. Add client api in org.apache.pulsar.client.admin.TopicPolicies.
> > ```
> > void getSubscriptionDispatchRate(String topic, String sub) throws
> > PulsarAdminException;
> > void getSubscriptionDispatchRate(String topic, String sub, boolean
> > applied) throws PulsarAdminException;
> > void setSubscriptionDispatchRate(String topic, String sub, DispatchRate
> > dispatchRate) throws PulsarAdminException;
> > void removeSubscriptionDispatchRate(String topic, String sub) throws
> > PulsarAdminException;
> >
> > //And the async version of these methods.
> >
> > ```
> >
> > 2. Add new admin API
> >
> > ```
> > @PUT @DELETE @GET
> > @Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate")
> >
> > ```
> >
> > ## Implementation
> >
> > The rate limiter itself is already implemented with each subscription. We
> > only need to update the rate limiter settings if subscription level
> config
> > is set.
> > I propose to just add a new field in
> > `org.apache.pulsar.common.policies.data.TopicPolicies` to store the data.
> > ```
> > private Map<String/*SubName*/, DispatchRateImpl>
> > subscriptionDispatchRateMap;
> > ```
> > And subscription level rate limiter setting has higher priority than
> topic
> > level. We need to calculate the applied value when we create the
> > subscription or any level config is changed.
> >
> > ## Reject Alternatives
> > None yet.
> >
> >
> > Thanks,
> > Haiting
> >
>
Re: [DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.
Posted by PengHui Li <pe...@apache.org>.
+1
Penghui
On Sat, Apr 9, 2022 at 4:24 PM Haiting Jiang <ji...@apache.org>
wrote:
> Hi Pulsar community,
>
> I created a PIP to add support for subscription level dispatch rate
> limiter setting.
>
> The proposal can be found: https://github.com/apache/pulsar/issues/15094
>
> ----
>
> ## Motivation
>
> Currently, for message dispatch rate limiter in a subscription , we have 3
> level setting :
> - Broker level setting: configured with
> `dispatchThrottlingRatePerSubscriptionInMsg` and
> `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf
> - Namespace level setting: configured with
> `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate`
> - Topic level setting: configured with
> `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate`
>
> As we all know, in the pub-sub messaging model, different subscriber of
> the same topic process the messages for various purpose, and they may have
> different requirement of message dispatch rate limiter. Here are some use
> case in my organization:
> - On the client side, subscriptions have different max-process-capacity.
> If the dispatch rate is too large, they may crush their downstream services.
> - We are billing base on the max message rate of the subscription. Some
> are sensitive to budgets and willing to pay less for lower throughput.
>
>
> ## Goal
>
> Support subscription level dispatch rate limiter setting.
>
> ## API Changes
>
>
> 1. Add client api in org.apache.pulsar.client.admin.TopicPolicies.
> ```
> void getSubscriptionDispatchRate(String topic, String sub) throws
> PulsarAdminException;
> void getSubscriptionDispatchRate(String topic, String sub, boolean
> applied) throws PulsarAdminException;
> void setSubscriptionDispatchRate(String topic, String sub, DispatchRate
> dispatchRate) throws PulsarAdminException;
> void removeSubscriptionDispatchRate(String topic, String sub) throws
> PulsarAdminException;
>
> //And the async version of these methods.
>
> ```
>
> 2. Add new admin API
>
> ```
> @PUT @DELETE @GET
> @Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate")
>
> ```
>
> ## Implementation
>
> The rate limiter itself is already implemented with each subscription. We
> only need to update the rate limiter settings if subscription level config
> is set.
> I propose to just add a new field in
> `org.apache.pulsar.common.policies.data.TopicPolicies` to store the data.
> ```
> private Map<String/*SubName*/, DispatchRateImpl>
> subscriptionDispatchRateMap;
> ```
> And subscription level rate limiter setting has higher priority than topic
> level. We need to calculate the applied value when we create the
> subscription or any level config is changed.
>
> ## Reject Alternatives
> None yet.
>
>
> Thanks,
> Haiting
>