You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Bilgin Ibryam <bi...@gmail.com> on 2016/01/30 19:15:13 UTC
Consumer throttling
Hi all,
Apologoes if this is a stupid question. But I wonder is there a way to
do throttling on the consumer side. Let me clarify what I mean.
PFC can do throttling for the producer, but I need a similar behaviour
on the consumer. For example I want to configure a message consure or
message queue so that messages are consumed/released one per second.
Usually the messages are consumed as fast as the cosnumers can take.
Then I can do some throttling in consumer Camel route, but that
throutling in Camel cannot be clustered accross instances. If the
throttling can be configured ot queue level, then I don't have to do
consumer throttling.
Thanks,
--
Bilgin Ibryam
Camel Committer at ASF & Integration Architect at Red Hat
Blog: http://ofbizian.com | Twitter: @bibryam
Camel Design Patterns https://leanpub.com/camel-design-patterns
Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
Re: Consumer throttling
Posted by Christopher Shannon <ch...@gmail.com>.
The only thing built in would be the scheduling feature. Take a look at
http://activemq.apache.org/delay-and-schedule-message-delivery.html
This allows you to delay a message or schedule so this may work for you.
Otherwise I think you will have to continue using client logic if you want
to throttle your consumers.
On Sat, Jan 30, 2016 at 1:15 PM, Bilgin Ibryam <bi...@gmail.com> wrote:
> Hi all,
>
> Apologoes if this is a stupid question. But I wonder is there a way to
> do throttling on the consumer side. Let me clarify what I mean.
>
> PFC can do throttling for the producer, but I need a similar behaviour
> on the consumer. For example I want to configure a message consure or
> message queue so that messages are consumed/released one per second.
> Usually the messages are consumed as fast as the cosnumers can take.
> Then I can do some throttling in consumer Camel route, but that
> throutling in Camel cannot be clustered accross instances. If the
> throttling can be configured ot queue level, then I don't have to do
> consumer throttling.
>
> Thanks,
>
> --
> Bilgin Ibryam
> Camel Committer at ASF & Integration Architect at Red Hat
> Blog: http://ofbizian.com | Twitter: @bibryam
>
> Camel Design Patterns https://leanpub.com/camel-design-patterns
> Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
>
Re: Consumer throttling
Posted by Tim Bain <tb...@alumni.duke.edu>.
The documentation you linked to sounds like Artemis allows rate-limiting
consumers individually but not rate-limiting an entire pool of consumers.
I was under the impression that Bilgin had more than one consumer and
wanted the ability to limit them as a group, but it wasn't entirely clear
so maybe I read the wrong things between the lines.
Tim
On Feb 2, 2016 6:59 AM, "Justin Bertram" <jb...@apache.com> wrote:
> For what it's worth, I believe the ActiveMQ Artemis broker (a sub-project
> of ActiveMQ) has the feature you're looking for. See more details in the
> documentation [1].
>
>
> Justin
>
> [1] http://activemq.apache.org/artemis/docs/1.2.0/flow-control.html
>
> ----- Original Message -----
> From: "Bilgin Ibryam" <bi...@gmail.com>
> To: users@activemq.apache.org
> Sent: Tuesday, February 2, 2016 5:54:28 AM
> Subject: Re: Consumer throttling
>
> Thanks all for your inputs. It seems there is no solution for my
> question in ActiveMQ yet.
>
> @Christopher: I'm aware of scheduling feature, but that is more of a
> Delayer rather than Throttler. It can delay messages, but cannot
> ensure throttled consumption. If I get a burst of messages delay them,
> at a later point in time there still will be a burst of messages.
> Nothing has improved.
>
> @Tim: I have described your suggestion with two queues and Camel in
> between in as a temporary solution in my recent Camel Design Patterns
> book, but that solution is not ideal either. It is still possible to
> have messages accumulate in queueu B (for example if the consumers are
> stopped temporarily) and then have non throttled consumption breaking
> SLAs, etc.
>
> @Rau I thoughtl PFC should take care of the message backlog and
> protect the broker regardless of the reason for message accumulation,
> whether that is slow consumers, throttled consumption or consumers
> being down for some reason. Having prefetch limit of 1 is good
> suggestion though, thanks.
>
>
>
> On 30 January 2016 at 21:30, Tim Bain <tb...@alumni.duke.edu> wrote:
> > You could achieve your goal by having two queues, A and B. Have
> producers
> > publish to A and consumers consume from B, and use an embedded Camel
> route
> > to move messages from A to B at the desired rate.
> >
> > But Raul's right: if you're going to do this, make sure there's some kind
> > of safety feature to prevent messages from backlogging on the broker.
> > Maybe you get it for free based on the characteristics of your clients,
> or
> > maybe that's some mechanism to discard built-up messages (based on time,
> > count, bytes, etc.), or a way to detect a backlog and temporarily speed
> up
> > the Camel route or to let consumers consume directly from A for a while,
> > or...
> >
> > Tim
> > On Jan 30, 2016 12:56 PM, "Raul Kripalani" <ra...@apache.org> wrote:
> >
> >> Just one remark. This could be dangerous as it could lead to an
> >> ever-growing backlog of messages in AMQ. I suppose you have already
> >> considered it, and either the producers are slower than the consumers,
> or
> >> you have multiple competing consumers, or there is a period in the day
> >> where the producers stop producing to give the consumers time to consume
> >> the existing backlog or something.
> >>
> >> If you implement this, I would reduce the prefect buffer to 1 if you
> have
> >> multiple competing consumers to achieve decent distribution and avoid
> >> starvation.
> >>
> >> Raúl.
> >> On 30 Jan 2016 18:15, "Bilgin Ibryam" <bi...@gmail.com> wrote:
> >>
> >> > Hi all,
> >> >
> >> > Apologoes if this is a stupid question. But I wonder is there a way to
> >> > do throttling on the consumer side. Let me clarify what I mean.
> >> >
> >> > PFC can do throttling for the producer, but I need a similar behaviour
> >> > on the consumer. For example I want to configure a message consure or
> >> > message queue so that messages are consumed/released one per second.
> >> > Usually the messages are consumed as fast as the cosnumers can take.
> >> > Then I can do some throttling in consumer Camel route, but that
> >> > throutling in Camel cannot be clustered accross instances. If the
> >> > throttling can be configured ot queue level, then I don't have to do
> >> > consumer throttling.
> >> >
> >> > Thanks,
> >> >
> >> > --
> >> > Bilgin Ibryam
> >> > Camel Committer at ASF & Integration Architect at Red Hat
> >> > Blog: http://ofbizian.com | Twitter: @bibryam
> >> >
> >> > Camel Design Patterns https://leanpub.com/camel-design-patterns
> >> > Instant Apache Camel Message Routing
> http://www.amazon.com/dp/1783283475
> >> >
> >>
>
>
>
> --
> Bilgin Ibryam
> Camel Committer at ASF & Integration Architect at Red Hat
> Blog: http://ofbizian.com | Twitter: @bibryam
>
> Camel Design Patterns https://leanpub.com/camel-design-patterns
> Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
>
Re: Consumer throttling
Posted by Justin Bertram <jb...@apache.com>.
For what it's worth, I believe the ActiveMQ Artemis broker (a sub-project of ActiveMQ) has the feature you're looking for. See more details in the documentation [1].
Justin
[1] http://activemq.apache.org/artemis/docs/1.2.0/flow-control.html
----- Original Message -----
From: "Bilgin Ibryam" <bi...@gmail.com>
To: users@activemq.apache.org
Sent: Tuesday, February 2, 2016 5:54:28 AM
Subject: Re: Consumer throttling
Thanks all for your inputs. It seems there is no solution for my
question in ActiveMQ yet.
@Christopher: I'm aware of scheduling feature, but that is more of a
Delayer rather than Throttler. It can delay messages, but cannot
ensure throttled consumption. If I get a burst of messages delay them,
at a later point in time there still will be a burst of messages.
Nothing has improved.
@Tim: I have described your suggestion with two queues and Camel in
between in as a temporary solution in my recent Camel Design Patterns
book, but that solution is not ideal either. It is still possible to
have messages accumulate in queueu B (for example if the consumers are
stopped temporarily) and then have non throttled consumption breaking
SLAs, etc.
@Rau I thoughtl PFC should take care of the message backlog and
protect the broker regardless of the reason for message accumulation,
whether that is slow consumers, throttled consumption or consumers
being down for some reason. Having prefetch limit of 1 is good
suggestion though, thanks.
On 30 January 2016 at 21:30, Tim Bain <tb...@alumni.duke.edu> wrote:
> You could achieve your goal by having two queues, A and B. Have producers
> publish to A and consumers consume from B, and use an embedded Camel route
> to move messages from A to B at the desired rate.
>
> But Raul's right: if you're going to do this, make sure there's some kind
> of safety feature to prevent messages from backlogging on the broker.
> Maybe you get it for free based on the characteristics of your clients, or
> maybe that's some mechanism to discard built-up messages (based on time,
> count, bytes, etc.), or a way to detect a backlog and temporarily speed up
> the Camel route or to let consumers consume directly from A for a while,
> or...
>
> Tim
> On Jan 30, 2016 12:56 PM, "Raul Kripalani" <ra...@apache.org> wrote:
>
>> Just one remark. This could be dangerous as it could lead to an
>> ever-growing backlog of messages in AMQ. I suppose you have already
>> considered it, and either the producers are slower than the consumers, or
>> you have multiple competing consumers, or there is a period in the day
>> where the producers stop producing to give the consumers time to consume
>> the existing backlog or something.
>>
>> If you implement this, I would reduce the prefect buffer to 1 if you have
>> multiple competing consumers to achieve decent distribution and avoid
>> starvation.
>>
>> Raúl.
>> On 30 Jan 2016 18:15, "Bilgin Ibryam" <bi...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > Apologoes if this is a stupid question. But I wonder is there a way to
>> > do throttling on the consumer side. Let me clarify what I mean.
>> >
>> > PFC can do throttling for the producer, but I need a similar behaviour
>> > on the consumer. For example I want to configure a message consure or
>> > message queue so that messages are consumed/released one per second.
>> > Usually the messages are consumed as fast as the cosnumers can take.
>> > Then I can do some throttling in consumer Camel route, but that
>> > throutling in Camel cannot be clustered accross instances. If the
>> > throttling can be configured ot queue level, then I don't have to do
>> > consumer throttling.
>> >
>> > Thanks,
>> >
>> > --
>> > Bilgin Ibryam
>> > Camel Committer at ASF & Integration Architect at Red Hat
>> > Blog: http://ofbizian.com | Twitter: @bibryam
>> >
>> > Camel Design Patterns https://leanpub.com/camel-design-patterns
>> > Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
>> >
>>
--
Bilgin Ibryam
Camel Committer at ASF & Integration Architect at Red Hat
Blog: http://ofbizian.com | Twitter: @bibryam
Camel Design Patterns https://leanpub.com/camel-design-patterns
Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
Re: Consumer throttling
Posted by Bilgin Ibryam <bi...@gmail.com>.
Thanks all for your inputs. It seems there is no solution for my
question in ActiveMQ yet.
@Christopher: I'm aware of scheduling feature, but that is more of a
Delayer rather than Throttler. It can delay messages, but cannot
ensure throttled consumption. If I get a burst of messages delay them,
at a later point in time there still will be a burst of messages.
Nothing has improved.
@Tim: I have described your suggestion with two queues and Camel in
between in as a temporary solution in my recent Camel Design Patterns
book, but that solution is not ideal either. It is still possible to
have messages accumulate in queueu B (for example if the consumers are
stopped temporarily) and then have non throttled consumption breaking
SLAs, etc.
@Rau I thoughtl PFC should take care of the message backlog and
protect the broker regardless of the reason for message accumulation,
whether that is slow consumers, throttled consumption or consumers
being down for some reason. Having prefetch limit of 1 is good
suggestion though, thanks.
On 30 January 2016 at 21:30, Tim Bain <tb...@alumni.duke.edu> wrote:
> You could achieve your goal by having two queues, A and B. Have producers
> publish to A and consumers consume from B, and use an embedded Camel route
> to move messages from A to B at the desired rate.
>
> But Raul's right: if you're going to do this, make sure there's some kind
> of safety feature to prevent messages from backlogging on the broker.
> Maybe you get it for free based on the characteristics of your clients, or
> maybe that's some mechanism to discard built-up messages (based on time,
> count, bytes, etc.), or a way to detect a backlog and temporarily speed up
> the Camel route or to let consumers consume directly from A for a while,
> or...
>
> Tim
> On Jan 30, 2016 12:56 PM, "Raul Kripalani" <ra...@apache.org> wrote:
>
>> Just one remark. This could be dangerous as it could lead to an
>> ever-growing backlog of messages in AMQ. I suppose you have already
>> considered it, and either the producers are slower than the consumers, or
>> you have multiple competing consumers, or there is a period in the day
>> where the producers stop producing to give the consumers time to consume
>> the existing backlog or something.
>>
>> If you implement this, I would reduce the prefect buffer to 1 if you have
>> multiple competing consumers to achieve decent distribution and avoid
>> starvation.
>>
>> Raúl.
>> On 30 Jan 2016 18:15, "Bilgin Ibryam" <bi...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > Apologoes if this is a stupid question. But I wonder is there a way to
>> > do throttling on the consumer side. Let me clarify what I mean.
>> >
>> > PFC can do throttling for the producer, but I need a similar behaviour
>> > on the consumer. For example I want to configure a message consure or
>> > message queue so that messages are consumed/released one per second.
>> > Usually the messages are consumed as fast as the cosnumers can take.
>> > Then I can do some throttling in consumer Camel route, but that
>> > throutling in Camel cannot be clustered accross instances. If the
>> > throttling can be configured ot queue level, then I don't have to do
>> > consumer throttling.
>> >
>> > Thanks,
>> >
>> > --
>> > Bilgin Ibryam
>> > Camel Committer at ASF & Integration Architect at Red Hat
>> > Blog: http://ofbizian.com | Twitter: @bibryam
>> >
>> > Camel Design Patterns https://leanpub.com/camel-design-patterns
>> > Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
>> >
>>
--
Bilgin Ibryam
Camel Committer at ASF & Integration Architect at Red Hat
Blog: http://ofbizian.com | Twitter: @bibryam
Camel Design Patterns https://leanpub.com/camel-design-patterns
Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
Re: Consumer throttling
Posted by Tim Bain <tb...@alumni.duke.edu>.
You could achieve your goal by having two queues, A and B. Have producers
publish to A and consumers consume from B, and use an embedded Camel route
to move messages from A to B at the desired rate.
But Raul's right: if you're going to do this, make sure there's some kind
of safety feature to prevent messages from backlogging on the broker.
Maybe you get it for free based on the characteristics of your clients, or
maybe that's some mechanism to discard built-up messages (based on time,
count, bytes, etc.), or a way to detect a backlog and temporarily speed up
the Camel route or to let consumers consume directly from A for a while,
or...
Tim
On Jan 30, 2016 12:56 PM, "Raul Kripalani" <ra...@apache.org> wrote:
> Just one remark. This could be dangerous as it could lead to an
> ever-growing backlog of messages in AMQ. I suppose you have already
> considered it, and either the producers are slower than the consumers, or
> you have multiple competing consumers, or there is a period in the day
> where the producers stop producing to give the consumers time to consume
> the existing backlog or something.
>
> If you implement this, I would reduce the prefect buffer to 1 if you have
> multiple competing consumers to achieve decent distribution and avoid
> starvation.
>
> Raúl.
> On 30 Jan 2016 18:15, "Bilgin Ibryam" <bi...@gmail.com> wrote:
>
> > Hi all,
> >
> > Apologoes if this is a stupid question. But I wonder is there a way to
> > do throttling on the consumer side. Let me clarify what I mean.
> >
> > PFC can do throttling for the producer, but I need a similar behaviour
> > on the consumer. For example I want to configure a message consure or
> > message queue so that messages are consumed/released one per second.
> > Usually the messages are consumed as fast as the cosnumers can take.
> > Then I can do some throttling in consumer Camel route, but that
> > throutling in Camel cannot be clustered accross instances. If the
> > throttling can be configured ot queue level, then I don't have to do
> > consumer throttling.
> >
> > Thanks,
> >
> > --
> > Bilgin Ibryam
> > Camel Committer at ASF & Integration Architect at Red Hat
> > Blog: http://ofbizian.com | Twitter: @bibryam
> >
> > Camel Design Patterns https://leanpub.com/camel-design-patterns
> > Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
> >
>
Re: Consumer throttling
Posted by Raul Kripalani <ra...@apache.org>.
Just one remark. This could be dangerous as it could lead to an
ever-growing backlog of messages in AMQ. I suppose you have already
considered it, and either the producers are slower than the consumers, or
you have multiple competing consumers, or there is a period in the day
where the producers stop producing to give the consumers time to consume
the existing backlog or something.
If you implement this, I would reduce the prefect buffer to 1 if you have
multiple competing consumers to achieve decent distribution and avoid
starvation.
Raúl.
On 30 Jan 2016 18:15, "Bilgin Ibryam" <bi...@gmail.com> wrote:
> Hi all,
>
> Apologoes if this is a stupid question. But I wonder is there a way to
> do throttling on the consumer side. Let me clarify what I mean.
>
> PFC can do throttling for the producer, but I need a similar behaviour
> on the consumer. For example I want to configure a message consure or
> message queue so that messages are consumed/released one per second.
> Usually the messages are consumed as fast as the cosnumers can take.
> Then I can do some throttling in consumer Camel route, but that
> throutling in Camel cannot be clustered accross instances. If the
> throttling can be configured ot queue level, then I don't have to do
> consumer throttling.
>
> Thanks,
>
> --
> Bilgin Ibryam
> Camel Committer at ASF & Integration Architect at Red Hat
> Blog: http://ofbizian.com | Twitter: @bibryam
>
> Camel Design Patterns https://leanpub.com/camel-design-patterns
> Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
>