You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by nw <ni...@gmail.com> on 2011/06/21 12:20:31 UTC
Rate limiting message uploads to a broker
Hi,
I have a store and forward network of brokers (A), with one broker at a
different geographical location (B) which consumers (C) are processing
messages from. Our problem is bandwidth restrictions from A to B.
My question is, is it possible to rate limit messages that are forwarded
from A to B? I looked at BrokerFilter, but that is designed to accept/reject
connections and sadly not individual messages.
I was thinking of something like having access to the underlying
InputStream, then reading a number of bytes, sleeping for a few millis, etc.
I don't think ActiveMQ Streams are what we'd like; the splitting of the data
into seperate JMS messages will confuse monitoring systems, and the lack of
ability to replay messages if the consumer fails isn't good.
Using message blobs would require a shared resource between the consumers C
and the producers that feed A, where none exists in my system.
Thanks,
Nick
--
View this message in context: http://activemq.2283324.n4.nabble.com/Rate-limiting-message-uploads-to-a-broker-tp3613653p3613653.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Rate limiting message uploads to a broker
Posted by nw <ni...@gmail.com>.
Hi,
Yeah, I looked at blob messages, the message streaming would be ideal as I
could read bytes and pause, but I don't have shared resource between the two
systems unfortunately.
The broker-broker link is the only mechanism for propagating information
across the tiers.
Thanks,
Nick
--
View this message in context: http://activemq.2283324.n4.nabble.com/Rate-limiting-message-uploads-to-a-broker-tp3613653p3621119.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Rate limiting message uploads to a broker
Posted by Dejan Bosanac <de...@nighttale.net>.
Did you consider using blob messages (
http://activemq.apache.org/blob-messages.html). This would have your JMS
messages be small and the actual payload transferrer using some other server
(FTP for example).
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Wed, Jun 22, 2011 at 10:43 PM, nw <ni...@gmail.com> wrote:
> Hi,
>
> Thanks for your reply. Your solution would be good. Unfortunately there is
> another constraint that I failed to mention; the messages are rather large,
> with the capacity to go up to at least 5 Megs.
>
> If we add this to the example with your suggestion applied, even a single
> message would saturate the upstream from A to B until it was uploaded.
>
> Thanks,
>
> Nick
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Rate-limiting-message-uploads-to-a-broker-tp3613653p3618192.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Rate limiting message uploads to a broker
Posted by nw <ni...@gmail.com>.
Hi,
Thanks for your reply. Your solution would be good. Unfortunately there is
another constraint that I failed to mention; the messages are rather large,
with the capacity to go up to at least 5 Megs.
If we add this to the example with your suggestion applied, even a single
message would saturate the upstream from A to B until it was uploaded.
Thanks,
Nick
--
View this message in context: http://activemq.2283324.n4.nabble.com/Rate-limiting-message-uploads-to-a-broker-tp3613653p3618192.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Rate limiting message uploads to a broker
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi,
you can consider using smaller prefetch size (1 or so) which will make local
broker send one message at the time. This could slow down the NC and thus
help with your bandwidth.
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Tue, Jun 21, 2011 at 12:20 PM, nw <ni...@gmail.com> wrote:
> Hi,
>
> I have a store and forward network of brokers (A), with one broker at a
> different geographical location (B) which consumers (C) are processing
> messages from. Our problem is bandwidth restrictions from A to B.
>
> My question is, is it possible to rate limit messages that are forwarded
> from A to B? I looked at BrokerFilter, but that is designed to
> accept/reject
> connections and sadly not individual messages.
>
> I was thinking of something like having access to the underlying
> InputStream, then reading a number of bytes, sleeping for a few millis,
> etc.
>
> I don't think ActiveMQ Streams are what we'd like; the splitting of the
> data
> into seperate JMS messages will confuse monitoring systems, and the lack of
> ability to replay messages if the consumer fails isn't good.
>
> Using message blobs would require a shared resource between the consumers C
> and the producers that feed A, where none exists in my system.
>
> Thanks,
>
> Nick
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Rate-limiting-message-uploads-to-a-broker-tp3613653p3613653.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>