You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Bruno Matos <br...@paradigmaxis.pt> on 2012/04/18 12:19:52 UTC
Reduce Sender traffic
Hi,
I have a sender that publishes about 10MB of messages per second for
different topics. Only about 1% of that traffic is delivered because
only about 1% of the topics are bound at a time. Is it possible, or
would it be desirable, that the sender only effectively send the
messages for witch there is a bound queue?
Tia.
--
Bruno Matos
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Reduce Sender traffic
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
I suppose as this point I'd start asking some philosophical questions.....
In general pub/sub is a fire and forget model and is most commonly used
when you want to decouple producers and consumers, what you are talking
about is essentially introducing a level of coupling between producers
and consumers.
Gordon has identified a good solution and the binding information can of
course be used to steer your producer. As has been stated this has the
clear advantage that your architecture (well your physical architecture)
doesn't need to change much.
I say physical because I'd assert that it is a change to your logical
architecture and this is a qpid specific approach not a "general"
messaging pattern. Not that that's necessarily a bad thing, qpid is
great and I'd do similar things myself, I'm just suggesting that you go
down this path with your eyes open.
An alternative approach might be to take the view that if only about 1%
of consumers are interested in the messages then a request/response
architecture might be a better fit.
I don't know your problem space and I'd suspect that Gordon's suggestion
is probably a quicker and simpler solution, and if you don't control the
consumers your options are limited etc.
The QMF2 API has an interesting approach it uses what might best be
described as "query subscriptions" where a QMF Console requests
management objects specified by a "query", the QMF Agent in this context
acts as a data producer and pushes back only messages meeting a
particular "pattern" to the consumer.
All of this stuff is way more complex than Gordon's suggestion, but
sometimes taking a step back and re-evaluating the overall problem space
is useful, just in case - at least then you'll be confident that any
implementation decision that you take will be purely an implementation
choice.
Frase
On 26/04/12 10:19, Bruno Matos wrote:
> On Wed, 2012-04-18 at 15:41 +0100, Gordon Sim wrote:
>> On 04/18/2012 11:19 AM, Bruno Matos wrote:
>>> Hi,
>>>
>>> I have a sender that publishes about 10MB of messages per second for
>>> different topics. Only about 1% of that traffic is delivered because
>>> only about 1% of the topics are bound at a time. Is it possible, or
>>> would it be desirable, that the sender only effectively send the
>>> messages for witch there is a bound queue?
>> I can see a couple of approaches that could work to limit the sending of
>> unwanted data.
>>
>> One is to subscribe to QMF event messages to detect binding and
>> unbinding at the sender, and deduce from that whether it is worth
>> sending anything.
>>
>> The other is to have the receivers explicitly send control messages
>> themselves indicating when they want and when they no longer need
>> particular data sets.
>>
>> Of course a lot depends on the nature of the data.
> Thank you Gordon, sorry for the late reply.
>
> I think the first option may solve the problem with little changes in
> architecture. I still have to measure the impact of the events'
> subscription.
>
> Thank you.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Reduce Sender traffic
Posted by Bruno Matos <br...@paradigmaxis.pt>.
On Wed, 2012-04-18 at 15:41 +0100, Gordon Sim wrote:
> On 04/18/2012 11:19 AM, Bruno Matos wrote:
> > Hi,
> >
> > I have a sender that publishes about 10MB of messages per second for
> > different topics. Only about 1% of that traffic is delivered because
> > only about 1% of the topics are bound at a time. Is it possible, or
> > would it be desirable, that the sender only effectively send the
> > messages for witch there is a bound queue?
>
> I can see a couple of approaches that could work to limit the sending of
> unwanted data.
>
> One is to subscribe to QMF event messages to detect binding and
> unbinding at the sender, and deduce from that whether it is worth
> sending anything.
>
> The other is to have the receivers explicitly send control messages
> themselves indicating when they want and when they no longer need
> particular data sets.
>
> Of course a lot depends on the nature of the data.
Thank you Gordon, sorry for the late reply.
I think the first option may solve the problem with little changes in
architecture. I still have to measure the impact of the events'
subscription.
Thank you.
--
Bruno Matos
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Reduce Sender traffic
Posted by Gordon Sim <gs...@redhat.com>.
On 04/18/2012 11:19 AM, Bruno Matos wrote:
> Hi,
>
> I have a sender that publishes about 10MB of messages per second for
> different topics. Only about 1% of that traffic is delivered because
> only about 1% of the topics are bound at a time. Is it possible, or
> would it be desirable, that the sender only effectively send the
> messages for witch there is a bound queue?
I can see a couple of approaches that could work to limit the sending of
unwanted data.
One is to subscribe to QMF event messages to detect binding and
unbinding at the sender, and deduce from that whether it is worth
sending anything.
The other is to have the receivers explicitly send control messages
themselves indicating when they want and when they no longer need
particular data sets.
Of course a lot depends on the nature of the data.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Reduce Sender traffic
Posted by Bruno Matos <br...@paradigmaxis.pt>.
Hi Pavel,
Thanks for the reply.
I'm using C++ broker with C++ clients.
Here is an example of what I'm trying to say:
I have a publisher that will send message #1 to news.weather and message
#2 to news.economy. If I only have one client and this client is only
listening news.weather, sending #2 would not be necessary.
This is what's happening but multiplied by thousands, so my sender is
occupying 10MB/s of my network, but 99% of the messages will be
discarded because no one is listening.
Best regards,
Bruno
On Wed, 2012-04-18 at 06:54 -0400, Pavel Moravec wrote:
> Hi Bruno,
> what broker (C++ / Java) and client (C++ / Java / Python / ..) are you using?
>
> Generally, I would recommend creating more AMQP sessions where messages to potentially blocked queues are sent on one session, time to time synchronize those sessions and use producer flow control that would block sender from sending messages to a queue occupied by 80% (default value) or more.
>
> Kind regards,
> Pavel
>
>
> ----- Original Message -----
> > From: "Bruno Matos" <br...@paradigmaxis.pt>
> > To: users@qpid.apache.org
> > Sent: Wednesday, April 18, 2012 12:19:52 PM
> > Subject: Reduce Sender traffic
> >
> > Hi,
> >
> > I have a sender that publishes about 10MB of messages per second for
> > different topics. Only about 1% of that traffic is delivered because
> > only about 1% of the topics are bound at a time. Is it possible, or
> > would it be desirable, that the sender only effectively send the
> > messages for witch there is a bound queue?
> >
> > Tia.
> >
> > --
> > Bruno Matos
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
--
Bruno Matos
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Reduce Sender traffic
Posted by Pavel Moravec <pm...@redhat.com>.
Hi Bruno,
what broker (C++ / Java) and client (C++ / Java / Python / ..) are you using?
Generally, I would recommend creating more AMQP sessions where messages to potentially blocked queues are sent on one session, time to time synchronize those sessions and use producer flow control that would block sender from sending messages to a queue occupied by 80% (default value) or more.
Kind regards,
Pavel
----- Original Message -----
> From: "Bruno Matos" <br...@paradigmaxis.pt>
> To: users@qpid.apache.org
> Sent: Wednesday, April 18, 2012 12:19:52 PM
> Subject: Reduce Sender traffic
>
> Hi,
>
> I have a sender that publishes about 10MB of messages per second for
> different topics. Only about 1% of that traffic is delivered because
> only about 1% of the topics are bound at a time. Is it possible, or
> would it be desirable, that the sender only effectively send the
> messages for witch there is a bound queue?
>
> Tia.
>
> --
> Bruno Matos
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org