You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Michael Goulish <mg...@redhat.com> on 2014/05/01 03:09:42 UTC
dispatch router handles 100,000 addresses
I just had a successful test in which a single dispatch router
handled 100,000 unique addresses.
The router was running on its own box. On a separate box, I
started 100 receiving clients ( proton messenger based ) each
receiving 1000 unique addresses. A single sending client
( qpid-messaging based ) sent the 100,000 messages to the
router -- 1 to each address.
All the receivers got their expected 1000 messages each,
and retired happy.
I tried to scale up to 1,000,000 addresses. The router seemed
OK with that, but my client-box melted down to slag so now I'm
a little shy to scale up further...
I need a bigger computer...
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Michael Goulish <mg...@redhat.com>.
----- Original Message -----
> On Thu, May 1, 2014 at 4:28 PM, Gordon Sim <gs...@redhat.com> wrote:
>
> > On 05/01/2014 08:55 PM, Rafael Schloming wrote:
> >
> >> On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <mg...@redhat.com>
> >> wrote:
> >>
> >>> I tried firing up my messenger-based receivers, each subscribing to 100
> >>> addresses, then 200, 300, 400, 500. The results are consistent across
> >>> that range, and show that each extra address costs 115 KB. ( Looking
> >>> only at resident-set size. )
> >>>
> >>> So when I tried to do a total of 1,000,000 addrs on one box, I did
> >>> indeed overwhelm my memory. That would come to 115 GB, which
> >>> would have been more than double my physical mem.
> >>>
> >>> Please note I did not actually send any messages. A router was running
> >>> for these receivers to attach to, but no senders were running.
> >>>
> >>> Does 115 KB per subscribed addr seem fairly reasonable?
> >>>
> >>>
> >> No, that seems quite excessive. Can you trace where the memory is actually
> >> coming from?
> >>
> >
> > Just for comparison, a qpid::messaging process with 1000 subscriptions
> > over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP 0-10
> > uses 8MB.
>
>
> Did you also have 1000 sessions, or were all the subscriptions on a single
> session?
>
> --Rafael
>
How do I do multiple sessions?
I think I only have one.
here, look:
int
main(int argc, char** argv)
{
size_t bufsize = BUFSIZE;
char buffer [ BUFSIZE ];
char addr [ 1000 ];
pn_message_t * message;
pn_messenger_t * messenger;
pn_data_t * body;
int const how_many_addrs = atoi(argv[4]);
int my_first_addr = atoi ( argv[3] );
char * addrs [ how_many_addrs ];
for ( int i = 0; i < how_many_addrs; ++ i )
{
addrs[i] = (char *) malloc ( 100 );
sprintf ( addrs[i],
"amqp://%s:%s/mick/%d",
argv[1],
argv[2],
my_first_addr + i
);
}
message = pn_message();
messenger = pn_messenger ( NULL );
pn_messenger_start(messenger);
for ( int i = 0; i < how_many_addrs; ++ i )
{
pn_messenger_subscribe ( messenger, addrs[i] );
}
fprintf ( stderr, "subscribed to %d addrs\n", how_many_addrs );
#ifdef EXPLICITLY_ACCEPT
pn_messenger_set_incoming_window ( messenger, 10 );
#endif
int message_count = 0;
fprintf ( stderr, "receiver |%d| starting.\n", my_first_addr );
#if 0
// remove this code for mem testing. no one is sending messages.
while ( 1 )
{
pn_messenger_recv ( messenger, BUFSIZE );
pn_messenger_get ( messenger, message );
#ifdef EXPLICITLY_ACCEPT
pn_tracker_t tracker = pn_messenger_incoming_tracker ( messenger );
pn_messenger_accept ( messenger, tracker, 0 );
#endif
++ message_count;
// fprintf ( stderr, "receiver |%d| got msg %d.\n", my_first_addr, message_count );
if ( message_count >= how_many_addrs )
{
break;
}
}
#endif
fprintf ( stderr, "receiver |%d| complete: %d messages.\n", my_first_addr, how_many_addrs );
print_time ( );
pn_messenger_stop(messenger);
pn_messenger_free(messenger);
return 0;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Rafael Schloming <rh...@alum.mit.edu>.
We should still be able to do a lot better. I suspect just minimizing the
overhead of an empty pn_data_t will help a lot.
--Rafael
On Fri, May 2, 2014 at 9:42 AM, Michael Goulish <mg...@redhat.com> wrote:
>
>
> That definitely helped!
>
> I did the same test as yesterday ( with 100 ... 500 addresses per receiver
> ).
>
> Yesterday I saw 115 KB per extra address, today I see 80 KB.
>
>
> That change *might* allow me to attempt a 1,000,000 address test, if I use
> my best boxes.... :-)
>
>
> Thanks for the memories!
>
>
>
> ----- Original Message -----
> I think a lot of the extra overhead is coming from some overly generous
> default allocation sizes that ended up getting used in a lot of places.
> I've adjusted these down. If you can retry your testing with trunk I'm
> hoping you should see some improvement.
>
> --Rafael
>
>
> On Fri, May 2, 2014 at 6:29 AM, Rafael Schloming <rh...@alum.mit.edu> wrote:
>
> > On Thu, May 1, 2014 at 8:14 PM, Michael Goulish <mgoulish@redhat.com
> >wrote:
> >
> >>
> >>
> >>
> >>
> >> ----- Original Message -----
> >> > On 05/01/2014 08:55 PM, Rafael Schloming wrote:
> >> > > On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <
> mgoulish@redhat.com>
> >> > > wrote:
> >> > >> I tried firing up my messenger-based receivers, each subscribing to
> >> 100
> >> > >> addresses, then 200, 300, 400, 500. The results are consistent
> >> across
> >> > >> that range, and show that each extra address costs 115 KB. (
> Looking
> >> > >> only at resident-set size. )
> >> > >>
> >> > >> So when I tried to do a total of 1,000,000 addrs on one box, I did
> >> > >> indeed overwhelm my memory. That would come to 115 GB, which
> >> > >> would have been more than double my physical mem.
> >> > >>
> >> > >> Please note I did not actually send any messages. A router was
> >> running
> >> > >> for these receivers to attach to, but no senders were running.
> >> > >>
> >> > >> Does 115 KB per subscribed addr seem fairly reasonable?
> >> > >>
> >> > >
> >> > > No, that seems quite excessive. Can you trace where the memory is
> >> actually
> >> > > coming from?
> >> >
> >> > Just for comparison, a qpid::messaging process with 1000 subscriptions
> >> > over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP
> 0-10
> >> > uses 8MB.
> >> >
> >> >
> >>
> >> Yikes. A 40 KB per subscription added cost.
> >> That's because 1.0 is ten times as good as 0.10 .
> >>
> >> Just off the top of your head -- is there anything inherent
> >> in 1.0 that would make you expect that kind of difference?
> >>
> >
> > Definitely not, the overhead should be comparable.
> >
> > --Rafael
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>
>
Re: dispatch router handles 100,000 addresses
Posted by Michael Goulish <mg...@redhat.com>.
That definitely helped!
I did the same test as yesterday ( with 100 ... 500 addresses per receiver ).
Yesterday I saw 115 KB per extra address, today I see 80 KB.
That change *might* allow me to attempt a 1,000,000 address test, if I use
my best boxes.... :-)
Thanks for the memories!
----- Original Message -----
I think a lot of the extra overhead is coming from some overly generous
default allocation sizes that ended up getting used in a lot of places.
I've adjusted these down. If you can retry your testing with trunk I'm
hoping you should see some improvement.
--Rafael
On Fri, May 2, 2014 at 6:29 AM, Rafael Schloming <rh...@alum.mit.edu> wrote:
> On Thu, May 1, 2014 at 8:14 PM, Michael Goulish <mg...@redhat.com>wrote:
>
>>
>>
>>
>>
>> ----- Original Message -----
>> > On 05/01/2014 08:55 PM, Rafael Schloming wrote:
>> > > On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <mg...@redhat.com>
>> > > wrote:
>> > >> I tried firing up my messenger-based receivers, each subscribing to
>> 100
>> > >> addresses, then 200, 300, 400, 500. The results are consistent
>> across
>> > >> that range, and show that each extra address costs 115 KB. ( Looking
>> > >> only at resident-set size. )
>> > >>
>> > >> So when I tried to do a total of 1,000,000 addrs on one box, I did
>> > >> indeed overwhelm my memory. That would come to 115 GB, which
>> > >> would have been more than double my physical mem.
>> > >>
>> > >> Please note I did not actually send any messages. A router was
>> running
>> > >> for these receivers to attach to, but no senders were running.
>> > >>
>> > >> Does 115 KB per subscribed addr seem fairly reasonable?
>> > >>
>> > >
>> > > No, that seems quite excessive. Can you trace where the memory is
>> actually
>> > > coming from?
>> >
>> > Just for comparison, a qpid::messaging process with 1000 subscriptions
>> > over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP 0-10
>> > uses 8MB.
>> >
>> >
>>
>> Yikes. A 40 KB per subscription added cost.
>> That's because 1.0 is ten times as good as 0.10 .
>>
>> Just off the top of your head -- is there anything inherent
>> in 1.0 that would make you expect that kind of difference?
>>
>
> Definitely not, the overhead should be comparable.
>
> --Rafael
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Rafael Schloming <rh...@alum.mit.edu>.
I think a lot of the extra overhead is coming from some overly generous
default allocation sizes that ended up getting used in a lot of places.
I've adjusted these down. If you can retry your testing with trunk I'm
hoping you should see some improvement.
--Rafael
On Fri, May 2, 2014 at 6:29 AM, Rafael Schloming <rh...@alum.mit.edu> wrote:
> On Thu, May 1, 2014 at 8:14 PM, Michael Goulish <mg...@redhat.com>wrote:
>
>>
>>
>>
>>
>> ----- Original Message -----
>> > On 05/01/2014 08:55 PM, Rafael Schloming wrote:
>> > > On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <mg...@redhat.com>
>> > > wrote:
>> > >> I tried firing up my messenger-based receivers, each subscribing to
>> 100
>> > >> addresses, then 200, 300, 400, 500. The results are consistent
>> across
>> > >> that range, and show that each extra address costs 115 KB. ( Looking
>> > >> only at resident-set size. )
>> > >>
>> > >> So when I tried to do a total of 1,000,000 addrs on one box, I did
>> > >> indeed overwhelm my memory. That would come to 115 GB, which
>> > >> would have been more than double my physical mem.
>> > >>
>> > >> Please note I did not actually send any messages. A router was
>> running
>> > >> for these receivers to attach to, but no senders were running.
>> > >>
>> > >> Does 115 KB per subscribed addr seem fairly reasonable?
>> > >>
>> > >
>> > > No, that seems quite excessive. Can you trace where the memory is
>> actually
>> > > coming from?
>> >
>> > Just for comparison, a qpid::messaging process with 1000 subscriptions
>> > over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP 0-10
>> > uses 8MB.
>> >
>> >
>>
>> Yikes. A 40 KB per subscription added cost.
>> That's because 1.0 is ten times as good as 0.10 .
>>
>> Just off the top of your head -- is there anything inherent
>> in 1.0 that would make you expect that kind of difference?
>>
>
> Definitely not, the overhead should be comparable.
>
> --Rafael
>
Re: dispatch router handles 100,000 addresses
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Thu, May 1, 2014 at 8:14 PM, Michael Goulish <mg...@redhat.com> wrote:
>
>
>
>
> ----- Original Message -----
> > On 05/01/2014 08:55 PM, Rafael Schloming wrote:
> > > On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <mg...@redhat.com>
> > > wrote:
> > >> I tried firing up my messenger-based receivers, each subscribing to
> 100
> > >> addresses, then 200, 300, 400, 500. The results are consistent across
> > >> that range, and show that each extra address costs 115 KB. ( Looking
> > >> only at resident-set size. )
> > >>
> > >> So when I tried to do a total of 1,000,000 addrs on one box, I did
> > >> indeed overwhelm my memory. That would come to 115 GB, which
> > >> would have been more than double my physical mem.
> > >>
> > >> Please note I did not actually send any messages. A router was
> running
> > >> for these receivers to attach to, but no senders were running.
> > >>
> > >> Does 115 KB per subscribed addr seem fairly reasonable?
> > >>
> > >
> > > No, that seems quite excessive. Can you trace where the memory is
> actually
> > > coming from?
> >
> > Just for comparison, a qpid::messaging process with 1000 subscriptions
> > over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP 0-10
> > uses 8MB.
> >
> >
>
> Yikes. A 40 KB per subscription added cost.
> That's because 1.0 is ten times as good as 0.10 .
>
> Just off the top of your head -- is there anything inherent
> in 1.0 that would make you expect that kind of difference?
>
Definitely not, the overhead should be comparable.
--Rafael
Re: dispatch router handles 100,000 addresses
Posted by Michael Goulish <mg...@redhat.com>.
----- Original Message -----
> On 05/01/2014 08:55 PM, Rafael Schloming wrote:
> > On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <mg...@redhat.com>
> > wrote:
> >> I tried firing up my messenger-based receivers, each subscribing to 100
> >> addresses, then 200, 300, 400, 500. The results are consistent across
> >> that range, and show that each extra address costs 115 KB. ( Looking
> >> only at resident-set size. )
> >>
> >> So when I tried to do a total of 1,000,000 addrs on one box, I did
> >> indeed overwhelm my memory. That would come to 115 GB, which
> >> would have been more than double my physical mem.
> >>
> >> Please note I did not actually send any messages. A router was running
> >> for these receivers to attach to, but no senders were running.
> >>
> >> Does 115 KB per subscribed addr seem fairly reasonable?
> >>
> >
> > No, that seems quite excessive. Can you trace where the memory is actually
> > coming from?
>
> Just for comparison, a qpid::messaging process with 1000 subscriptions
> over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP 0-10
> uses 8MB.
>
>
Yikes. A 40 KB per subscription added cost.
That's because 1.0 is ten times as good as 0.10 .
Just off the top of your head -- is there anything inherent
in 1.0 that would make you expect that kind of difference?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Gordon Sim <gs...@redhat.com>.
On 05/01/2014 09:49 PM, Rafael Schloming wrote:
> On Thu, May 1, 2014 at 4:28 PM, Gordon Sim <gs...@redhat.com> wrote:
>> Just for comparison, a qpid::messaging process with 1000 subscriptions
>> over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP 0-10
>> uses 8MB.
>
>
> Did you also have 1000 sessions, or were all the subscriptions on a single
> session?
In both cases it was a single session.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Thu, May 1, 2014 at 4:28 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 05/01/2014 08:55 PM, Rafael Schloming wrote:
>
>> On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <mg...@redhat.com>
>> wrote:
>>
>>> I tried firing up my messenger-based receivers, each subscribing to 100
>>> addresses, then 200, 300, 400, 500. The results are consistent across
>>> that range, and show that each extra address costs 115 KB. ( Looking
>>> only at resident-set size. )
>>>
>>> So when I tried to do a total of 1,000,000 addrs on one box, I did
>>> indeed overwhelm my memory. That would come to 115 GB, which
>>> would have been more than double my physical mem.
>>>
>>> Please note I did not actually send any messages. A router was running
>>> for these receivers to attach to, but no senders were running.
>>>
>>> Does 115 KB per subscribed addr seem fairly reasonable?
>>>
>>>
>> No, that seems quite excessive. Can you trace where the memory is actually
>> coming from?
>>
>
> Just for comparison, a qpid::messaging process with 1000 subscriptions
> over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP 0-10
> uses 8MB.
Did you also have 1000 sessions, or were all the subscriptions on a single
session?
--Rafael
Re: dispatch router handles 100,000 addresses
Posted by Gordon Sim <gs...@redhat.com>.
On 05/01/2014 08:55 PM, Rafael Schloming wrote:
> On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <mg...@redhat.com> wrote:
>> I tried firing up my messenger-based receivers, each subscribing to 100
>> addresses, then 200, 300, 400, 500. The results are consistent across
>> that range, and show that each extra address costs 115 KB. ( Looking
>> only at resident-set size. )
>>
>> So when I tried to do a total of 1,000,000 addrs on one box, I did
>> indeed overwhelm my memory. That would come to 115 GB, which
>> would have been more than double my physical mem.
>>
>> Please note I did not actually send any messages. A router was running
>> for these receivers to attach to, but no senders were running.
>>
>> Does 115 KB per subscribed addr seem fairly reasonable?
>>
>
> No, that seems quite excessive. Can you trace where the memory is actually
> coming from?
Just for comparison, a qpid::messaging process with 1000 subscriptions
over AMQP 1.0 uses 48MB on my laptop. A similar process using AMQP 0-10
uses 8MB.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Thu, May 1, 2014 at 3:42 PM, Michael Goulish <mg...@redhat.com> wrote:
>
>
> ----- Original Message -----
> On Thu, May 1, 2014 at 10:34 AM, Gordon Sim <gs...@redhat.com> wrote:
>
> > On 05/01/2014 03:09 PM, Rafael Schloming wrote:
> >
> >> On Thu, May 1, 2014 at 9:36 AM, Michael Goulish <mg...@redhat.com>
> >> wrote:
> >>
> >>> Since I reported earlier that 1 messenger-based sender grew to
> >>> 3.4 GB after sending to 30,000 unique addrs, it seems reasonable
> >>> that 1000 messenger-based receivers, attempting to receive from a total
> >>> of 1,000,000 addrs, would have attempted to grow to a total of more
> >>> than 100 GB. Which would account very nicely for the behavior I saw.
> >>> ( The box had 45 GB mem. )
> >>>
> >>
> > It would be worth actually confirming the growth of memory as you start
> > your receivers. The memory usage on the sender side isn't necessarily the
> > same as on the receiver side (depends of course what the memory is being
> > used for).
> >
> >
>
>
> I tried firing up my messenger-based receivers, each subscribing to 100
> addresses, then 200, 300, 400, 500. The results are consistent across
> that range, and show that each extra address costs 115 KB. ( Looking
> only at resident-set size. )
>
> So when I tried to do a total of 1,000,000 addrs on one box, I did
> indeed overwhelm my memory. That would come to 115 GB, which
> would have been more than double my physical mem.
>
> Please note I did not actually send any messages. A router was running
> for these receivers to attach to, but no senders were running.
>
> Does 115 KB per subscribed addr seem fairly reasonable?
>
No, that seems quite excessive. Can you trace where the memory is actually
coming from?
--Rafael
Re: dispatch router handles 100,000 addresses
Posted by Michael Goulish <mg...@redhat.com>.
----- Original Message -----
On Thu, May 1, 2014 at 10:34 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 05/01/2014 03:09 PM, Rafael Schloming wrote:
>
>> On Thu, May 1, 2014 at 9:36 AM, Michael Goulish <mg...@redhat.com>
>> wrote:
>>
>>> Since I reported earlier that 1 messenger-based sender grew to
>>> 3.4 GB after sending to 30,000 unique addrs, it seems reasonable
>>> that 1000 messenger-based receivers, attempting to receive from a total
>>> of 1,000,000 addrs, would have attempted to grow to a total of more
>>> than 100 GB. Which would account very nicely for the behavior I saw.
>>> ( The box had 45 GB mem. )
>>>
>>
> It would be worth actually confirming the growth of memory as you start
> your receivers. The memory usage on the sender side isn't necessarily the
> same as on the receiver side (depends of course what the memory is being
> used for).
>
>
I tried firing up my messenger-based receivers, each subscribing to 100
addresses, then 200, 300, 400, 500. The results are consistent across
that range, and show that each extra address costs 115 KB. ( Looking
only at resident-set size. )
So when I tried to do a total of 1,000,000 addrs on one box, I did
indeed overwhelm my memory. That would come to 115 GB, which
would have been more than double my physical mem.
Please note I did not actually send any messages. A router was running
for these receivers to attach to, but no senders were running.
Does 115 KB per subscribed addr seem fairly reasonable?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Thu, May 1, 2014 at 10:34 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 05/01/2014 03:09 PM, Rafael Schloming wrote:
>
>> On Thu, May 1, 2014 at 9:36 AM, Michael Goulish <mg...@redhat.com>
>> wrote:
>>
>>> Since I reported earlier that 1 messenger-based sender grew to
>>> 3.4 GB after sending to 30,000 unique addrs, it seems reasonable
>>> that 1000 messenger-based receivers, attempting to receive from a total
>>> of 1,000,000 addrs, would have attempted to grow to a total of more
>>> than 100 GB. Which would account very nicely for the behavior I saw.
>>> ( The box had 45 GB mem. )
>>>
>>
> It would be worth actually confirming the growth of memory as you start
> your receivers. The memory usage on the sender side isn't necessarily the
> same as on the receiver side (depends of course what the memory is being
> used for).
>
>
> The receive side is a bit different, and using qpid-messaging will not
>> necessarily help you scale up. Fundamentally in order to receive messages
>> from N different addresses you need to create N subscriptions. That's
>> going
>> to be just as expensive regardless of which API you use to do it.
>>
>
> On the client side, an extra subscription shouldn't require a great deal
> of extra memory though, at least I wouldn't expect it to.
Yeah, it would definitely be worth understanding where exactly the memory
usage is coming from. I would actually expect messaging, messenger, and
dispatch router to all have similar scaling characteristics when it comes
to multiple links on a connection. They are all using the engine underneath
and there is a certain minimum amount of per link state that is required by
the protocol, so the expected memory utilization would just be this +
whatever per link information is kept on top of that.
It occurs to me that one possible source of memory consumption might be the
delivery objects held on each link. Currently every link has it's own pool
of delivery objects. This won't be particularly efficient if you've got a
whole lot of links many of which are often idle. It should be pretty
straightforward to use a single connection scoped pool for the delivery
objects. This would probably improve things for dispatch, messenger, and
messaging, and I don't think there would be any drawbacks. I don't know if
this could account for as much memory usage as Mick is reporting though,
and I still don't see why there would be any difference in usage between
dispatch, messaging, and messenger.
--Rafael
Re: dispatch router handles 100,000 addresses
Posted by Gordon Sim <gs...@redhat.com>.
On 05/01/2014 03:09 PM, Rafael Schloming wrote:
> On Thu, May 1, 2014 at 9:36 AM, Michael Goulish <mg...@redhat.com> wrote:
>> Since I reported earlier that 1 messenger-based sender grew to
>> 3.4 GB after sending to 30,000 unique addrs, it seems reasonable
>> that 1000 messenger-based receivers, attempting to receive from a total
>> of 1,000,000 addrs, would have attempted to grow to a total of more
>> than 100 GB. Which would account very nicely for the behavior I saw.
>> ( The box had 45 GB mem. )
It would be worth actually confirming the growth of memory as you start
your receivers. The memory usage on the sender side isn't necessarily
the same as on the receiver side (depends of course what the memory is
being used for).
> The receive side is a bit different, and using qpid-messaging will not
> necessarily help you scale up. Fundamentally in order to receive messages
> from N different addresses you need to create N subscriptions. That's going
> to be just as expensive regardless of which API you use to do it.
On the client side, an extra subscription shouldn't require a great deal
of extra memory though, at least I wouldn't expect it to.
> The one
> option you might have with qpid-messaging is that you can close your
> receivers explicitly, whereas messenger doesn't have a way to cancel
> subscriptions. This isn't really a matter of intent though, it's just a
> missing feature.
I don't believe that would help in this test, since all the receivers
are active before the sender starts.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Thu, May 1, 2014 at 10:23 AM, Michael Goulish <mg...@redhat.com>wrote:
> Ah, good. Care to give a free example?
>
The big hammer would be:
messenger.route("*", "amqp://hostname")
This will funnel every single address you use into a single link to the
anonymous terminus on the given hostname. You can do finer grained stuff
too, e.g.:
messenger.route("foo/*", "amqp://hostname1/foo")
messenger.route("bar/*", "amqp://hostname2/bar")
The key thing to understand is that route doesn't at all impact the address
carried on the message, simply what underlying connection and link are
established and used to deliver the message, and because the rule on the
right doesn't include a substitution variable for the part of the address
that matches the wildcard, you will get the entire class of addresses
matching the pattern on the left funneled down whatever you specify on the
right.
Note that this does assume that whatever you are sending the message to on
the right is capable of understanding the address specified on the message
and routing based on this address.
>
> >
> > The receive side is a bit different, and using qpid-messaging will not
> > necessarily help you scale up. Fundamentally in order to receive messages
> > from N different addresses you need to create N subscriptions. That's
> going
> > to be just as expensive regardless of which API you use to do it. The one
> > option you might have with qpid-messaging is that you can close your
> > receivers explicitly, whereas messenger doesn't have a way to cancel
> > subscriptions. This isn't really a matter of intent though, it's just a
> > missing feature.
>
> Good point. 1 million subscriptions, no matter what I use.
>
> Dang. I wish there were a way for a receiver to multiplex many
> addresses through a single link, and a single subscription -- and
> if I want to know what addr a given msg came from, I could look it
> up in the message header or something.
>
The address is carried in the message header as you suggest, and in
principle you could create a subscription to the anonymous terminus with a
filter expressing interest in a broad category of messages. I believe the
dispatch router would need to support this for it to work though. There is
some work going on in the AMQP TC to specify some standard capabilities
around the anonymous terminus to make this sort of thing interoperable.
>
> >
> > I don't know if cancelling subscriptions will help you much or not, but
> I'm
> > guessing it would be of similar difficulty (and at least more generally
> > useful) to supply a patch for canceling subscriptions rather than
> retooling
> > your receivers.
> >
>
> No, that wouldn't help me right now -- although it does sound like a very
> good
> feature. I need all the zillion subscriptions set up before I ever start
> sending.
>
Yeah, I realized that after I posted. If you don't have all zillion
subscriptions up simultaneously, you aren't really testing scale. ;-)
--Rafael
Re: dispatch router handles 100,000 addresses
Posted by Michael Goulish <mg...@redhat.com>.
----- Original Message -----
> On Thu, May 1, 2014 at 9:36 AM, Michael Goulish <mg...@redhat.com> wrote:
>
> >
> >
> >
> >
> > ----- Original Message -----
> > > On 05/01/2014 02:09 AM, Michael Goulish wrote:
> > > >
> > > > I just had a successful test in which a single dispatch router
> > > > handled 100,000 unique addresses.
> > > >
> > > > The router was running on its own box. On a separate box, I
> > > > started 100 receiving clients ( proton messenger based ) each
> > > > receiving 1000 unique addresses. A single sending client
> > > > ( qpid-messaging based ) sent the 100,000 messages to the
> > > > router -- 1 to each address.
> > > >
> > > > All the receivers got their expected 1000 messages each,
> > > > and retired happy.
> > > >
> > > > I tried to scale up to 1,000,000 addresses. The router seemed
> > > > OK with that, but my client-box melted down to slag so now I'm
> > > > a little shy to scale up further...
> > >
> > > Was this with 100 clients each receiving on 10,000 unique addresses? Or
> > > 1000 clients each receiving on 1000 unique addresses?
> > >
> >
> > 1000 * 1000
> >
> > > On the client-box, was it a cpu issue? a memory issue? something else?
> > >
> >
> >
> > Unfortunately, I don't know. It became so unresponsive that I couldn't
> > get a prompt, couldn't run top. It felt like swapping.
> >
> > I suspect that it was the messenger 1-link-per-address thing.
> >
> > Since I reported earlier that 1 messenger-based sender grew to
> > 3.4 GB after sending to 30,000 unique addrs, it seems reasonable
> > that 1000 messenger-based receivers, attempting to receive from a total
> > of 1,000,000 addrs, would have attempted to grow to a total of more
> > than 100 GB. Which would account very nicely for the behavior I saw.
> > ( The box had 45 GB mem. )
> >
> > This kind of extreme scale-up with addresses on a single box is not
> > what messenger was intended for -- I am retooling to use qpid-messaging
> > based clients on both sides of the router.
> >
>
> I wouldn't say messenger isn't intended for this sort of scale up. The
> fundamental issue here if you need to scale is multiplexing a single link
> with messages to different addresses. This is something you can trivially
> instruct messenger to do with the appropriate route instruction. This
> should let you scale up indefinitely on the send side at least.
Ah, good. Care to give a free example?
>
> The receive side is a bit different, and using qpid-messaging will not
> necessarily help you scale up. Fundamentally in order to receive messages
> from N different addresses you need to create N subscriptions. That's going
> to be just as expensive regardless of which API you use to do it. The one
> option you might have with qpid-messaging is that you can close your
> receivers explicitly, whereas messenger doesn't have a way to cancel
> subscriptions. This isn't really a matter of intent though, it's just a
> missing feature.
Good point. 1 million subscriptions, no matter what I use.
Dang. I wish there were a way for a receiver to multiplex many
addresses through a single link, and a single subscription -- and
if I want to know what addr a given msg came from, I could look it
up in the message header or something.
>
> I don't know if cancelling subscriptions will help you much or not, but I'm
> guessing it would be of similar difficulty (and at least more generally
> useful) to supply a patch for canceling subscriptions rather than retooling
> your receivers.
>
No, that wouldn't help me right now -- although it does sound like a very good
feature. I need all the zillion subscriptions set up before I ever start sending.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Thu, May 1, 2014 at 9:36 AM, Michael Goulish <mg...@redhat.com> wrote:
>
>
>
>
> ----- Original Message -----
> > On 05/01/2014 02:09 AM, Michael Goulish wrote:
> > >
> > > I just had a successful test in which a single dispatch router
> > > handled 100,000 unique addresses.
> > >
> > > The router was running on its own box. On a separate box, I
> > > started 100 receiving clients ( proton messenger based ) each
> > > receiving 1000 unique addresses. A single sending client
> > > ( qpid-messaging based ) sent the 100,000 messages to the
> > > router -- 1 to each address.
> > >
> > > All the receivers got their expected 1000 messages each,
> > > and retired happy.
> > >
> > > I tried to scale up to 1,000,000 addresses. The router seemed
> > > OK with that, but my client-box melted down to slag so now I'm
> > > a little shy to scale up further...
> >
> > Was this with 100 clients each receiving on 10,000 unique addresses? Or
> > 1000 clients each receiving on 1000 unique addresses?
> >
>
> 1000 * 1000
>
> > On the client-box, was it a cpu issue? a memory issue? something else?
> >
>
>
> Unfortunately, I don't know. It became so unresponsive that I couldn't
> get a prompt, couldn't run top. It felt like swapping.
>
> I suspect that it was the messenger 1-link-per-address thing.
>
> Since I reported earlier that 1 messenger-based sender grew to
> 3.4 GB after sending to 30,000 unique addrs, it seems reasonable
> that 1000 messenger-based receivers, attempting to receive from a total
> of 1,000,000 addrs, would have attempted to grow to a total of more
> than 100 GB. Which would account very nicely for the behavior I saw.
> ( The box had 45 GB mem. )
>
> This kind of extreme scale-up with addresses on a single box is not
> what messenger was intended for -- I am retooling to use qpid-messaging
> based clients on both sides of the router.
>
I wouldn't say messenger isn't intended for this sort of scale up. The
fundamental issue here if you need to scale is multiplexing a single link
with messages to different addresses. This is something you can trivially
instruct messenger to do with the appropriate route instruction. This
should let you scale up indefinitely on the send side at least.
The receive side is a bit different, and using qpid-messaging will not
necessarily help you scale up. Fundamentally in order to receive messages
from N different addresses you need to create N subscriptions. That's going
to be just as expensive regardless of which API you use to do it. The one
option you might have with qpid-messaging is that you can close your
receivers explicitly, whereas messenger doesn't have a way to cancel
subscriptions. This isn't really a matter of intent though, it's just a
missing feature.
I don't know if cancelling subscriptions will help you much or not, but I'm
guessing it would be of similar difficulty (and at least more generally
useful) to supply a patch for canceling subscriptions rather than retooling
your receivers.
--Rafael
Re: dispatch router handles 100,000 addresses
Posted by Michael Goulish <mg...@redhat.com>.
----- Original Message -----
> On 05/01/2014 02:09 AM, Michael Goulish wrote:
> >
> > I just had a successful test in which a single dispatch router
> > handled 100,000 unique addresses.
> >
> > The router was running on its own box. On a separate box, I
> > started 100 receiving clients ( proton messenger based ) each
> > receiving 1000 unique addresses. A single sending client
> > ( qpid-messaging based ) sent the 100,000 messages to the
> > router -- 1 to each address.
> >
> > All the receivers got their expected 1000 messages each,
> > and retired happy.
> >
> > I tried to scale up to 1,000,000 addresses. The router seemed
> > OK with that, but my client-box melted down to slag so now I'm
> > a little shy to scale up further...
>
> Was this with 100 clients each receiving on 10,000 unique addresses? Or
> 1000 clients each receiving on 1000 unique addresses?
>
1000 * 1000
> On the client-box, was it a cpu issue? a memory issue? something else?
>
Unfortunately, I don't know. It became so unresponsive that I couldn't
get a prompt, couldn't run top. It felt like swapping.
I suspect that it was the messenger 1-link-per-address thing.
Since I reported earlier that 1 messenger-based sender grew to
3.4 GB after sending to 30,000 unique addrs, it seems reasonable
that 1000 messenger-based receivers, attempting to receive from a total
of 1,000,000 addrs, would have attempted to grow to a total of more
than 100 GB. Which would account very nicely for the behavior I saw.
( The box had 45 GB mem. )
This kind of extreme scale-up with addresses on a single box is not
what messenger was intended for -- I am retooling to use qpid-messaging
based clients on both sides of the router.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Gordon Sim <gs...@redhat.com>.
On 05/01/2014 02:09 AM, Michael Goulish wrote:
>
> I just had a successful test in which a single dispatch router
> handled 100,000 unique addresses.
>
> The router was running on its own box. On a separate box, I
> started 100 receiving clients ( proton messenger based ) each
> receiving 1000 unique addresses. A single sending client
> ( qpid-messaging based ) sent the 100,000 messages to the
> router -- 1 to each address.
>
> All the receivers got their expected 1000 messages each,
> and retired happy.
>
> I tried to scale up to 1,000,000 addresses. The router seemed
> OK with that, but my client-box melted down to slag so now I'm
> a little shy to scale up further...
Was this with 100 clients each receiving on 10,000 unique addresses? Or
1000 clients each receiving on 1000 unique addresses?
On the client-box, was it a cpu issue? a memory issue? something else?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: dispatch router handles 100,000 addresses
Posted by Steve Huston <sh...@riverace.com>.
Ooooooo.... The wheels are turning now :-)
Thanks for those numbers!
-Steve
> On Apr 30, 2014, at 9:11 PM, "Michael Goulish" <mg...@redhat.com> wrote:
>
>
> I just had a successful test in which a single dispatch router
> handled 100,000 unique addresses.
>
> The router was running on its own box. On a separate box, I
> started 100 receiving clients ( proton messenger based ) each
> receiving 1000 unique addresses. A single sending client
> ( qpid-messaging based ) sent the 100,000 messages to the
> router -- 1 to each address.
>
> All the receivers got their expected 1000 messages each,
> and retired happy.
>
> I tried to scale up to 1,000,000 addresses. The router seemed
> OK with that, but my client-box melted down to slag so now I'm
> a little shy to scale up further...
>
> I need a bigger computer...
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org