You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Edwin <ed...@gmail.com> on 2012/06/26 19:09:41 UTC
Footprint of SEDA queues
Hi Folks,
I have a route which takes larges volumes of messages from an
endpoint(custom), splits the message in to subsets of messages via the load
balancer EIP and places on 5 different SEDA queues.
I have 200 of these routes:
from("custom:endpoint?param1=session1").loadBalance()....to("seda:session1_1","seda:session1_2",..."seda:session15")
.......
.......
from("custom:endpoint?param1=session200").loadBalance()....to("seda:session200_1","seda:session200_2",..."seda:session2_200")
Consequently, I now have potentially in excess of 1000 SEDA queues. Just
wondering what kind of footprint this number of SEDA queues have in terms of
performance and memory?
Would folks have concerns with this number of SEDA queues?
Thanks,
Edwin
--
View this message in context: http://camel.465427.n5.nabble.com/Footprint-of-SEDA-queues-tp5715128.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Footprint of SEDA queues
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Jun 26, 2012 at 7:09 PM, Edwin <ed...@gmail.com> wrote:
> Hi Folks,
>
> I have a route which takes larges volumes of messages from an
> endpoint(custom), splits the message in to subsets of messages via the load
> balancer EIP and places on 5 different SEDA queues.
>
> I have 200 of these routes:
> from("custom:endpoint?param1=session1").loadBalance()....to("seda:session1_1","seda:session1_2",..."seda:session15")
> .......
> .......
> from("custom:endpoint?param1=session200").loadBalance()....to("seda:session200_1","seda:session200_2",..."seda:session2_200")
>
> Consequently, I now have potentially in excess of 1000 SEDA queues. Just
> wondering what kind of footprint this number of SEDA queues have in terms of
> performance and memory?
>
> Would folks have concerns with this number of SEDA queues?
>
No, as each consumer from a seda queue has its own thread pool /
threads, which mean you will have 1000+ threads to consume from the
seda queues, if you have all queues in use at the same time.
> Thanks,
> Edwin
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Footprint-of-SEDA-queues-tp5715128.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Footprint of SEDA queues
Posted by Donald Whytock <dw...@gmail.com>.
If the number of threads truly isn't an issue, you can at least reduce
the number of SEDA queues to the number of routes by using the
concurrentConsumers option.
Is the reason for the 200 routes solely because of having 200
semi-dynamically-determined endpoints? Are messages from these 200
endpoints sufficiently distinct that you require 200 matching
processors? Or could you feed these 200 endpoints into a single queue
with a large concurrentConsumers setting?
Don
On Fri, Jun 29, 2012 at 6:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Fri, Jun 29, 2012 at 11:15 AM, Henryk Konsek <he...@gmail.com> wrote:
>>> Sounds like you echo my concerns Henryk. Was wondering if there is any kind
>>> of metrics on the overhead of SEDA queues. I have concerns similar to those
>>> outlined by Henryk
>>
>> My concerns don't relate to the performance since the quantity of
>> resources needed to handle 1000 queues is linear. You just got more
>> threads running on the JVM.
>>
>> My concerns rather affects the complexity of design of this kind of
>> solutions (thousands of similar SEDA queues in single Camel context).
>>
>
> Yeah I would reconsider the design.
>
>
>
>> --
>> Henryk Konsek
>> http://henryk-konsek.blogspot.com
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
Re: Footprint of SEDA queues
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Jun 29, 2012 at 11:15 AM, Henryk Konsek <he...@gmail.com> wrote:
>> Sounds like you echo my concerns Henryk. Was wondering if there is any kind
>> of metrics on the overhead of SEDA queues. I have concerns similar to those
>> outlined by Henryk
>
> My concerns don't relate to the performance since the quantity of
> resources needed to handle 1000 queues is linear. You just got more
> threads running on the JVM.
>
> My concerns rather affects the complexity of design of this kind of
> solutions (thousands of similar SEDA queues in single Camel context).
>
Yeah I would reconsider the design.
> --
> Henryk Konsek
> http://henryk-konsek.blogspot.com
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Footprint of SEDA queues
Posted by Henryk Konsek <he...@gmail.com>.
> Sounds like you echo my concerns Henryk. Was wondering if there is any kind
> of metrics on the overhead of SEDA queues. I have concerns similar to those
> outlined by Henryk
My concerns don't relate to the performance since the quantity of
resources needed to handle 1000 queues is linear. You just got more
threads running on the JVM.
My concerns rather affects the complexity of design of this kind of
solutions (thousands of similar SEDA queues in single Camel context).
--
Henryk Konsek
http://henryk-konsek.blogspot.com
Re: Footprint of SEDA queues
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Jun 28, 2012 at 6:26 PM, Edwin <ed...@gmail.com> wrote:
> Thanks for the responses Claus, Henryk
>
> I understand that each seda queue consumer will have its own thread pool.
> However, my concern more relates to the footprint imposed on the JVM by
> creating 1000+ SEDA Queues.
>
> Sounds like you echo my concerns Henryk. Was wondering if there is any kind
> of metrics on the overhead of SEDA queues. I have concerns similar to those
> outlined by Henryk
>
A SEDA queue is a pure in memory queue using the BlockingQueue API from the JDK.
You can easily create 1000 of these SEDA queues. Its only when you
either or do both
- add a lot of messages to these queues, eg adding messages faster
than you can consume (taking up memory)
- has 1000 consumers running at the same time, as that is at least
1000 threads in the JVM.
> Thanks
> Edwin
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Footprint-of-SEDA-queues-tp5715128p5715232.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Footprint of SEDA queues
Posted by Edwin <ed...@gmail.com>.
Thanks for the responses Claus, Henryk
I understand that each seda queue consumer will have its own thread pool.
However, my concern more relates to the footprint imposed on the JVM by
creating 1000+ SEDA Queues.
Sounds like you echo my concerns Henryk. Was wondering if there is any kind
of metrics on the overhead of SEDA queues. I have concerns similar to those
outlined by Henryk
Thanks
Edwin
--
View this message in context: http://camel.465427.n5.nabble.com/Footprint-of-SEDA-queues-tp5715128p5715232.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Footprint of SEDA queues
Posted by Henryk Konsek <he...@gmail.com>.
Hi Edwin,
> I have 200 of these routes:
> Would folks have concerns with this number of SEDA queues?
Just wanted to say that having 1000 similar SEDA queues in your router
(i.e. on the same VM) is a "routing smell" [1] for me. :) From my
experience - this looks bit too complex. When I end up with similar
architecture I usually try to rethink my solution to make it less
"epic".
Best regards.
[1] http://en.wikipedia.org/wiki/Code_smell
--
Henryk Konsek
http://henryk-konsek.blogspot.com