You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Claus Ibsen <cl...@gmail.com> on 2010/03/18 11:53:27 UTC
Camel 2.3 - Default threading profile - Your view
Hi
I am posting this to user so Camel end users have a chance to make an
impact on this matter.
I am working on making configuring thread pools in Camel much easier.
https://issues.apache.org/activemq/browse/CAMEL-1588
In light of this work we are introducing a ThreadPoolProfile which
contains settings to be used when a thread pool is to be created.
The idea currently is that there is a *default* profile that is used
normally, and its has the following options
private Integer poolSize = 10;
private Integer maxPoolSize = 20;
private Long keepAliveTime = 60L;
private TimeUnit timeUnit = TimeUnit.SECONDS;
private Integer maxQueueSize = 1000;
private ThreadPoolRejectedPolicy rejectedPolicy;
And the rejectedPolicy will default to use: CallerRuns. There are 4
options: Abort, CallerRuns, DiscardOldest, Discard;
I am currently wondering if these default settings is matching what
you would expect / like to be default in Camel 2.3 onwards.
How this works in Java is that the thread pool will have as min 10
threads in the pool and a worker queue with a 1000 capacity.
And only when its exhausted, eg the capacity has been hit, it will
grow the pool up till 20 threads to help.
And if threads is idle for more than 60 seconds it will be terminated.
There are two options which I may want to adjust change
- maxQueueSize = 1000
- rejectedPolicy = CallerRuns
Is the default capacity of 1000 to high? For example having 100, will
let the thread pool to grow sooner to help out.
Or is 100 to low. Also mind that the task queue is in memory so if the
server crashes those tasks are gone.
And the rejectedPolicy with CallerRuns allows to let the caller thread
execute the task instead of handing it over to the thread pool.
This is like a last resort if the thread pool is really exhausted and
otherwise would have to either abort or discard a task.
However the good point is that by doing this its automatically
throttling the caller as it cannot at the same time send new tasks to
the thread pool.
We could also add a new option: Wait. Which lets the caller wait a bit
to see if the task queue has room to handover the task. But when
waiting you
have to cater for timeout to avoid potentially waiting for a long
time. And this requires custom code to write, as the existing 4
options is provided
out of the box from JDK.
So when are these thread pools being used?
- EIPs (splitter, aggregator, multicast, wiretap etc. etc.)
- ProducerTemplate having a cached pool of Producer
- Threads DSL
- toAsync DSL
The various components may have their own thread pool, such as Jetty,
Mina, Spring JMS etc.
Any thoughts?
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Camel 2.3 - Default threading profile - Your view
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Most of the work is done now. What remains is to do some wiki documentation.
I recon we got a good solution now with thread pool profiles which
makes it easy to setup the threading model you need.
On Fri, Mar 19, 2010 at 11:07 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> You can see notes here as well
> http://camel.apache.org/camel-23-threadpool-configuration.html
>
>
> On Thu, Mar 18, 2010 at 2:24 PM, Ashwin Karpe <as...@progress.com> wrote:
>>
>> Hi Claus,
>>
>> I do like the settings as initial values which could be potentially
>> overridden by the components. I also agree with the workQueue filling up
>> being the trigger to add more threads.
>>
>> It would be great if all components could uniformly apply these initial
>> values which could ten be changed by the application/component users
>> themselves. This will prevent components from starving each other out with
>> their individual settings
>>
>
> Moving forward Components can leverage
> CamelContext.getExecutorServiceStrategy() to create the kind of thread
> pool they want.
> But using this strategy you let Camel aid you and have it
> - export the pool for JMX so it can be managed
> - handle lifecycle by shutting it down when Camel shutdown
> - logging details about the created pool
> - allows 3rd party strategies to hook in and use their own thread pool
> factory (eg WorkManager from a J2EE server)
> - easier to create thread pool than the ugly ThreadPoolExecutor constructor
>
> When time permits I will create a wiki page with more details.
> Its also being documented in chapter 10 of the Camel in Action book as well.
>
>
>
>> Hope this helps.
>>
>> Cheers,
>>
>> Ashwin...
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> I am posting this to user so Camel end users have a chance to make an
>>> impact on this matter.
>>>
>>>
>>> I am working on making configuring thread pools in Camel much easier.
>>> https://issues.apache.org/activemq/browse/CAMEL-1588
>>>
>>> In light of this work we are introducing a ThreadPoolProfile which
>>> contains settings to be used when a thread pool is to be created.
>>>
>>> The idea currently is that there is a *default* profile that is used
>>> normally, and its has the following options
>>> private Integer poolSize = 10;
>>> private Integer maxPoolSize = 20;
>>> private Long keepAliveTime = 60L;
>>> private TimeUnit timeUnit = TimeUnit.SECONDS;
>>> private Integer maxQueueSize = 1000;
>>> private ThreadPoolRejectedPolicy rejectedPolicy;
>>>
>>> And the rejectedPolicy will default to use: CallerRuns. There are 4
>>> options: Abort, CallerRuns, DiscardOldest, Discard;
>>>
>>>
>>> I am currently wondering if these default settings is matching what
>>> you would expect / like to be default in Camel 2.3 onwards.
>>> How this works in Java is that the thread pool will have as min 10
>>> threads in the pool and a worker queue with a 1000 capacity.
>>> And only when its exhausted, eg the capacity has been hit, it will
>>> grow the pool up till 20 threads to help.
>>> And if threads is idle for more than 60 seconds it will be terminated.
>>>
>>> There are two options which I may want to adjust change
>>> - maxQueueSize = 1000
>>> - rejectedPolicy = CallerRuns
>>>
>>> Is the default capacity of 1000 to high? For example having 100, will
>>> let the thread pool to grow sooner to help out.
>>> Or is 100 to low. Also mind that the task queue is in memory so if the
>>> server crashes those tasks are gone.
>>>
>>> And the rejectedPolicy with CallerRuns allows to let the caller thread
>>> execute the task instead of handing it over to the thread pool.
>>> This is like a last resort if the thread pool is really exhausted and
>>> otherwise would have to either abort or discard a task.
>>> However the good point is that by doing this its automatically
>>> throttling the caller as it cannot at the same time send new tasks to
>>> the thread pool.
>>>
>>> We could also add a new option: Wait. Which lets the caller wait a bit
>>> to see if the task queue has room to handover the task. But when
>>> waiting you
>>> have to cater for timeout to avoid potentially waiting for a long
>>> time. And this requires custom code to write, as the existing 4
>>> options is provided
>>> out of the box from JDK.
>>>
>>>
>>> So when are these thread pools being used?
>>> - EIPs (splitter, aggregator, multicast, wiretap etc. etc.)
>>> - ProducerTemplate having a cached pool of Producer
>>> - Threads DSL
>>> - toAsync DSL
>>>
>>> The various components may have their own thread pool, such as Jetty,
>>> Mina, Spring JMS etc.
>>>
>>>
>>> Any thoughts?
>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>>
>> -----
>> ---
>> Ashwin Karpe, Principal Consultant, PS - Opensource Center of Competence
>> Progress Software Corporation
>> 14 Oak Park Drive
>> Bedford, MA 01730
>> ---
>> +1-972-304-9084 (Office)
>> +1-972-971-1700 (Mobile)
>> ----
>> Blog: http://opensourceknowledge.blogspot.com/
>>
>>
>> --
>> View this message in context: http://old.nabble.com/Camel-2.3---Default-threading-profile---Your-view-tp27943985p27945489.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Camel 2.3 - Default threading profile - Your view
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
You can see notes here as well
http://camel.apache.org/camel-23-threadpool-configuration.html
On Thu, Mar 18, 2010 at 2:24 PM, Ashwin Karpe <as...@progress.com> wrote:
>
> Hi Claus,
>
> I do like the settings as initial values which could be potentially
> overridden by the components. I also agree with the workQueue filling up
> being the trigger to add more threads.
>
> It would be great if all components could uniformly apply these initial
> values which could ten be changed by the application/component users
> themselves. This will prevent components from starving each other out with
> their individual settings
>
Moving forward Components can leverage
CamelContext.getExecutorServiceStrategy() to create the kind of thread
pool they want.
But using this strategy you let Camel aid you and have it
- export the pool for JMX so it can be managed
- handle lifecycle by shutting it down when Camel shutdown
- logging details about the created pool
- allows 3rd party strategies to hook in and use their own thread pool
factory (eg WorkManager from a J2EE server)
- easier to create thread pool than the ugly ThreadPoolExecutor constructor
When time permits I will create a wiki page with more details.
Its also being documented in chapter 10 of the Camel in Action book as well.
> Hope this helps.
>
> Cheers,
>
> Ashwin...
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> I am posting this to user so Camel end users have a chance to make an
>> impact on this matter.
>>
>>
>> I am working on making configuring thread pools in Camel much easier.
>> https://issues.apache.org/activemq/browse/CAMEL-1588
>>
>> In light of this work we are introducing a ThreadPoolProfile which
>> contains settings to be used when a thread pool is to be created.
>>
>> The idea currently is that there is a *default* profile that is used
>> normally, and its has the following options
>> private Integer poolSize = 10;
>> private Integer maxPoolSize = 20;
>> private Long keepAliveTime = 60L;
>> private TimeUnit timeUnit = TimeUnit.SECONDS;
>> private Integer maxQueueSize = 1000;
>> private ThreadPoolRejectedPolicy rejectedPolicy;
>>
>> And the rejectedPolicy will default to use: CallerRuns. There are 4
>> options: Abort, CallerRuns, DiscardOldest, Discard;
>>
>>
>> I am currently wondering if these default settings is matching what
>> you would expect / like to be default in Camel 2.3 onwards.
>> How this works in Java is that the thread pool will have as min 10
>> threads in the pool and a worker queue with a 1000 capacity.
>> And only when its exhausted, eg the capacity has been hit, it will
>> grow the pool up till 20 threads to help.
>> And if threads is idle for more than 60 seconds it will be terminated.
>>
>> There are two options which I may want to adjust change
>> - maxQueueSize = 1000
>> - rejectedPolicy = CallerRuns
>>
>> Is the default capacity of 1000 to high? For example having 100, will
>> let the thread pool to grow sooner to help out.
>> Or is 100 to low. Also mind that the task queue is in memory so if the
>> server crashes those tasks are gone.
>>
>> And the rejectedPolicy with CallerRuns allows to let the caller thread
>> execute the task instead of handing it over to the thread pool.
>> This is like a last resort if the thread pool is really exhausted and
>> otherwise would have to either abort or discard a task.
>> However the good point is that by doing this its automatically
>> throttling the caller as it cannot at the same time send new tasks to
>> the thread pool.
>>
>> We could also add a new option: Wait. Which lets the caller wait a bit
>> to see if the task queue has room to handover the task. But when
>> waiting you
>> have to cater for timeout to avoid potentially waiting for a long
>> time. And this requires custom code to write, as the existing 4
>> options is provided
>> out of the box from JDK.
>>
>>
>> So when are these thread pools being used?
>> - EIPs (splitter, aggregator, multicast, wiretap etc. etc.)
>> - ProducerTemplate having a cached pool of Producer
>> - Threads DSL
>> - toAsync DSL
>>
>> The various components may have their own thread pool, such as Jetty,
>> Mina, Spring JMS etc.
>>
>>
>> Any thoughts?
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
>
> -----
> ---
> Ashwin Karpe, Principal Consultant, PS - Opensource Center of Competence
> Progress Software Corporation
> 14 Oak Park Drive
> Bedford, MA 01730
> ---
> +1-972-304-9084 (Office)
> +1-972-971-1700 (Mobile)
> ----
> Blog: http://opensourceknowledge.blogspot.com/
>
>
> --
> View this message in context: http://old.nabble.com/Camel-2.3---Default-threading-profile---Your-view-tp27943985p27945489.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Camel 2.3 - Default threading profile - Your view
Posted by Norman Maurer <no...@googlemail.com>.
+1,
I really like the idea..
Bye,
Norman
2010/3/18 Ashwin Karpe <as...@progress.com>:
>
> Hi Claus,
>
> I do like the settings as initial values which could be potentially
> overridden by the components. I also agree with the workQueue filling up
> being the trigger to add more threads.
>
> It would be great if all components could uniformly apply these initial
> values which could ten be changed by the application/component users
> themselves. This will prevent components from starving each other out with
> their individual settings
>
> Hope this helps.
>
> Cheers,
>
> Ashwin...
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> I am posting this to user so Camel end users have a chance to make an
>> impact on this matter.
>>
>>
>> I am working on making configuring thread pools in Camel much easier.
>> https://issues.apache.org/activemq/browse/CAMEL-1588
>>
>> In light of this work we are introducing a ThreadPoolProfile which
>> contains settings to be used when a thread pool is to be created.
>>
>> The idea currently is that there is a *default* profile that is used
>> normally, and its has the following options
>> private Integer poolSize = 10;
>> private Integer maxPoolSize = 20;
>> private Long keepAliveTime = 60L;
>> private TimeUnit timeUnit = TimeUnit.SECONDS;
>> private Integer maxQueueSize = 1000;
>> private ThreadPoolRejectedPolicy rejectedPolicy;
>>
>> And the rejectedPolicy will default to use: CallerRuns. There are 4
>> options: Abort, CallerRuns, DiscardOldest, Discard;
>>
>>
>> I am currently wondering if these default settings is matching what
>> you would expect / like to be default in Camel 2.3 onwards.
>> How this works in Java is that the thread pool will have as min 10
>> threads in the pool and a worker queue with a 1000 capacity.
>> And only when its exhausted, eg the capacity has been hit, it will
>> grow the pool up till 20 threads to help.
>> And if threads is idle for more than 60 seconds it will be terminated.
>>
>> There are two options which I may want to adjust change
>> - maxQueueSize = 1000
>> - rejectedPolicy = CallerRuns
>>
>> Is the default capacity of 1000 to high? For example having 100, will
>> let the thread pool to grow sooner to help out.
>> Or is 100 to low. Also mind that the task queue is in memory so if the
>> server crashes those tasks are gone.
>>
>> And the rejectedPolicy with CallerRuns allows to let the caller thread
>> execute the task instead of handing it over to the thread pool.
>> This is like a last resort if the thread pool is really exhausted and
>> otherwise would have to either abort or discard a task.
>> However the good point is that by doing this its automatically
>> throttling the caller as it cannot at the same time send new tasks to
>> the thread pool.
>>
>> We could also add a new option: Wait. Which lets the caller wait a bit
>> to see if the task queue has room to handover the task. But when
>> waiting you
>> have to cater for timeout to avoid potentially waiting for a long
>> time. And this requires custom code to write, as the existing 4
>> options is provided
>> out of the box from JDK.
>>
>>
>> So when are these thread pools being used?
>> - EIPs (splitter, aggregator, multicast, wiretap etc. etc.)
>> - ProducerTemplate having a cached pool of Producer
>> - Threads DSL
>> - toAsync DSL
>>
>> The various components may have their own thread pool, such as Jetty,
>> Mina, Spring JMS etc.
>>
>>
>> Any thoughts?
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
>
> -----
> ---
> Ashwin Karpe, Principal Consultant, PS - Opensource Center of Competence
> Progress Software Corporation
> 14 Oak Park Drive
> Bedford, MA 01730
> ---
> +1-972-304-9084 (Office)
> +1-972-971-1700 (Mobile)
> ----
> Blog: http://opensourceknowledge.blogspot.com/
>
>
> --
> View this message in context: http://old.nabble.com/Camel-2.3---Default-threading-profile---Your-view-tp27943985p27945489.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
Re: Camel 2.3 - Default threading profile - Your view
Posted by Ashwin Karpe <as...@progress.com>.
Hi Claus,
I do like the settings as initial values which could be potentially
overridden by the components. I also agree with the workQueue filling up
being the trigger to add more threads.
It would be great if all components could uniformly apply these initial
values which could ten be changed by the application/component users
themselves. This will prevent components from starving each other out with
their individual settings
Hope this helps.
Cheers,
Ashwin...
Claus Ibsen-2 wrote:
>
> Hi
>
> I am posting this to user so Camel end users have a chance to make an
> impact on this matter.
>
>
> I am working on making configuring thread pools in Camel much easier.
> https://issues.apache.org/activemq/browse/CAMEL-1588
>
> In light of this work we are introducing a ThreadPoolProfile which
> contains settings to be used when a thread pool is to be created.
>
> The idea currently is that there is a *default* profile that is used
> normally, and its has the following options
> private Integer poolSize = 10;
> private Integer maxPoolSize = 20;
> private Long keepAliveTime = 60L;
> private TimeUnit timeUnit = TimeUnit.SECONDS;
> private Integer maxQueueSize = 1000;
> private ThreadPoolRejectedPolicy rejectedPolicy;
>
> And the rejectedPolicy will default to use: CallerRuns. There are 4
> options: Abort, CallerRuns, DiscardOldest, Discard;
>
>
> I am currently wondering if these default settings is matching what
> you would expect / like to be default in Camel 2.3 onwards.
> How this works in Java is that the thread pool will have as min 10
> threads in the pool and a worker queue with a 1000 capacity.
> And only when its exhausted, eg the capacity has been hit, it will
> grow the pool up till 20 threads to help.
> And if threads is idle for more than 60 seconds it will be terminated.
>
> There are two options which I may want to adjust change
> - maxQueueSize = 1000
> - rejectedPolicy = CallerRuns
>
> Is the default capacity of 1000 to high? For example having 100, will
> let the thread pool to grow sooner to help out.
> Or is 100 to low. Also mind that the task queue is in memory so if the
> server crashes those tasks are gone.
>
> And the rejectedPolicy with CallerRuns allows to let the caller thread
> execute the task instead of handing it over to the thread pool.
> This is like a last resort if the thread pool is really exhausted and
> otherwise would have to either abort or discard a task.
> However the good point is that by doing this its automatically
> throttling the caller as it cannot at the same time send new tasks to
> the thread pool.
>
> We could also add a new option: Wait. Which lets the caller wait a bit
> to see if the task queue has room to handover the task. But when
> waiting you
> have to cater for timeout to avoid potentially waiting for a long
> time. And this requires custom code to write, as the existing 4
> options is provided
> out of the box from JDK.
>
>
> So when are these thread pools being used?
> - EIPs (splitter, aggregator, multicast, wiretap etc. etc.)
> - ProducerTemplate having a cached pool of Producer
> - Threads DSL
> - toAsync DSL
>
> The various components may have their own thread pool, such as Jetty,
> Mina, Spring JMS etc.
>
>
> Any thoughts?
>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
-----
---
Ashwin Karpe, Principal Consultant, PS - Opensource Center of Competence
Progress Software Corporation
14 Oak Park Drive
Bedford, MA 01730
---
+1-972-304-9084 (Office)
+1-972-971-1700 (Mobile)
----
Blog: http://opensourceknowledge.blogspot.com/
--
View this message in context: http://old.nabble.com/Camel-2.3---Default-threading-profile---Your-view-tp27943985p27945489.html
Sent from the Camel - Users mailing list archive at Nabble.com.