Re: TaskManager thread limit

[Peter Firmstone <ji...@zeus.net.au>]

Patricia Shanahan wrote:
> In addition to a limit related to the number of runnable tasks, a 
> TaskManager has a hard limit on the number of threads it will create.
>
> The parameterless constructor has limit 10, and most other uses have 
> compile time limits in the range of 10 to 15 threads.
>
> com.sun.jini.reggie.RegisrarImpl has a compile time limit of 50 threads.
>
> com.sun.jini.mahalo.TxnManagerImpl creates two pools, settlerpool and 
> taskpool. The settlerpool has limit 150 threads. The taskpool has 
> limit 50 threads.
>
> Even 150 threads could be low in a large server, especially if the 
> threads are used to wait for anything, so that each thread does not 
> need a hardware thread for a significant fraction of its life.
>
> As noted in the NIO vs. IO discussion that Peter pointed out, the key 
> to getting good performance simply is to take advantage of the fact 
> that an idle thread is a cheap, simple way to remember the state of 
> some activity.
>
> There is one approach that would minimize the number of changes but 
> increase flexibility. We could redefine the maximum thread count as 
> being the maximum number of threads per X, where X is a measure of 
> system size with a minimum of 1, but increasing on large systems.
>
> X could be based on the number of processors, the maximum heap memory, 
> or some combination.

I'm in favour of this suggestion.

Peter.

>
> Patricia
>

Re: TaskManager thread limit

[Patricia Shanahan <pa...@acm.org>]

On 7/31/2010 9:05 PM, Peter Firmstone wrote:
> Patricia Shanahan wrote:
>> In addition to a limit related to the number of runnable tasks, a
>> TaskManager has a hard limit on the number of threads it will create.
>>
>> The parameterless constructor has limit 10, and most other uses have
>> compile time limits in the range of 10 to 15 threads.
>>
>> com.sun.jini.reggie.RegisrarImpl has a compile time limit of 50 threads.
>>
>> com.sun.jini.mahalo.TxnManagerImpl creates two pools, settlerpool and
>> taskpool. The settlerpool has limit 150 threads. The taskpool has
>> limit 50 threads.
>>
>> Even 150 threads could be low in a large server, especially if the
>> threads are used to wait for anything, so that each thread does not
>> need a hardware thread for a significant fraction of its life.
>>
>> As noted in the NIO vs. IO discussion that Peter pointed out, the key
>> to getting good performance simply is to take advantage of the fact
>> that an idle thread is a cheap, simple way to remember the state of
>> some activity.
>>
>> There is one approach that would minimize the number of changes but
>> increase flexibility. We could redefine the maximum thread count as
>> being the maximum number of threads per X, where X is a measure of
>> system size with a minimum of 1, but increasing on large systems.
>>
>> X could be based on the number of processors, the maximum heap memory,
>> or some combination.
>
> I'm in favour of this suggestion.

As a further refinement, perhaps we could make the value of X an 
optional configuration parameter, and only calculate it from the 
processor and memory data if it is not specified. As you have already 
pointed out, we don't have effective access to very large systems, so 
any setting we do for the largest systems will untested theory.

Patricia

Re: TaskManager thread limit

[Dan Rollo <da...@gmail.com>]

On 08/01/2010 10:00 PM, Patricia Shanahan wrote:
> Gregg Wonderly wrote:
>> Peter Firmstone wrote:
>>> Patricia Shanahan wrote:
>>>> In addition to a limit related to the number of runnable tasks, a
>>>> TaskManager has a hard limit on the number of threads it will create.
>>>>
>>>> The parameterless constructor has limit 10, and most other uses have
>>>> compile time limits in the range of 10 to 15 threads.
>>>>
>>>> com.sun.jini.reggie.RegisrarImpl has a compile time limit of 50
>>>> threads.
>>>>
>>>> com.sun.jini.mahalo.TxnManagerImpl creates two pools, settlerpool
>>>> and taskpool. The settlerpool has limit 150 threads. The taskpool
>>>> has limit 50 threads.
>>>>
>>>> Even 150 threads could be low in a large server, especially if the
>>>> threads are used to wait for anything, so that each thread does not
>>>> need a hardware thread for a significant fraction of its life.
>>>>
>>>> As noted in the NIO vs. IO discussion that Peter pointed out, the
>>>> key to getting good performance simply is to take advantage of the
>>>> fact that an idle thread is a cheap, simple way to remember the
>>>> state of some activity.
>>>>
>>>> There is one approach that would minimize the number of changes but
>>>> increase flexibility. We could redefine the maximum thread count as
>>>> being the maximum number of threads per X, where X is a measure of
>>>> system size with a minimum of 1, but increasing on large systems.
>>>>
>>>> X could be based on the number of processors, the maximum heap
>>>> memory, or some combination.
>>>
>>> I'm in favour of this suggestion.
>>
>> Auto sizing is good, but we should also consider putting in logging
>> that announces when the limit is reached and waiting will occur. This
>> will help people diagnose the potential system pauses, if not deadlock
>> that will be outwardly visible.
>
> Agreed. I would be interested both in the distribution of times that
> tasks spend waiting for other tasks and in the distribution of times
> that they spend as ready tasks, waiting for a thread.
>
>> On top of that, I'd be strongly in favor the introduction of JMX as an
>> external observation and management capability for these types of values.
>
> I don't know much about JMX, but I strongly agree with external
> observation and management.
>
> Incidentally, do you know why the default load level is 3, not 1?
>
> Patricia

Re: JMX - Not even sure this applies, but I remembered the JMX info 
below, and wanted to post it:

https://jmx-lookup.dev.java.net/public/index.html

Dan

Re: TaskManager thread limit

[Gregg Wonderly <gr...@wonderly.org>]

Patricia Shanahan wrote:
> Gregg Wonderly wrote:
>> Patricia Shanahan wrote:
> ...
>>> Incidentally, do you know why the default load level is 3, not 1?
>>
>> I am not sure which load level you are referring to.
> ...
> 
> Sorry for the lack of context. I meant the third parameter on the main
> TaskManager constructor:
> 
> loadFactor
> 
> threshold for creating new threads. A new thread is created if the total
> number of runnable tasks (both active and pending) exceeds the number of
> threads times the loadFactor, and the maximum number of threads has not
> been reached.
> 
> The parameterless constructor sets it to 3.0. I can see some special
> cases for a value greater than 1, but would have expected the default to
> be 1. That is, create another thread if the current number of threads is
> less than the maximum and there is at least one task that could be
> running but is waiting for a thread to be available.

Okay, I am not sure of the history of that decision.  I'm with you, that a value 
of 1 would seem the better default, but I am sure that there were system load 
and other issues that happened to affect that when machines were "small" and 
"single processor".

Gregg

Re: TaskManager thread limit

[Patricia Shanahan <pa...@acm.org>]

Gregg Wonderly wrote:
> Patricia Shanahan wrote:
...
>> Incidentally, do you know why the default load level is 3, not 1?
> 
> I am not sure which load level you are referring to.
...

Sorry for the lack of context. I meant the third parameter on the main
TaskManager constructor:

loadFactor

threshold for creating new threads. A new thread is created if the total
number of runnable tasks (both active and pending) exceeds the number of
threads times the loadFactor, and the maximum number of threads has not
been reached.

The parameterless constructor sets it to 3.0. I can see some special
cases for a value greater than 1, but would have expected the default to
be 1. That is, create another thread if the current number of threads is
less than the maximum and there is at least one task that could be
running but is waiting for a thread to be available.

Patricia

Re: TaskManager thread limit

[Gregg Wonderly <gr...@wonderly.org>]

Patricia Shanahan wrote:
> Gregg Wonderly wrote:
>> Peter Firmstone wrote:
>>> Patricia Shanahan wrote:
>>>> In addition to a limit related to the number of runnable tasks, a 
>>>> TaskManager has a hard limit on the number of threads it will create.
>>>>
>>>> The parameterless constructor has limit 10, and most other uses have 
>>>> compile time limits in the range of 10 to 15 threads.
>>>>
>>>> com.sun.jini.reggie.RegisrarImpl has a compile time limit of 50 
>>>> threads.
>>>>
>>>> com.sun.jini.mahalo.TxnManagerImpl creates two pools, settlerpool 
>>>> and taskpool. The settlerpool has limit 150 threads. The taskpool 
>>>> has limit 50 threads.
>>>>
>>>> Even 150 threads could be low in a large server, especially if the 
>>>> threads are used to wait for anything, so that each thread does not 
>>>> need a hardware thread for a significant fraction of its life.
>>>>
>>>> As noted in the NIO vs. IO discussion that Peter pointed out, the 
>>>> key to getting good performance simply is to take advantage of the 
>>>> fact that an idle thread is a cheap, simple way to remember the 
>>>> state of some activity.
>>>>
>>>> There is one approach that would minimize the number of changes but 
>>>> increase flexibility. We could redefine the maximum thread count as 
>>>> being the maximum number of threads per X, where X is a measure of 
>>>> system size with a minimum of 1, but increasing on large systems.
>>>>
>>>> X could be based on the number of processors, the maximum heap 
>>>> memory, or some combination.
>>>
>>> I'm in favour of this suggestion.
>>
>> Auto sizing is good, but we should also consider putting in logging 
>> that announces when the limit is reached and waiting will occur.  This 
>> will help people diagnose the potential system pauses, if not deadlock 
>> that will be outwardly visible.
> 
> Agreed. I would be interested both in the distribution of times that 
> tasks spend waiting for other tasks and in the distribution of times 
> that they spend as ready tasks, waiting for a thread.
> 
>> On top of that, I'd be strongly in favor the introduction of JMX as an 
>> external observation and management capability for these types of values.
> 
> I don't know much about JMX, but I strongly agree with external 
> observation and management.

Also, just FYI, the Flavio project, http://flavio.dev.java.net/, was about 
creating a mechanism to provide the ability to "lookup" a JMX accessible entity. 
  I've used this, but there are things about it that I am not fond of.  In 
particular, it creates a rather large URL for jmx access because of encoding the 
lookup information into the URL as a base-64 (I think) string representation of 
the MarshalledObject.

This would be something to look at though, because it does provide a bridge of 
sorts between JMX client software and Jini services.

> Incidentally, do you know why the default load level is 3, not 1?

I am not sure which load level you are referring to.

Gregg

> Patricia
> 

Re: TaskManager thread limit

[Patricia Shanahan <pa...@acm.org>]

Gregg Wonderly wrote:
> Peter Firmstone wrote:
>> Patricia Shanahan wrote:
>>> In addition to a limit related to the number of runnable tasks, a 
>>> TaskManager has a hard limit on the number of threads it will create.
>>>
>>> The parameterless constructor has limit 10, and most other uses have 
>>> compile time limits in the range of 10 to 15 threads.
>>>
>>> com.sun.jini.reggie.RegisrarImpl has a compile time limit of 50 threads.
>>>
>>> com.sun.jini.mahalo.TxnManagerImpl creates two pools, settlerpool and 
>>> taskpool. The settlerpool has limit 150 threads. The taskpool has 
>>> limit 50 threads.
>>>
>>> Even 150 threads could be low in a large server, especially if the 
>>> threads are used to wait for anything, so that each thread does not 
>>> need a hardware thread for a significant fraction of its life.
>>>
>>> As noted in the NIO vs. IO discussion that Peter pointed out, the key 
>>> to getting good performance simply is to take advantage of the fact 
>>> that an idle thread is a cheap, simple way to remember the state of 
>>> some activity.
>>>
>>> There is one approach that would minimize the number of changes but 
>>> increase flexibility. We could redefine the maximum thread count as 
>>> being the maximum number of threads per X, where X is a measure of 
>>> system size with a minimum of 1, but increasing on large systems.
>>>
>>> X could be based on the number of processors, the maximum heap 
>>> memory, or some combination.
>>
>> I'm in favour of this suggestion.
> 
> Auto sizing is good, but we should also consider putting in logging that 
> announces when the limit is reached and waiting will occur.  This will 
> help people diagnose the potential system pauses, if not deadlock that 
> will be outwardly visible.

Agreed. I would be interested both in the distribution of times that 
tasks spend waiting for other tasks and in the distribution of times 
that they spend as ready tasks, waiting for a thread.

> On top of that, I'd be strongly in favor the introduction of JMX as an 
> external observation and management capability for these types of values.

I don't know much about JMX, but I strongly agree with external 
observation and management.

Incidentally, do you know why the default load level is 3, not 1?

Patricia

Re: TaskManager thread limit

[Gregg Wonderly <gr...@wonderly.org>]

Peter Firmstone wrote:
> Patricia Shanahan wrote:
>> In addition to a limit related to the number of runnable tasks, a 
>> TaskManager has a hard limit on the number of threads it will create.
>>
>> The parameterless constructor has limit 10, and most other uses have 
>> compile time limits in the range of 10 to 15 threads.
>>
>> com.sun.jini.reggie.RegisrarImpl has a compile time limit of 50 threads.
>>
>> com.sun.jini.mahalo.TxnManagerImpl creates two pools, settlerpool and 
>> taskpool. The settlerpool has limit 150 threads. The taskpool has 
>> limit 50 threads.
>>
>> Even 150 threads could be low in a large server, especially if the 
>> threads are used to wait for anything, so that each thread does not 
>> need a hardware thread for a significant fraction of its life.
>>
>> As noted in the NIO vs. IO discussion that Peter pointed out, the key 
>> to getting good performance simply is to take advantage of the fact 
>> that an idle thread is a cheap, simple way to remember the state of 
>> some activity.
>>
>> There is one approach that would minimize the number of changes but 
>> increase flexibility. We could redefine the maximum thread count as 
>> being the maximum number of threads per X, where X is a measure of 
>> system size with a minimum of 1, but increasing on large systems.
>>
>> X could be based on the number of processors, the maximum heap memory, 
>> or some combination.
> 
> I'm in favour of this suggestion.

Auto sizing is good, but we should also consider putting in logging that 
announces when the limit is reached and waiting will occur.  This will help 
people diagnose the potential system pauses, if not deadlock that will be 
outwardly visible.

On top of that, I'd be strongly in favor the introduction of JMX as an external 
observation and management capability for these types of values.

Gregg Wonderly