Re: [Fwd: Re: DGC threads issue]

[Peter Jones <pc...@roundroom.net>]

Peter,

That internal Executor interface permits neither synchronous execution in the calling thread ("this method itself must not block") nor any small bound on the number of Runnables that can be executed concurrently ("execution of a given action must not be delayed indefinitely in order to complete execution of a different action passed to a different invocation of this method").  In other words, concurrency-wise, it should be equivalent to "new Thread(runnable, name)", or what the result of j.u.c.Executors.newCachedThreadPool() would do.  Otherwise, callers would not function correctly, because they expect to be able to pass Runnables that execute indefinitely, not just short-lived work tasks.

The real purpose of this internal Executor/ThreadPool was to provide an internal alternative to "new Thread(...)" that takes care of certain things in common, like:

- thread reuse instead of creation when reasonable (idle timeout)
- but still with threads (re)named for the task being executed
- insulation of any thread creation permission requirement from context calling execute (NewThreadAction)
- but instead requiring direct user of thread pool to be trusted not to abuse that insulation, with permission requirement for access (ThreadPoolPermission and GetThreadPoolAction)
- common logging of uncaught exceptions

-- Peter


On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:

> ThreadPool implements the com.sun.jini.thread.Executor interface.
> 
> Because the interface states it should not block, I think the calling thread should execute the task when the thread pool becomes saturated, rather than continue to create new threads as per the current implementation.  This will ensure that the task is completed, the calling thread, if it uses a sequence of tasks with dependencies will have to ensure that it submits the tasks in order.  Since only the caller knows the order, it makes sense for this to be the callers responsibility.  As a result the executor will honor the non blocking contract.  For that reason we'll use a zero length queue, probably a SynchronousQueue.
> 
> Regards,
> 
> Peter.
> 
> /**
> * Executor is an abstraction for a thread factory or thread pool for
> * executing actions asynchronously.
> *
> * @author    Sun Microsystems, Inc.
> *
> */
> public interface Executor {
> 
>   /**
>    * Executes the given Runnable action asynchronously in some thread.
>    *
>    * The implementation may create a new thread to execute the action,
>    * or it may execute the action in an existing thread.
>    *
>    * The execution of a given action must not be delayed indefinitely
>    * in order to complete execution of a different action passed to a
>    * different invocation of this method.  In other words, the
>    * implementation must assume that there may be arbitrary dependencies
>    * between actions passed to this method, so it needs to be careful
>    * to avoid potential deadlock by delaying execution of one action
>    * indefinitely until another completes.
>    *
>    * Also, this method itself must not block, because it may be invoked
>    * by code that is serially processing data to produce multiple such
>    * arbitrarily-dependent actions that need to be executed.
>    *
>    * @param    runnable the Runnable action to execute
>    *
>    * @param    name string to include in the name of the thread used
>    * to execute the action
>    */
>   void execute(Runnable runnable, String name);
> }
> 
> Peter Firmstone wrote:
>> Thanks Brian,
>> 
>> Looking at our implementation code, DGC uses an Executor called ThreadPool, it's javadoc states:
>> 
>> /**
>> * ThreadPool is a simple thread pool implementation of the Executor
>> * interface.
>> *
>> * A new task is always given to an idle thread, if one is available;
>> * otherwise, a new thread is always created.  There is no minimum
>> * warm thread count, nor is there a maximum thread count (tasks are
>> * never queued unless there are sufficient idle threads to execute
>> * them).
>> *
>> * New threads are created as daemon threads in the thread group that
>> * was passed to the ThreadPool instance's constructor.  Each thread's
>> * name is the prefix NewThreadAction.NAME_PREFIX followed by the name
>> * of the task it is currently executing, or "Idle" if it is currently
>> * idle.
>> 
>> ThreadPool predates Java 5, it looks like we can fix this by using an Executor from Java 5, we can look at limiting the number of threads created based on available CPU's and a scaling factor and place the tasks in a BlockingQueue, so if the queue is filled, it blocks.
>> 
>> Can you report the issue as a Bug on Jira for me, I'll fix this before the next release.
>> 
>> Regards,
>> 
>> Peter.
>> 
>> Peter Firmstone wrote:
>>> Hi Peter,
>>> 
>>> I was wondering if you had any thoughts on this post from Bryan on River users?
>>> 
>>> Hope you don't mind me asking ;)
>>> 
>>> Best Regards,
>>> 
>>> Peter Firmstone.
>>> 
>>> ------------------------------------------------------------------------
>>> 
>>> Subject:
>>> Re: DGC threads issue
>>> From:
>>> Tom Hobbs <tv...@googlemail.com>
>>> Date:
>>> Thu, 12 Jan 2012 20:45:01 +0000
>>> To:
>>> user@river.apache.org, dev@river.apache.org
>>> 
>>> To:
>>> user@river.apache.org, dev@river.apache.org
>>> 
>>> 
>>> Hi Bryan,
>>> 
>>> Sorry that no one got back to you about this.  I'm afraid that I don't
>>> know the answer to your question, I've copied the dev list into this
>>> email in case someone who monitors that list (but not this one) has
>>> any ideas.
>>> 
>>> Best regards,
>>> 
>>> Tom
>>> 
>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson <br...@systap.com> wrote:
>>> 
>>>> Just to follow up on this thread myself.  I modified the pattern to return a "thick" future rather than a proxy for the future.  This caused the RMI call to wait on the server until the future was done and then sent back the outcome.  This "fixed" the DGC memory/thread leak by reducing the number of exported proxies drammatically.
>>>> 
>>>> In terms of best practices, is distributed DGC simply not useful for exported objects with short life spans?  Can it only be used with proxies for relatively long lived services?
>>>> 
>>>> Thanks,
>>>> Bryan
>>>> 
>>>>   
>>>>> -----Original Message-----
>>>>> From: Bryan Thompson
>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
>>>>> To: user@river.apache.org
>>>>> Subject: DGC threads issue
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> Background:
>>>>> 
>>>>> I am seeing what would appear to be one DGC thread allocated
>>>>> per exported object.  This is using River 2.2 and Sun JDK
>>>>> 1.6.0_17.  Relevant configuration parameters are below.
>>>>> 
>>>>> I am observing problems with the DGC threads not being
>>>>> retired on a timely basis.  The exported objects are proxies
>>>>> for Futures which are being executed on the service.  The
>>>>> code pattern is such that the proxied Future goes out of
>>>>> lexical scope quite quickly.  E.g.,
>>>>> rmiCallReturningProxyForFuture().get().
>>>>> 
>>>>> Under a modest load, a large number of such Futures are
>>>>> exported which results in a large number of long lived DGC
>>>>> threads.  This turns into a problem for the JVM due to the
>>>>> stack allocation per thread.  Presumably this is not good for
>>>>> other reasons as well (e.g., scheduling).
>>>>> 
>>>>> I have tried to override the leaseValue and checkInterval
>>>>> defaults per the configuration options below.  I suspect that
>>>>> the lease interval is somehow not being obeyed, which is
>>>>> presumably a problem on my end.  However, I can verify that
>>>>> the configuration values are in fact showing up in
>>>>> System.getProperties() for at least some of the JVMs involved
>>>>> (the one which drives the workload and the one that I am
>>>>> monitoring with the large number of DGC lease threads).
>>>>> 
>>>>> Some questions:
>>>>> 
>>>>> Is this one-thread-per-exported proxy the expected behavior
>>>>> when DGC is requested for the exported object?
>>>>> 
>>>>> The DGC lease checker threads appear to expire ~14 - 15
>>>>> minutes after I terminate the process which was originating
>>>>> the RMI requests.  This is close the sum of the default
>>>>> leaseValue (10m) and checkInterval (5m) parameters, but maybe
>>>>> there is some other timeout which is controlling this?  If
>>>>> this is the sum of those parameters, why would the DGC lease
>>>>> threads live until the sum of those values?  I thought that
>>>>> the lease would expire after the leaseValue (10m default).
>>>>> 
>>>>> Can the issue I am observing be caused by a low heap pressure
>>>>> on the JVM to which the RMI proxies were exported?  If it
>>>>> fails to GC those proxies, even though they are reachable,
>>>>> could that cause DGC to continue to retain those proxies on
>>>>> the JVM which exported them?
>>>>> 
>>>>> Is there any way to configure DGC to use a thread pool or to
>>>>> have the leases managed by a single thread?
>>>>> 
>>>>> Is it possible that there is an interaction with the useNIO option?
>>>>> 
>>>>> Relevant options that I am using include:
>>>>> 
>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
>>>>>   -Djava.rmi.dgc.leaseValue=30000
>>>>>   -Dsun.rmi.dgc.checkInterval=15000
>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
>>>>> 
>>>>> Thanks in advance,
>>>>> Bryan
>>>>>      
>>> 
>>>  
>> 
>> 
> 

RE: RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

Tom,

Ok.  I will fetch the code for 2.2 and work through the build process.  I have filed an issue for this [1].  If someone can post a patch for that issue against 2.2, I will build the modified JARs and verify whether the problem is resolved by that change.

Thanks,
Bryan

PS: River 2.2 is listed as an "Unreleased Version" in Jira.

[1] https://issues.apache.org/jira/browse/RIVER-403

> -----Original Message-----
> From: Tom Hobbs [mailto:tvhobbs@googlemail.com] 
> Sent: Friday, January 13, 2012 7:35 AM
> To: user@river.apache.org
> Subject: Re: RE: [Fwd: Re: DGC threads issue]
> 
> The easiest way would be to the source distribution of River. 
>  Unpack it somewhere.  Modify the code and then run the ant 
> task called "build-all" (I think, whatever it's called it 
> default target for the build.xml).  The build file should be 
> in the root of the unpacked source.
> 
> I don't think Peter created a jira fir it because I've not 
> seen a notification email.  I'm also unsure what code libel 
> he's referring to.
> 
> Sent via mobile device, please forgive typos and spacing errors.
> 
> On 13 Jan 2012 12:30, "Bryan Thompson" <br...@systap.com> wrote:
> 
> > Peter,
> >
> > I would be happy to test out a fix on this.  Can you point out the 
> > modified line and link me to any directions on how to rebuild the 
> > necessary jar(s)?
> >
> > Did you already file an issue for this?  If not, I will 
> file an issue 
> > and then update it when I test out this fix.
> >
> > Thanks,
> > Bryan
> >
> > > -----Original Message-----
> > > From: Peter Firmstone [mailto:jini@zeus.net.au]
> > > Sent: Friday, January 13, 2012 3:21 AM
> > > To: dev@river.apache.org
> > > Cc: Bryan Thompson; user@river.apache.org
> > > Subject: Re: [Fwd: Re: DGC threads issue]
> > >
> > > Ok, found the problem, when I fixed River-142, I introduced a new 
> > > bug, after creating a new DGC thread, I didn't set the 
> running flag 
> > > to true, so each time, a new thread is created.
> > >
> > > Thankfully it's a very easy fix.
> > >
> > > Cheers,
> > >
> > > Peter.
> > >
> > > Peter Jones wrote:
> > > > Peter,
> > > >
> > > > That internal Executor interface permits neither
> > > synchronous execution in the calling thread ("this method itself 
> > > must not block") nor any small bound on the number of 
> Runnables that 
> > > can be executed concurrently ("execution of a given 
> action must not 
> > > be delayed indefinitely in order to complete execution of a 
> > > different action passed to a different invocation of this 
> method").  
> > > In other words, concurrency-wise, it should be equivalent to "new 
> > > Thread(runnable, name)", or what the result of
> > > j.u.c.Executors.newCachedThreadPool() would do.  
> Otherwise, callers 
> > > would not function correctly, because they expect to be 
> able to pass 
> > > Runnables that execute indefinitely, not just short-lived work 
> > > tasks.
> > > >
> > > > The real purpose of this internal Executor/ThreadPool was
> > > to provide an internal alternative to "new Thread(...)" 
> that takes 
> > > care of certain things in common, like:
> > > >
> > > > - thread reuse instead of creation when reasonable 
> (idle timeout)
> > > > - but still with threads (re)named for the task being executed
> > > > - insulation of any thread creation permission requirement from 
> > > > context calling execute (NewThreadAction)
> > > > - but instead requiring direct user of thread pool to be
> > > trusted not
> > > > to abuse that insulation, with permission requirement 
> for access 
> > > > (ThreadPoolPermission and GetThreadPoolAction)
> > > > - common logging of uncaught exceptions
> > > >
> > > > -- Peter
> > > >
> > > >
> > > > On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> > > >
> > > >
> > > >> ThreadPool implements the com.sun.jini.thread.Executor 
> interface.
> > > >>
> > > >> Because the interface states it should not block, I think
> > > the calling thread should execute the task when the thread pool 
> > > becomes saturated, rather than continue to create new 
> threads as per 
> > > the current implementation.  This will ensure that the task is 
> > > completed, the calling thread, if it uses a sequence of 
> tasks with 
> > > dependencies will have to ensure that it submits the 
> tasks in order.  
> > > Since only the caller knows the order, it makes sense for 
> this to be 
> > > the callers responsibility.  As a result the executor 
> will honor the 
> > > non blocking contract.  For that reason we'll use a zero length 
> > > queue, probably a SynchronousQueue.
> > > >>
> > > >> Regards,
> > > >>
> > > >> Peter.
> > > >>
> > > >> /**
> > > >> * Executor is an abstraction for a thread factory or
> > > thread pool for
> > > >> * executing actions asynchronously.
> > > >> *
> > > >> * @author    Sun Microsystems, Inc.
> > > >> *
> > > >> */
> > > >> public interface Executor {
> > > >>
> > > >>   /**
> > > >>    * Executes the given Runnable action asynchronously in
> > > some thread.
> > > >>    *
> > > >>    * The implementation may create a new thread to execute
> > > the action,
> > > >>    * or it may execute the action in an existing thread.
> > > >>    *
> > > >>    * The execution of a given action must not be delayed
> > > indefinitely
> > > >>    * in order to complete execution of a different action
> > > passed to a
> > > >>    * different invocation of this method.  In other words, the
> > > >>    * implementation must assume that there may be
> > > arbitrary dependencies
> > > >>    * between actions passed to this method, so it needs to
> > > be careful
> > > >>    * to avoid potential deadlock by delaying execution of
> > > one action
> > > >>    * indefinitely until another completes.
> > > >>    *
> > > >>    * Also, this method itself must not block, because it
> > > may be invoked
> > > >>    * by code that is serially processing data to produce
> > > multiple such
> > > >>    * arbitrarily-dependent actions that need to be executed.
> > > >>    *
> > > >>    * @param    runnable the Runnable action to execute
> > > >>    *
> > > >>    * @param    name string to include in the name of the
> > > thread used
> > > >>    * to execute the action
> > > >>    */
> > > >>   void execute(Runnable runnable, String name); }
> > > >>
> > > >> Peter Firmstone wrote:
> > > >>
> > > >>> Thanks Brian,
> > > >>>
> > > >>> Looking at our implementation code, DGC uses an Executor
> > > called ThreadPool, it's javadoc states:
> > > >>>
> > > >>> /**
> > > >>> * ThreadPool is a simple thread pool implementation of
> > > the Executor
> > > >>> * interface.
> > > >>> *
> > > >>> * A new task is always given to an idle thread, if one is
> > > available;
> > > >>> * otherwise, a new thread is always created.  There is no 
> > > >>> minimum
> > > >>> * warm thread count, nor is there a maximum thread count
> > > (tasks are
> > > >>> * never queued unless there are sufficient idle threads to 
> > > >>> execute
> > > >>> * them).
> > > >>> *
> > > >>> * New threads are created as daemon threads in the thread
> > > group that
> > > >>> * was passed to the ThreadPool instance's constructor.  Each 
> > > >>> thread's
> > > >>> * name is the prefix NewThreadAction.NAME_PREFIX 
> followed by the 
> > > >>> name
> > > >>> * of the task it is currently executing, or "Idle" if it is 
> > > >>> currently
> > > >>> * idle.
> > > >>>
> > > >>> ThreadPool predates Java 5, it looks like we can fix this
> > > by using an Executor from Java 5, we can look at limiting 
> the number 
> > > of threads created based on available CPU's and a scaling 
> factor and 
> > > place the tasks in a BlockingQueue, so if the queue is filled, it 
> > > blocks.
> > > >>>
> > > >>> Can you report the issue as a Bug on Jira for me, I'll
> > > fix this before the next release.
> > > >>>
> > > >>> Regards,
> > > >>>
> > > >>> Peter.
> > > >>>
> > > >>> Peter Firmstone wrote:
> > > >>>
> > > >>>> Hi Peter,
> > > >>>>
> > > >>>> I was wondering if you had any thoughts on this post
> > > from Bryan on River users?
> > > >>>>
> > > >>>> Hope you don't mind me asking ;)
> > > >>>>
> > > >>>> Best Regards,
> > > >>>>
> > > >>>> Peter Firmstone.
> > > >>>>
> > > >>>>
> > > 
> -------------------------------------------------------------------
> > > >>>> -----
> > > >>>>
> > > >>>> Subject:
> > > >>>> Re: DGC threads issue
> > > >>>> From:
> > > >>>> Tom Hobbs <tv...@googlemail.com>
> > > >>>> Date:
> > > >>>> Thu, 12 Jan 2012 20:45:01 +0000
> > > >>>> To:
> > > >>>> user@river.apache.org, dev@river.apache.org
> > > >>>>
> > > >>>> To:
> > > >>>> user@river.apache.org, dev@river.apache.org
> > > >>>>
> > > >>>>
> > > >>>> Hi Bryan,
> > > >>>>
> > > >>>> Sorry that no one got back to you about this.  I'm 
> afraid that 
> > > >>>> I don't know the answer to your question, I've 
> copied the dev 
> > > >>>> list into this email in case someone who monitors that list 
> > > >>>> (but not this one) has any ideas.
> > > >>>>
> > > >>>> Best regards,
> > > >>>>
> > > >>>> Tom
> > > >>>>
> > > >>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> > > <br...@systap.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>>> Just to follow up on this thread myself.  I modified
> > > the pattern to return a "thick" future rather than a 
> proxy for the 
> > > future.  This caused the RMI call to wait on the server until the 
> > > future was done and then sent back the outcome.  This "fixed" the 
> > > DGC memory/thread leak by reducing the number of exported proxies 
> > > drammatically.
> > > >>>>>
> > > >>>>> In terms of best practices, is distributed DGC simply
> > > not useful for exported objects with short life spans?  
> Can it only 
> > > be used with proxies for relatively long lived services?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Bryan
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>> -----Original Message-----
> > > >>>>>> From: Bryan Thompson
> > > >>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> > > >>>>>> To: user@river.apache.org
> > > >>>>>> Subject: DGC threads issue
> > > >>>>>>
> > > >>>>>> Hello,
> > > >>>>>>
> > > >>>>>> Background:
> > > >>>>>>
> > > >>>>>> I am seeing what would appear to be one DGC thread
> > > allocated per
> > > >>>>>> exported object.  This is using River 2.2 and Sun JDK
> > > 1.6.0_17.
> > > >>>>>> Relevant configuration parameters are below.
> > > >>>>>>
> > > >>>>>> I am observing problems with the DGC threads not being
> > > retired on
> > > >>>>>> a timely basis.  The exported objects are proxies 
> for Futures 
> > > >>>>>> which are being executed on the service.  The code 
> pattern is 
> > > >>>>>> such that the proxied Future goes out of lexical 
> scope quite 
> > > >>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> > > >>>>>>
> > > >>>>>> Under a modest load, a large number of such Futures
> > > are exported
> > > >>>>>> which results in a large number of long lived DGC
> > > threads.  This
> > > >>>>>> turns into a problem for the JVM due to the stack
> > > allocation per
> > > >>>>>> thread.  Presumably this is not good for other reasons as 
> > > >>>>>> well (e.g., scheduling).
> > > >>>>>>
> > > >>>>>> I have tried to override the leaseValue and checkInterval 
> > > >>>>>> defaults per the configuration options below.  I
> > > suspect that the
> > > >>>>>> lease interval is somehow not being obeyed, which is
> > > presumably a
> > > >>>>>> problem on my end.  However, I can verify that the
> > > configuration
> > > >>>>>> values are in fact showing up in
> > > >>>>>> System.getProperties() for at least some of the 
> JVMs involved 
> > > >>>>>> (the one which drives the workload and the one that I am 
> > > >>>>>> monitoring with the large number of DGC lease threads).
> > > >>>>>>
> > > >>>>>> Some questions:
> > > >>>>>>
> > > >>>>>> Is this one-thread-per-exported proxy the expected
> > > behavior when
> > > >>>>>> DGC is requested for the exported object?
> > > >>>>>>
> > > >>>>>> The DGC lease checker threads appear to expire ~14 -
> > > 15 minutes
> > > >>>>>> after I terminate the process which was 
> originating the RMI 
> > > >>>>>> requests.  This is close the sum of the default
> > > leaseValue (10m)
> > > >>>>>> and checkInterval (5m) parameters, but maybe there is
> > > some other
> > > >>>>>> timeout which is controlling this?  If this is the sum
> > > of those
> > > >>>>>> parameters, why would the DGC lease threads live until
> > > the sum of
> > > >>>>>> those values?  I thought that the lease would expire after 
> > > >>>>>> the leaseValue (10m default).
> > > >>>>>>
> > > >>>>>> Can the issue I am observing be caused by a low heap
> > > pressure on
> > > >>>>>> the JVM to which the RMI proxies were exported?  If it
> > > fails to
> > > >>>>>> GC those proxies, even though they are reachable, 
> could that 
> > > >>>>>> cause DGC to continue to retain those proxies on the JVM 
> > > >>>>>> which exported them?
> > > >>>>>>
> > > >>>>>> Is there any way to configure DGC to use a thread pool
> > > or to have
> > > >>>>>> the leases managed by a single thread?
> > > >>>>>>
> > > >>>>>> Is it possible that there is an interaction with the
> > > useNIO option?
> > > >>>>>>
> > > >>>>>> Relevant options that I am using include:
> > > >>>>>>
> > > >>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> > > >>>>>>   -Djava.rmi.dgc.leaseValue=30000
> > > >>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> > > >>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> > > >>>>>>
> > > >>>>>> Thanks in advance,
> > > >>>>>> Bryan
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >
> > > >
> > > >
> > >
> > >
> 

RE: RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

Tom,

Ok.  I will fetch the code for 2.2 and work through the build process.  I have filed an issue for this [1].  If someone can post a patch for that issue against 2.2, I will build the modified JARs and verify whether the problem is resolved by that change.

Thanks,
Bryan

PS: River 2.2 is listed as an "Unreleased Version" in Jira.

[1] https://issues.apache.org/jira/browse/RIVER-403

> -----Original Message-----
> From: Tom Hobbs [mailto:tvhobbs@googlemail.com] 
> Sent: Friday, January 13, 2012 7:35 AM
> To: user@river.apache.org
> Subject: Re: RE: [Fwd: Re: DGC threads issue]
> 
> The easiest way would be to the source distribution of River. 
>  Unpack it somewhere.  Modify the code and then run the ant 
> task called "build-all" (I think, whatever it's called it 
> default target for the build.xml).  The build file should be 
> in the root of the unpacked source.
> 
> I don't think Peter created a jira fir it because I've not 
> seen a notification email.  I'm also unsure what code libel 
> he's referring to.
> 
> Sent via mobile device, please forgive typos and spacing errors.
> 
> On 13 Jan 2012 12:30, "Bryan Thompson" <br...@systap.com> wrote:
> 
> > Peter,
> >
> > I would be happy to test out a fix on this.  Can you point out the 
> > modified line and link me to any directions on how to rebuild the 
> > necessary jar(s)?
> >
> > Did you already file an issue for this?  If not, I will 
> file an issue 
> > and then update it when I test out this fix.
> >
> > Thanks,
> > Bryan
> >
> > > -----Original Message-----
> > > From: Peter Firmstone [mailto:jini@zeus.net.au]
> > > Sent: Friday, January 13, 2012 3:21 AM
> > > To: dev@river.apache.org
> > > Cc: Bryan Thompson; user@river.apache.org
> > > Subject: Re: [Fwd: Re: DGC threads issue]
> > >
> > > Ok, found the problem, when I fixed River-142, I introduced a new 
> > > bug, after creating a new DGC thread, I didn't set the 
> running flag 
> > > to true, so each time, a new thread is created.
> > >
> > > Thankfully it's a very easy fix.
> > >
> > > Cheers,
> > >
> > > Peter.
> > >
> > > Peter Jones wrote:
> > > > Peter,
> > > >
> > > > That internal Executor interface permits neither
> > > synchronous execution in the calling thread ("this method itself 
> > > must not block") nor any small bound on the number of 
> Runnables that 
> > > can be executed concurrently ("execution of a given 
> action must not 
> > > be delayed indefinitely in order to complete execution of a 
> > > different action passed to a different invocation of this 
> method").  
> > > In other words, concurrency-wise, it should be equivalent to "new 
> > > Thread(runnable, name)", or what the result of
> > > j.u.c.Executors.newCachedThreadPool() would do.  
> Otherwise, callers 
> > > would not function correctly, because they expect to be 
> able to pass 
> > > Runnables that execute indefinitely, not just short-lived work 
> > > tasks.
> > > >
> > > > The real purpose of this internal Executor/ThreadPool was
> > > to provide an internal alternative to "new Thread(...)" 
> that takes 
> > > care of certain things in common, like:
> > > >
> > > > - thread reuse instead of creation when reasonable 
> (idle timeout)
> > > > - but still with threads (re)named for the task being executed
> > > > - insulation of any thread creation permission requirement from 
> > > > context calling execute (NewThreadAction)
> > > > - but instead requiring direct user of thread pool to be
> > > trusted not
> > > > to abuse that insulation, with permission requirement 
> for access 
> > > > (ThreadPoolPermission and GetThreadPoolAction)
> > > > - common logging of uncaught exceptions
> > > >
> > > > -- Peter
> > > >
> > > >
> > > > On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> > > >
> > > >
> > > >> ThreadPool implements the com.sun.jini.thread.Executor 
> interface.
> > > >>
> > > >> Because the interface states it should not block, I think
> > > the calling thread should execute the task when the thread pool 
> > > becomes saturated, rather than continue to create new 
> threads as per 
> > > the current implementation.  This will ensure that the task is 
> > > completed, the calling thread, if it uses a sequence of 
> tasks with 
> > > dependencies will have to ensure that it submits the 
> tasks in order.  
> > > Since only the caller knows the order, it makes sense for 
> this to be 
> > > the callers responsibility.  As a result the executor 
> will honor the 
> > > non blocking contract.  For that reason we'll use a zero length 
> > > queue, probably a SynchronousQueue.
> > > >>
> > > >> Regards,
> > > >>
> > > >> Peter.
> > > >>
> > > >> /**
> > > >> * Executor is an abstraction for a thread factory or
> > > thread pool for
> > > >> * executing actions asynchronously.
> > > >> *
> > > >> * @author    Sun Microsystems, Inc.
> > > >> *
> > > >> */
> > > >> public interface Executor {
> > > >>
> > > >>   /**
> > > >>    * Executes the given Runnable action asynchronously in
> > > some thread.
> > > >>    *
> > > >>    * The implementation may create a new thread to execute
> > > the action,
> > > >>    * or it may execute the action in an existing thread.
> > > >>    *
> > > >>    * The execution of a given action must not be delayed
> > > indefinitely
> > > >>    * in order to complete execution of a different action
> > > passed to a
> > > >>    * different invocation of this method.  In other words, the
> > > >>    * implementation must assume that there may be
> > > arbitrary dependencies
> > > >>    * between actions passed to this method, so it needs to
> > > be careful
> > > >>    * to avoid potential deadlock by delaying execution of
> > > one action
> > > >>    * indefinitely until another completes.
> > > >>    *
> > > >>    * Also, this method itself must not block, because it
> > > may be invoked
> > > >>    * by code that is serially processing data to produce
> > > multiple such
> > > >>    * arbitrarily-dependent actions that need to be executed.
> > > >>    *
> > > >>    * @param    runnable the Runnable action to execute
> > > >>    *
> > > >>    * @param    name string to include in the name of the
> > > thread used
> > > >>    * to execute the action
> > > >>    */
> > > >>   void execute(Runnable runnable, String name); }
> > > >>
> > > >> Peter Firmstone wrote:
> > > >>
> > > >>> Thanks Brian,
> > > >>>
> > > >>> Looking at our implementation code, DGC uses an Executor
> > > called ThreadPool, it's javadoc states:
> > > >>>
> > > >>> /**
> > > >>> * ThreadPool is a simple thread pool implementation of
> > > the Executor
> > > >>> * interface.
> > > >>> *
> > > >>> * A new task is always given to an idle thread, if one is
> > > available;
> > > >>> * otherwise, a new thread is always created.  There is no 
> > > >>> minimum
> > > >>> * warm thread count, nor is there a maximum thread count
> > > (tasks are
> > > >>> * never queued unless there are sufficient idle threads to 
> > > >>> execute
> > > >>> * them).
> > > >>> *
> > > >>> * New threads are created as daemon threads in the thread
> > > group that
> > > >>> * was passed to the ThreadPool instance's constructor.  Each 
> > > >>> thread's
> > > >>> * name is the prefix NewThreadAction.NAME_PREFIX 
> followed by the 
> > > >>> name
> > > >>> * of the task it is currently executing, or "Idle" if it is 
> > > >>> currently
> > > >>> * idle.
> > > >>>
> > > >>> ThreadPool predates Java 5, it looks like we can fix this
> > > by using an Executor from Java 5, we can look at limiting 
> the number 
> > > of threads created based on available CPU's and a scaling 
> factor and 
> > > place the tasks in a BlockingQueue, so if the queue is filled, it 
> > > blocks.
> > > >>>
> > > >>> Can you report the issue as a Bug on Jira for me, I'll
> > > fix this before the next release.
> > > >>>
> > > >>> Regards,
> > > >>>
> > > >>> Peter.
> > > >>>
> > > >>> Peter Firmstone wrote:
> > > >>>
> > > >>>> Hi Peter,
> > > >>>>
> > > >>>> I was wondering if you had any thoughts on this post
> > > from Bryan on River users?
> > > >>>>
> > > >>>> Hope you don't mind me asking ;)
> > > >>>>
> > > >>>> Best Regards,
> > > >>>>
> > > >>>> Peter Firmstone.
> > > >>>>
> > > >>>>
> > > 
> -------------------------------------------------------------------
> > > >>>> -----
> > > >>>>
> > > >>>> Subject:
> > > >>>> Re: DGC threads issue
> > > >>>> From:
> > > >>>> Tom Hobbs <tv...@googlemail.com>
> > > >>>> Date:
> > > >>>> Thu, 12 Jan 2012 20:45:01 +0000
> > > >>>> To:
> > > >>>> user@river.apache.org, dev@river.apache.org
> > > >>>>
> > > >>>> To:
> > > >>>> user@river.apache.org, dev@river.apache.org
> > > >>>>
> > > >>>>
> > > >>>> Hi Bryan,
> > > >>>>
> > > >>>> Sorry that no one got back to you about this.  I'm 
> afraid that 
> > > >>>> I don't know the answer to your question, I've 
> copied the dev 
> > > >>>> list into this email in case someone who monitors that list 
> > > >>>> (but not this one) has any ideas.
> > > >>>>
> > > >>>> Best regards,
> > > >>>>
> > > >>>> Tom
> > > >>>>
> > > >>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> > > <br...@systap.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>>> Just to follow up on this thread myself.  I modified
> > > the pattern to return a "thick" future rather than a 
> proxy for the 
> > > future.  This caused the RMI call to wait on the server until the 
> > > future was done and then sent back the outcome.  This "fixed" the 
> > > DGC memory/thread leak by reducing the number of exported proxies 
> > > drammatically.
> > > >>>>>
> > > >>>>> In terms of best practices, is distributed DGC simply
> > > not useful for exported objects with short life spans?  
> Can it only 
> > > be used with proxies for relatively long lived services?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Bryan
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>> -----Original Message-----
> > > >>>>>> From: Bryan Thompson
> > > >>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> > > >>>>>> To: user@river.apache.org
> > > >>>>>> Subject: DGC threads issue
> > > >>>>>>
> > > >>>>>> Hello,
> > > >>>>>>
> > > >>>>>> Background:
> > > >>>>>>
> > > >>>>>> I am seeing what would appear to be one DGC thread
> > > allocated per
> > > >>>>>> exported object.  This is using River 2.2 and Sun JDK
> > > 1.6.0_17.
> > > >>>>>> Relevant configuration parameters are below.
> > > >>>>>>
> > > >>>>>> I am observing problems with the DGC threads not being
> > > retired on
> > > >>>>>> a timely basis.  The exported objects are proxies 
> for Futures 
> > > >>>>>> which are being executed on the service.  The code 
> pattern is 
> > > >>>>>> such that the proxied Future goes out of lexical 
> scope quite 
> > > >>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> > > >>>>>>
> > > >>>>>> Under a modest load, a large number of such Futures
> > > are exported
> > > >>>>>> which results in a large number of long lived DGC
> > > threads.  This
> > > >>>>>> turns into a problem for the JVM due to the stack
> > > allocation per
> > > >>>>>> thread.  Presumably this is not good for other reasons as 
> > > >>>>>> well (e.g., scheduling).
> > > >>>>>>
> > > >>>>>> I have tried to override the leaseValue and checkInterval 
> > > >>>>>> defaults per the configuration options below.  I
> > > suspect that the
> > > >>>>>> lease interval is somehow not being obeyed, which is
> > > presumably a
> > > >>>>>> problem on my end.  However, I can verify that the
> > > configuration
> > > >>>>>> values are in fact showing up in
> > > >>>>>> System.getProperties() for at least some of the 
> JVMs involved 
> > > >>>>>> (the one which drives the workload and the one that I am 
> > > >>>>>> monitoring with the large number of DGC lease threads).
> > > >>>>>>
> > > >>>>>> Some questions:
> > > >>>>>>
> > > >>>>>> Is this one-thread-per-exported proxy the expected
> > > behavior when
> > > >>>>>> DGC is requested for the exported object?
> > > >>>>>>
> > > >>>>>> The DGC lease checker threads appear to expire ~14 -
> > > 15 minutes
> > > >>>>>> after I terminate the process which was 
> originating the RMI 
> > > >>>>>> requests.  This is close the sum of the default
> > > leaseValue (10m)
> > > >>>>>> and checkInterval (5m) parameters, but maybe there is
> > > some other
> > > >>>>>> timeout which is controlling this?  If this is the sum
> > > of those
> > > >>>>>> parameters, why would the DGC lease threads live until
> > > the sum of
> > > >>>>>> those values?  I thought that the lease would expire after 
> > > >>>>>> the leaseValue (10m default).
> > > >>>>>>
> > > >>>>>> Can the issue I am observing be caused by a low heap
> > > pressure on
> > > >>>>>> the JVM to which the RMI proxies were exported?  If it
> > > fails to
> > > >>>>>> GC those proxies, even though they are reachable, 
> could that 
> > > >>>>>> cause DGC to continue to retain those proxies on the JVM 
> > > >>>>>> which exported them?
> > > >>>>>>
> > > >>>>>> Is there any way to configure DGC to use a thread pool
> > > or to have
> > > >>>>>> the leases managed by a single thread?
> > > >>>>>>
> > > >>>>>> Is it possible that there is an interaction with the
> > > useNIO option?
> > > >>>>>>
> > > >>>>>> Relevant options that I am using include:
> > > >>>>>>
> > > >>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> > > >>>>>>   -Djava.rmi.dgc.leaseValue=30000
> > > >>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> > > >>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> > > >>>>>>
> > > >>>>>> Thanks in advance,
> > > >>>>>> Bryan
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >
> > > >
> > > >
> > >
> > >
> 

Re: RE: [Fwd: Re: DGC threads issue]

[Tom Hobbs <tv...@googlemail.com>]

The easiest way would be to the source distribution of River.  Unpack it
somewhere.  Modify the code and then run the ant task called "build-all" (I
think, whatever it's called it default target for the build.xml).  The
build file should be in the root of the unpacked source.

I don't think Peter created a jira fir it because I've not seen a
notification email.  I'm also unsure what code libel he's referring to.

Sent via mobile device, please forgive typos and spacing errors.

On 13 Jan 2012 12:30, "Bryan Thompson" <br...@systap.com> wrote:

> Peter,
>
> I would be happy to test out a fix on this.  Can you point out the
> modified line and link me to any directions on how to rebuild the necessary
> jar(s)?
>
> Did you already file an issue for this?  If not, I will file an issue and
> then update it when I test out this fix.
>
> Thanks,
> Bryan
>
> > -----Original Message-----
> > From: Peter Firmstone [mailto:jini@zeus.net.au]
> > Sent: Friday, January 13, 2012 3:21 AM
> > To: dev@river.apache.org
> > Cc: Bryan Thompson; user@river.apache.org
> > Subject: Re: [Fwd: Re: DGC threads issue]
> >
> > Ok, found the problem, when I fixed River-142, I introduced a
> > new bug, after creating a new DGC thread, I didn't set the
> > running flag to true, so each time, a new thread is created.
> >
> > Thankfully it's a very easy fix.
> >
> > Cheers,
> >
> > Peter.
> >
> > Peter Jones wrote:
> > > Peter,
> > >
> > > That internal Executor interface permits neither
> > synchronous execution in the calling thread ("this method
> > itself must not block") nor any small bound on the number of
> > Runnables that can be executed concurrently ("execution of a
> > given action must not be delayed indefinitely in order to
> > complete execution of a different action passed to a
> > different invocation of this method").  In other words,
> > concurrency-wise, it should be equivalent to "new
> > Thread(runnable, name)", or what the result of
> > j.u.c.Executors.newCachedThreadPool() would do.  Otherwise,
> > callers would not function correctly, because they expect to
> > be able to pass Runnables that execute indefinitely, not just
> > short-lived work tasks.
> > >
> > > The real purpose of this internal Executor/ThreadPool was
> > to provide an internal alternative to "new Thread(...)" that
> > takes care of certain things in common, like:
> > >
> > > - thread reuse instead of creation when reasonable (idle timeout)
> > > - but still with threads (re)named for the task being executed
> > > - insulation of any thread creation permission requirement from
> > > context calling execute (NewThreadAction)
> > > - but instead requiring direct user of thread pool to be
> > trusted not
> > > to abuse that insulation, with permission requirement for access
> > > (ThreadPoolPermission and GetThreadPoolAction)
> > > - common logging of uncaught exceptions
> > >
> > > -- Peter
> > >
> > >
> > > On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> > >
> > >
> > >> ThreadPool implements the com.sun.jini.thread.Executor interface.
> > >>
> > >> Because the interface states it should not block, I think
> > the calling thread should execute the task when the thread
> > pool becomes saturated, rather than continue to create new
> > threads as per the current implementation.  This will ensure
> > that the task is completed, the calling thread, if it uses a
> > sequence of tasks with dependencies will have to ensure that
> > it submits the tasks in order.  Since only the caller knows
> > the order, it makes sense for this to be the callers
> > responsibility.  As a result the executor will honor the non
> > blocking contract.  For that reason we'll use a zero length
> > queue, probably a SynchronousQueue.
> > >>
> > >> Regards,
> > >>
> > >> Peter.
> > >>
> > >> /**
> > >> * Executor is an abstraction for a thread factory or
> > thread pool for
> > >> * executing actions asynchronously.
> > >> *
> > >> * @author    Sun Microsystems, Inc.
> > >> *
> > >> */
> > >> public interface Executor {
> > >>
> > >>   /**
> > >>    * Executes the given Runnable action asynchronously in
> > some thread.
> > >>    *
> > >>    * The implementation may create a new thread to execute
> > the action,
> > >>    * or it may execute the action in an existing thread.
> > >>    *
> > >>    * The execution of a given action must not be delayed
> > indefinitely
> > >>    * in order to complete execution of a different action
> > passed to a
> > >>    * different invocation of this method.  In other words, the
> > >>    * implementation must assume that there may be
> > arbitrary dependencies
> > >>    * between actions passed to this method, so it needs to
> > be careful
> > >>    * to avoid potential deadlock by delaying execution of
> > one action
> > >>    * indefinitely until another completes.
> > >>    *
> > >>    * Also, this method itself must not block, because it
> > may be invoked
> > >>    * by code that is serially processing data to produce
> > multiple such
> > >>    * arbitrarily-dependent actions that need to be executed.
> > >>    *
> > >>    * @param    runnable the Runnable action to execute
> > >>    *
> > >>    * @param    name string to include in the name of the
> > thread used
> > >>    * to execute the action
> > >>    */
> > >>   void execute(Runnable runnable, String name); }
> > >>
> > >> Peter Firmstone wrote:
> > >>
> > >>> Thanks Brian,
> > >>>
> > >>> Looking at our implementation code, DGC uses an Executor
> > called ThreadPool, it's javadoc states:
> > >>>
> > >>> /**
> > >>> * ThreadPool is a simple thread pool implementation of
> > the Executor
> > >>> * interface.
> > >>> *
> > >>> * A new task is always given to an idle thread, if one is
> > available;
> > >>> * otherwise, a new thread is always created.  There is no minimum
> > >>> * warm thread count, nor is there a maximum thread count
> > (tasks are
> > >>> * never queued unless there are sufficient idle threads to execute
> > >>> * them).
> > >>> *
> > >>> * New threads are created as daemon threads in the thread
> > group that
> > >>> * was passed to the ThreadPool instance's constructor.  Each
> > >>> thread's
> > >>> * name is the prefix NewThreadAction.NAME_PREFIX followed by the
> > >>> name
> > >>> * of the task it is currently executing, or "Idle" if it is
> > >>> currently
> > >>> * idle.
> > >>>
> > >>> ThreadPool predates Java 5, it looks like we can fix this
> > by using an Executor from Java 5, we can look at limiting the
> > number of threads created based on available CPU's and a
> > scaling factor and place the tasks in a BlockingQueue, so if
> > the queue is filled, it blocks.
> > >>>
> > >>> Can you report the issue as a Bug on Jira for me, I'll
> > fix this before the next release.
> > >>>
> > >>> Regards,
> > >>>
> > >>> Peter.
> > >>>
> > >>> Peter Firmstone wrote:
> > >>>
> > >>>> Hi Peter,
> > >>>>
> > >>>> I was wondering if you had any thoughts on this post
> > from Bryan on River users?
> > >>>>
> > >>>> Hope you don't mind me asking ;)
> > >>>>
> > >>>> Best Regards,
> > >>>>
> > >>>> Peter Firmstone.
> > >>>>
> > >>>>
> > -------------------------------------------------------------------
> > >>>> -----
> > >>>>
> > >>>> Subject:
> > >>>> Re: DGC threads issue
> > >>>> From:
> > >>>> Tom Hobbs <tv...@googlemail.com>
> > >>>> Date:
> > >>>> Thu, 12 Jan 2012 20:45:01 +0000
> > >>>> To:
> > >>>> user@river.apache.org, dev@river.apache.org
> > >>>>
> > >>>> To:
> > >>>> user@river.apache.org, dev@river.apache.org
> > >>>>
> > >>>>
> > >>>> Hi Bryan,
> > >>>>
> > >>>> Sorry that no one got back to you about this.  I'm afraid that I
> > >>>> don't know the answer to your question, I've copied the dev list
> > >>>> into this email in case someone who monitors that list (but not
> > >>>> this one) has any ideas.
> > >>>>
> > >>>> Best regards,
> > >>>>
> > >>>> Tom
> > >>>>
> > >>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> > <br...@systap.com> wrote:
> > >>>>
> > >>>>
> > >>>>> Just to follow up on this thread myself.  I modified
> > the pattern to return a "thick" future rather than a proxy
> > for the future.  This caused the RMI call to wait on the
> > server until the future was done and then sent back the
> > outcome.  This "fixed" the DGC memory/thread leak by reducing
> > the number of exported proxies drammatically.
> > >>>>>
> > >>>>> In terms of best practices, is distributed DGC simply
> > not useful for exported objects with short life spans?  Can
> > it only be used with proxies for relatively long lived services?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Bryan
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>> -----Original Message-----
> > >>>>>> From: Bryan Thompson
> > >>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> > >>>>>> To: user@river.apache.org
> > >>>>>> Subject: DGC threads issue
> > >>>>>>
> > >>>>>> Hello,
> > >>>>>>
> > >>>>>> Background:
> > >>>>>>
> > >>>>>> I am seeing what would appear to be one DGC thread
> > allocated per
> > >>>>>> exported object.  This is using River 2.2 and Sun JDK
> > 1.6.0_17.
> > >>>>>> Relevant configuration parameters are below.
> > >>>>>>
> > >>>>>> I am observing problems with the DGC threads not being
> > retired on
> > >>>>>> a timely basis.  The exported objects are proxies for Futures
> > >>>>>> which are being executed on the service.  The code pattern is
> > >>>>>> such that the proxied Future goes out of lexical scope quite
> > >>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> > >>>>>>
> > >>>>>> Under a modest load, a large number of such Futures
> > are exported
> > >>>>>> which results in a large number of long lived DGC
> > threads.  This
> > >>>>>> turns into a problem for the JVM due to the stack
> > allocation per
> > >>>>>> thread.  Presumably this is not good for other reasons as well
> > >>>>>> (e.g., scheduling).
> > >>>>>>
> > >>>>>> I have tried to override the leaseValue and checkInterval
> > >>>>>> defaults per the configuration options below.  I
> > suspect that the
> > >>>>>> lease interval is somehow not being obeyed, which is
> > presumably a
> > >>>>>> problem on my end.  However, I can verify that the
> > configuration
> > >>>>>> values are in fact showing up in
> > >>>>>> System.getProperties() for at least some of the JVMs involved
> > >>>>>> (the one which drives the workload and the one that I am
> > >>>>>> monitoring with the large number of DGC lease threads).
> > >>>>>>
> > >>>>>> Some questions:
> > >>>>>>
> > >>>>>> Is this one-thread-per-exported proxy the expected
> > behavior when
> > >>>>>> DGC is requested for the exported object?
> > >>>>>>
> > >>>>>> The DGC lease checker threads appear to expire ~14 -
> > 15 minutes
> > >>>>>> after I terminate the process which was originating the RMI
> > >>>>>> requests.  This is close the sum of the default
> > leaseValue (10m)
> > >>>>>> and checkInterval (5m) parameters, but maybe there is
> > some other
> > >>>>>> timeout which is controlling this?  If this is the sum
> > of those
> > >>>>>> parameters, why would the DGC lease threads live until
> > the sum of
> > >>>>>> those values?  I thought that the lease would expire after the
> > >>>>>> leaseValue (10m default).
> > >>>>>>
> > >>>>>> Can the issue I am observing be caused by a low heap
> > pressure on
> > >>>>>> the JVM to which the RMI proxies were exported?  If it
> > fails to
> > >>>>>> GC those proxies, even though they are reachable, could that
> > >>>>>> cause DGC to continue to retain those proxies on the JVM which
> > >>>>>> exported them?
> > >>>>>>
> > >>>>>> Is there any way to configure DGC to use a thread pool
> > or to have
> > >>>>>> the leases managed by a single thread?
> > >>>>>>
> > >>>>>> Is it possible that there is an interaction with the
> > useNIO option?
> > >>>>>>
> > >>>>>> Relevant options that I am using include:
> > >>>>>>
> > >>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> > >>>>>>   -Djava.rmi.dgc.leaseValue=30000
> > >>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> > >>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> > >>>>>>
> > >>>>>> Thanks in advance,
> > >>>>>> Bryan
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >
> > >
> > >
> >
> >

Re: [Fwd: Re: DGC threads issue]

[Peter Firmstone <ji...@zeus.net.au>]

Only if you use JrmpExporter.

I'll get back to shortly with some build instructions.

Cheers,

Peter.

Bryan Thompson wrote:
> What about the following property?  Is it still valid?
>
>         -Dsun.rmi.transport.tcp.connectionPool=true
>
> Thanks,
> Bryan
>
>   
>> -----Original Message-----
>> From: Peter Firmstone [mailto:jini@zeus.net.au]
>> Sent: Friday, January 13, 2012 8:41 AM
>> To: Bryan Thompson
>> Cc: dev@river.apache.org; user@river.apache.org
>> Subject: Re: [Fwd: Re: DGC threads issue]
>>
>> Brian,
>>
>> Do you have a Linux / Unix build environment?
>>
>> River isn't building yet on Window's, FreeBSD, or OpenBSD,
>> known build environments that work are Ubuntu and Solaris.
>> If not, that's ok, well be able to get the necessary jar's off Hudson.
>>
>> The DGC lease property values you need to set are:
>>
>> com.sun.jini.jeri.dgc.leaseValue=30000
>> com.sun.jini.jeri.dgc.checkInterval=15000
>>
>> RMI (called JRMP) is superseded code now, River doesn't use
>> that unless you configure your Exporter to be JrmpExporter,
>> but you're better off sticking with BasicJeriExporter.
>>
>> Regards,
>>
>> Peter.
>>
>> Bryan Thompson wrote:
>>     
>>> Peter,
>>>
>>> I would be happy to test out a fix on this.  Can you point
>>>       
>> out the modified line and link me to any directions on how to
>> rebuild the necessary jar(s)?
>>     
>>> Did you already file an issue for this?  If not, I will
>>>       
>> file an issue and then update it when I test out this fix.
>>     
>>> Thanks,
>>> Bryan
>>>
>>>
>>>       
>>>> -----Original Message-----
>>>> From: Peter Firmstone [mailto:jini@zeus.net.au]
>>>> Sent: Friday, January 13, 2012 3:21 AM
>>>> To: dev@river.apache.org
>>>> Cc: Bryan Thompson; user@river.apache.org
>>>> Subject: Re: [Fwd: Re: DGC threads issue]
>>>>
>>>> Ok, found the problem, when I fixed River-142, I introduced a new
>>>> bug, after creating a new DGC thread, I didn't set the
>>>>         
>> running flag
>>     
>>>> to true, so each time, a new thread is created.
>>>>
>>>> Thankfully it's a very easy fix.
>>>>
>>>> Cheers,
>>>>
>>>> Peter.
>>>>
>>>> Peter Jones wrote:
>>>>
>>>>         
>>>>> Peter,
>>>>>
>>>>> That internal Executor interface permits neither
>>>>>
>>>>>           
>>>> synchronous execution in the calling thread ("this method
>>>>         
>> itself must
>>     
>>>> not block") nor any small bound on the number of Runnables
>>>>         
>> that can
>>     
>>>> be executed concurrently ("execution of a given action must not be
>>>> delayed indefinitely in order to complete execution of a different
>>>> action passed to a different invocation of this method").
>>>>         
>> In other
>>     
>>>> words, concurrency-wise, it should be equivalent to "new
>>>> Thread(runnable, name)", or what the result of
>>>> j.u.c.Executors.newCachedThreadPool() would do.
>>>>         
>> Otherwise, callers
>>     
>>>> would not function correctly, because they expect to be
>>>>         
>> able to pass
>>     
>>>> Runnables that execute indefinitely, not just short-lived
>>>>         
>> work tasks.
>>     
>>>>> The real purpose of this internal Executor/ThreadPool was
>>>>>
>>>>>           
>>>> to provide an internal alternative to "new Thread(...)" that takes
>>>> care of certain things in common, like:
>>>>
>>>>         
>>>>> - thread reuse instead of creation when reasonable (idle timeout)
>>>>> - but still with threads (re)named for the task being executed
>>>>> - insulation of any thread creation permission requirement from
>>>>> context calling execute (NewThreadAction)
>>>>> - but instead requiring direct user of thread pool to be
>>>>>
>>>>>           
>>>> trusted not
>>>>
>>>>         
>>>>> to abuse that insulation, with permission requirement for access
>>>>> (ThreadPoolPermission and GetThreadPoolAction)
>>>>> - common logging of uncaught exceptions
>>>>>
>>>>> -- Peter
>>>>>
>>>>>
>>>>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> ThreadPool implements the com.sun.jini.thread.Executor interface.
>>>>>>
>>>>>> Because the interface states it should not block, I think
>>>>>>
>>>>>>             
>>>> the calling thread should execute the task when the thread pool
>>>> becomes saturated, rather than continue to create new
>>>>         
>> threads as per
>>     
>>>> the current implementation.  This will ensure that the task is
>>>> completed, the calling thread, if it uses a sequence of tasks with
>>>> dependencies will have to ensure that it submits the tasks
>>>>         
>> in order.
>>     
>>>> Since only the caller knows the order, it makes sense for
>>>>         
>> this to be
>>     
>>>> the callers responsibility.  As a result the executor will
>>>>         
>> honor the
>>     
>>>> non blocking contract.  For that reason we'll use a zero length
>>>> queue, probably a SynchronousQueue.
>>>>
>>>>         
>>>>>> Regards,
>>>>>>
>>>>>> Peter.
>>>>>>
>>>>>> /**
>>>>>> * Executor is an abstraction for a thread factory or
>>>>>>
>>>>>>             
>>>> thread pool for
>>>>
>>>>         
>>>>>> * executing actions asynchronously.
>>>>>> *
>>>>>> * @author    Sun Microsystems, Inc.
>>>>>> *
>>>>>> */
>>>>>> public interface Executor {
>>>>>>
>>>>>>   /**
>>>>>>    * Executes the given Runnable action asynchronously in
>>>>>>
>>>>>>             
>>>> some thread.
>>>>
>>>>         
>>>>>>    *
>>>>>>    * The implementation may create a new thread to execute
>>>>>>
>>>>>>             
>>>> the action,
>>>>
>>>>         
>>>>>>    * or it may execute the action in an existing thread.
>>>>>>    *
>>>>>>    * The execution of a given action must not be delayed
>>>>>>
>>>>>>             
>>>> indefinitely
>>>>
>>>>         
>>>>>>    * in order to complete execution of a different action
>>>>>>
>>>>>>             
>>>> passed to a
>>>>
>>>>         
>>>>>>    * different invocation of this method.  In other words, the
>>>>>>    * implementation must assume that there may be
>>>>>>
>>>>>>             
>>>> arbitrary dependencies
>>>>
>>>>         
>>>>>>    * between actions passed to this method, so it needs to
>>>>>>
>>>>>>             
>>>> be careful
>>>>
>>>>         
>>>>>>    * to avoid potential deadlock by delaying execution of
>>>>>>
>>>>>>             
>>>> one action
>>>>
>>>>         
>>>>>>    * indefinitely until another completes.
>>>>>>    *
>>>>>>    * Also, this method itself must not block, because it
>>>>>>
>>>>>>             
>>>> may be invoked
>>>>
>>>>         
>>>>>>    * by code that is serially processing data to produce
>>>>>>
>>>>>>             
>>>> multiple such
>>>>
>>>>         
>>>>>>    * arbitrarily-dependent actions that need to be executed.
>>>>>>    *
>>>>>>    * @param    runnable the Runnable action to execute
>>>>>>    *
>>>>>>    * @param    name string to include in the name of the
>>>>>>
>>>>>>             
>>>> thread used
>>>>
>>>>         
>>>>>>    * to execute the action
>>>>>>    */
>>>>>>   void execute(Runnable runnable, String name); }
>>>>>>
>>>>>> Peter Firmstone wrote:
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Thanks Brian,
>>>>>>>
>>>>>>> Looking at our implementation code, DGC uses an Executor
>>>>>>>
>>>>>>>               
>>>> called ThreadPool, it's javadoc states:
>>>>
>>>>         
>>>>>>> /**
>>>>>>> * ThreadPool is a simple thread pool implementation of
>>>>>>>
>>>>>>>               
>>>> the Executor
>>>>
>>>>         
>>>>>>> * interface.
>>>>>>> *
>>>>>>> * A new task is always given to an idle thread, if one is
>>>>>>>
>>>>>>>               
>>>> available;
>>>>
>>>>         
>>>>>>> * otherwise, a new thread is always created.  There is
>>>>>>>               
>> no minimum
>>     
>>>>>>> * warm thread count, nor is there a maximum thread count
>>>>>>>
>>>>>>>               
>>>> (tasks are
>>>>
>>>>         
>>>>>>> * never queued unless there are sufficient idle threads
>>>>>>>               
>> to execute
>>     
>>>>>>> * them).
>>>>>>> *
>>>>>>> * New threads are created as daemon threads in the thread
>>>>>>>
>>>>>>>               
>>>> group that
>>>>
>>>>         
>>>>>>> * was passed to the ThreadPool instance's constructor.  Each
>>>>>>> thread's
>>>>>>> * name is the prefix NewThreadAction.NAME_PREFIX
>>>>>>>               
>> followed by the
>>     
>>>>>>> name
>>>>>>> * of the task it is currently executing, or "Idle" if it is
>>>>>>> currently
>>>>>>> * idle.
>>>>>>>
>>>>>>> ThreadPool predates Java 5, it looks like we can fix this
>>>>>>>
>>>>>>>               
>>>> by using an Executor from Java 5, we can look at limiting
>>>>         
>> the number
>>     
>>>> of threads created based on available CPU's and a scaling
>>>>         
>> factor and
>>     
>>>> place the tasks in a BlockingQueue, so if the queue is filled, it
>>>> blocks.
>>>>
>>>>         
>>>>>>> Can you report the issue as a Bug on Jira for me, I'll
>>>>>>>
>>>>>>>               
>>>> fix this before the next release.
>>>>
>>>>         
>>>>>>> Regards,
>>>>>>>
>>>>>>> Peter.
>>>>>>>
>>>>>>> Peter Firmstone wrote:
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Hi Peter,
>>>>>>>>
>>>>>>>> I was wondering if you had any thoughts on this post
>>>>>>>>
>>>>>>>>                 
>>>> from Bryan on River users?
>>>>
>>>>         
>>>>>>>> Hope you don't mind me asking ;)
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>>
>>>>>>>> Peter Firmstone.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>> -------------------------------------------------------------------
>>>>
>>>>         
>>>>>>>> -----
>>>>>>>>
>>>>>>>> Subject:
>>>>>>>> Re: DGC threads issue
>>>>>>>> From:
>>>>>>>> Tom Hobbs <tv...@googlemail.com>
>>>>>>>> Date:
>>>>>>>> Thu, 12 Jan 2012 20:45:01 +0000
>>>>>>>> To:
>>>>>>>> user@river.apache.org, dev@river.apache.org
>>>>>>>>
>>>>>>>> To:
>>>>>>>> user@river.apache.org, dev@river.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Bryan,
>>>>>>>>
>>>>>>>> Sorry that no one got back to you about this.  I'm
>>>>>>>>                 
>> afraid that I
>>     
>>>>>>>> don't know the answer to your question, I've copied
>>>>>>>>                 
>> the dev list
>>     
>>>>>>>> into this email in case someone who monitors that list
>>>>>>>>                 
>> (but not
>>     
>>>>>>>> this one) has any ideas.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Tom
>>>>>>>>
>>>>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
>>>>>>>>
>>>>>>>>                 
>>>> <br...@systap.com> wrote:
>>>>
>>>>         
>>>>>>>>                 
>>>>>>>>> Just to follow up on this thread myself.  I modified
>>>>>>>>>
>>>>>>>>>                   
>>>> the pattern to return a "thick" future rather than a proxy for the
>>>> future.  This caused the RMI call to wait on the server until the
>>>> future was done and then sent back the outcome.  This
>>>>         
>> "fixed" the DGC
>>     
>>>> memory/thread leak by reducing the number of exported proxies
>>>> drammatically.
>>>>
>>>>         
>>>>>>>>> In terms of best practices, is distributed DGC simply
>>>>>>>>>
>>>>>>>>>                   
>>>> not useful for exported objects with short life spans?
>>>>         
>> Can it only
>>     
>>>> be used with proxies for relatively long lived services?
>>>>
>>>>         
>>>>>>>>> Thanks,
>>>>>>>>> Bryan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Bryan Thompson
>>>>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
>>>>>>>>>> To: user@river.apache.org
>>>>>>>>>> Subject: DGC threads issue
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> Background:
>>>>>>>>>>
>>>>>>>>>> I am seeing what would appear to be one DGC thread
>>>>>>>>>>
>>>>>>>>>>                     
>>>> allocated per
>>>>
>>>>         
>>>>>>>>>> exported object.  This is using River 2.2 and Sun JDK
>>>>>>>>>>
>>>>>>>>>>                     
>>>> 1.6.0_17.
>>>>
>>>>         
>>>>>>>>>> Relevant configuration parameters are below.
>>>>>>>>>>
>>>>>>>>>> I am observing problems with the DGC threads not being
>>>>>>>>>>
>>>>>>>>>>                     
>>>> retired on
>>>>
>>>>         
>>>>>>>>>> a timely basis.  The exported objects are proxies
>>>>>>>>>>                     
>> for Futures
>>     
>>>>>>>>>> which are being executed on the service.  The code
>>>>>>>>>>                     
>> pattern is
>>     
>>>>>>>>>> such that the proxied Future goes out of lexical scope quite
>>>>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
>>>>>>>>>>
>>>>>>>>>> Under a modest load, a large number of such Futures
>>>>>>>>>>
>>>>>>>>>>                     
>>>> are exported
>>>>
>>>>         
>>>>>>>>>> which results in a large number of long lived DGC
>>>>>>>>>>
>>>>>>>>>>                     
>>>> threads.  This
>>>>
>>>>         
>>>>>>>>>> turns into a problem for the JVM due to the stack
>>>>>>>>>>
>>>>>>>>>>                     
>>>> allocation per
>>>>
>>>>         
>>>>>>>>>> thread.  Presumably this is not good for other
>>>>>>>>>>                     
>> reasons as well
>>     
>>>>>>>>>> (e.g., scheduling).
>>>>>>>>>>
>>>>>>>>>> I have tried to override the leaseValue and checkInterval
>>>>>>>>>> defaults per the configuration options below.  I
>>>>>>>>>>
>>>>>>>>>>                     
>>>> suspect that the
>>>>
>>>>         
>>>>>>>>>> lease interval is somehow not being obeyed, which is
>>>>>>>>>>
>>>>>>>>>>                     
>>>> presumably a
>>>>
>>>>         
>>>>>>>>>> problem on my end.  However, I can verify that the
>>>>>>>>>>
>>>>>>>>>>                     
>>>> configuration
>>>>
>>>>         
>>>>>>>>>> values are in fact showing up in
>>>>>>>>>> System.getProperties() for at least some of the JVMs
>>>>>>>>>>                     
>> involved
>>     
>>>>>>>>>> (the one which drives the workload and the one that I am
>>>>>>>>>> monitoring with the large number of DGC lease threads).
>>>>>>>>>>
>>>>>>>>>> Some questions:
>>>>>>>>>>
>>>>>>>>>> Is this one-thread-per-exported proxy the expected
>>>>>>>>>>
>>>>>>>>>>                     
>>>> behavior when
>>>>
>>>>         
>>>>>>>>>> DGC is requested for the exported object?
>>>>>>>>>>
>>>>>>>>>> The DGC lease checker threads appear to expire ~14 -
>>>>>>>>>>
>>>>>>>>>>                     
>>>> 15 minutes
>>>>
>>>>         
>>>>>>>>>> after I terminate the process which was originating the RMI
>>>>>>>>>> requests.  This is close the sum of the default
>>>>>>>>>>
>>>>>>>>>>                     
>>>> leaseValue (10m)
>>>>
>>>>         
>>>>>>>>>> and checkInterval (5m) parameters, but maybe there is
>>>>>>>>>>
>>>>>>>>>>                     
>>>> some other
>>>>
>>>>         
>>>>>>>>>> timeout which is controlling this?  If this is the sum
>>>>>>>>>>
>>>>>>>>>>                     
>>>> of those
>>>>
>>>>         
>>>>>>>>>> parameters, why would the DGC lease threads live until
>>>>>>>>>>
>>>>>>>>>>                     
>>>> the sum of
>>>>
>>>>         
>>>>>>>>>> those values?  I thought that the lease would expire
>>>>>>>>>>                     
>> after the
>>     
>>>>>>>>>> leaseValue (10m default).
>>>>>>>>>>
>>>>>>>>>> Can the issue I am observing be caused by a low heap
>>>>>>>>>>
>>>>>>>>>>                     
>>>> pressure on
>>>>
>>>>         
>>>>>>>>>> the JVM to which the RMI proxies were exported?  If it
>>>>>>>>>>
>>>>>>>>>>                     
>>>> fails to
>>>>
>>>>         
>>>>>>>>>> GC those proxies, even though they are reachable, could that
>>>>>>>>>> cause DGC to continue to retain those proxies on the
>>>>>>>>>>                     
>> JVM which
>>     
>>>>>>>>>> exported them?
>>>>>>>>>>
>>>>>>>>>> Is there any way to configure DGC to use a thread pool
>>>>>>>>>>
>>>>>>>>>>                     
>>>> or to have
>>>>
>>>>         
>>>>>>>>>> the leases managed by a single thread?
>>>>>>>>>>
>>>>>>>>>> Is it possible that there is an interaction with the
>>>>>>>>>>
>>>>>>>>>>                     
>>>> useNIO option?
>>>>
>>>>         
>>>>>>>>>> Relevant options that I am using include:
>>>>>>>>>>
>>>>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
>>>>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
>>>>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
>>>>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
>>>>>>>>>>
>>>>>>>>>> Thanks in advance,
>>>>>>>>>> Bryan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>
>>>>>>>>                 
>>>>>>>               
>>>>>           
>>     
>
>   

Re: [Fwd: Re: DGC threads issue]

[Peter Firmstone <ji...@zeus.net.au>]

Bryan Thompson wrote:
> Peter,
>
> Rather than testing against the trunk, I would prefer to test the change against the 2.2 release.  That will allow me to avoid validating the entire state of the trunk and let me focus on whether the change fixes the specific DGC problem.
>
> It looks like [1] is the commit containing this fix.  Can I simply apply that patch to the clean checkout of 2.2?
>
> Thanks,
> Bryan
>
> [1] http://mail-archives.apache.org/mod_mbox/river-commits/201201.mbox/%3C20120114105227.9890723888E7%40eris.apache.org%3E
>
>   

Yes,

See attached.

Regards,

Peter.

Re: [Fwd: Re: DGC threads issue]

[Peter Firmstone <ji...@zeus.net.au>]

Bryan Thompson wrote:
> Peter,
>
> Rather than testing against the trunk, I would prefer to test the change against the 2.2 release.  That will allow me to avoid validating the entire state of the trunk and let me focus on whether the change fixes the specific DGC problem.
>
> It looks like [1] is the commit containing this fix.  Can I simply apply that patch to the clean checkout of 2.2?
>
> Thanks,
> Bryan
>
> [1] http://mail-archives.apache.org/mod_mbox/river-commits/201201.mbox/%3C20120114105227.9890723888E7%40eris.apache.org%3E
>
>   

Yes,

See attached.

Regards,

Peter.

RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

Peter,

Rather than testing against the trunk, I would prefer to test the change against the 2.2 release.  That will allow me to avoid validating the entire state of the trunk and let me focus on whether the change fixes the specific DGC problem.

It looks like [1] is the commit containing this fix.  Can I simply apply that patch to the clean checkout of 2.2?

Thanks,
Bryan

[1] http://mail-archives.apache.org/mod_mbox/river-commits/201201.mbox/%3C20120114105227.9890723888E7%40eris.apache.org%3E



> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au]
> Sent: Saturday, January 14, 2012 7:04 AM
> To: Bryan Thompson
> Cc: dev@river.apache.org; user@river.apache.org
> Subject: Re: [Fwd: Re: DGC threads issue]
>
> Brian,
>
> First checkout the main trunk (I've only just committed the
> fix so make sure it's updated, if you've checked it out already):
>
> svn checkout http://svn.apache.org/repos/asf/river/jtsk/trunk river
>
>
> Then create a build.properties file in trunk that contains
> something like the following:
>
> java.home=/usr/jdk/jdk1.6.0_25
> river.home=/opt/src/river
> log.config=/home/peter/logging.properties
> debug=true
> debuglevel=lines,source,vars
>
> Then from the command line
>
> $ ant all.clean all.build
>
> If you want to run the qa tests:
>
> $ ant qa.run
>
> Cheers,
>
> Peter.
>
> Bryan Thompson wrote:
> > What about the following property?  Is it still valid?
> >
> >         -Dsun.rmi.transport.tcp.connectionPool=true
> >
> > Thanks,
> > Bryan
> >
> >
> >> -----Original Message-----
> >> From: Peter Firmstone [mailto:jini@zeus.net.au]
> >> Sent: Friday, January 13, 2012 8:41 AM
> >> To: Bryan Thompson
> >> Cc: dev@river.apache.org; user@river.apache.org
> >> Subject: Re: [Fwd: Re: DGC threads issue]
> >>
> >> Brian,
> >>
> >> Do you have a Linux / Unix build environment?
> >>
> >> River isn't building yet on Window's, FreeBSD, or OpenBSD, known
> >> build environments that work are Ubuntu and Solaris.
> >> If not, that's ok, well be able to get the necessary jar's
> off Hudson.
> >>
> >> The DGC lease property values you need to set are:
> >>
> >> com.sun.jini.jeri.dgc.leaseValue=30000
> >> com.sun.jini.jeri.dgc.checkInterval=15000
> >>
> >> RMI (called JRMP) is superseded code now, River doesn't use that
> >> unless you configure your Exporter to be JrmpExporter, but you're
> >> better off sticking with BasicJeriExporter.
> >>
> >> Regards,
> >>
> >> Peter.
> >>
> >> Bryan Thompson wrote:
> >>
> >>> Peter,
> >>>
> >>> I would be happy to test out a fix on this.  Can you point
> >>>
> >> out the modified line and link me to any directions on how
> to rebuild
> >> the necessary jar(s)?
> >>
> >>> Did you already file an issue for this?  If not, I will
> >>>
> >> file an issue and then update it when I test out this fix.
> >>
> >>> Thanks,
> >>> Bryan
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Peter Firmstone [mailto:jini@zeus.net.au]
> >>>> Sent: Friday, January 13, 2012 3:21 AM
> >>>> To: dev@river.apache.org
> >>>> Cc: Bryan Thompson; user@river.apache.org
> >>>> Subject: Re: [Fwd: Re: DGC threads issue]
> >>>>
> >>>> Ok, found the problem, when I fixed River-142, I
> introduced a new
> >>>> bug, after creating a new DGC thread, I didn't set the
> >>>>
> >> running flag
> >>
> >>>> to true, so each time, a new thread is created.
> >>>>
> >>>> Thankfully it's a very easy fix.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Peter.
> >>>>
> >>>> Peter Jones wrote:
> >>>>
> >>>>
> >>>>> Peter,
> >>>>>
> >>>>> That internal Executor interface permits neither
> >>>>>
> >>>>>
> >>>> synchronous execution in the calling thread ("this method
> >>>>
> >> itself must
> >>
> >>>> not block") nor any small bound on the number of Runnables
> >>>>
> >> that can
> >>
> >>>> be executed concurrently ("execution of a given action
> must not be
> >>>> delayed indefinitely in order to complete execution of a
> different
> >>>> action passed to a different invocation of this method").
> >>>>
> >> In other
> >>
> >>>> words, concurrency-wise, it should be equivalent to "new
> >>>> Thread(runnable, name)", or what the result of
> >>>> j.u.c.Executors.newCachedThreadPool() would do.
> >>>>
> >> Otherwise, callers
> >>
> >>>> would not function correctly, because they expect to be
> >>>>
> >> able to pass
> >>
> >>>> Runnables that execute indefinitely, not just short-lived
> >>>>
> >> work tasks.
> >>
> >>>>> The real purpose of this internal Executor/ThreadPool was
> >>>>>
> >>>>>
> >>>> to provide an internal alternative to "new Thread(...)"
> that takes
> >>>> care of certain things in common, like:
> >>>>
> >>>>
> >>>>> - thread reuse instead of creation when reasonable
> (idle timeout)
> >>>>> - but still with threads (re)named for the task being executed
> >>>>> - insulation of any thread creation permission requirement from
> >>>>> context calling execute (NewThreadAction)
> >>>>> - but instead requiring direct user of thread pool to be
> >>>>>
> >>>>>
> >>>> trusted not
> >>>>
> >>>>
> >>>>> to abuse that insulation, with permission requirement
> for access
> >>>>> (ThreadPoolPermission and GetThreadPoolAction)
> >>>>> - common logging of uncaught exceptions
> >>>>>
> >>>>> -- Peter
> >>>>>
> >>>>>
> >>>>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> ThreadPool implements the com.sun.jini.thread.Executor
> interface.
> >>>>>>
> >>>>>> Because the interface states it should not block, I think
> >>>>>>
> >>>>>>
> >>>> the calling thread should execute the task when the thread pool
> >>>> becomes saturated, rather than continue to create new
> >>>>
> >> threads as per
> >>
> >>>> the current implementation.  This will ensure that the task is
> >>>> completed, the calling thread, if it uses a sequence of
> tasks with
> >>>> dependencies will have to ensure that it submits the tasks
> >>>>
> >> in order.
> >>
> >>>> Since only the caller knows the order, it makes sense for
> >>>>
> >> this to be
> >>
> >>>> the callers responsibility.  As a result the executor will
> >>>>
> >> honor the
> >>
> >>>> non blocking contract.  For that reason we'll use a zero length
> >>>> queue, probably a SynchronousQueue.
> >>>>
> >>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Peter.
> >>>>>>
> >>>>>> /**
> >>>>>> * Executor is an abstraction for a thread factory or
> >>>>>>
> >>>>>>
> >>>> thread pool for
> >>>>
> >>>>
> >>>>>> * executing actions asynchronously.
> >>>>>> *
> >>>>>> * @author    Sun Microsystems, Inc.
> >>>>>> *
> >>>>>> */
> >>>>>> public interface Executor {
> >>>>>>
> >>>>>>   /**
> >>>>>>    * Executes the given Runnable action asynchronously in
> >>>>>>
> >>>>>>
> >>>> some thread.
> >>>>
> >>>>
> >>>>>>    *
> >>>>>>    * The implementation may create a new thread to execute
> >>>>>>
> >>>>>>
> >>>> the action,
> >>>>
> >>>>
> >>>>>>    * or it may execute the action in an existing thread.
> >>>>>>    *
> >>>>>>    * The execution of a given action must not be delayed
> >>>>>>
> >>>>>>
> >>>> indefinitely
> >>>>
> >>>>
> >>>>>>    * in order to complete execution of a different action
> >>>>>>
> >>>>>>
> >>>> passed to a
> >>>>
> >>>>
> >>>>>>    * different invocation of this method.  In other words, the
> >>>>>>    * implementation must assume that there may be
> >>>>>>
> >>>>>>
> >>>> arbitrary dependencies
> >>>>
> >>>>
> >>>>>>    * between actions passed to this method, so it needs to
> >>>>>>
> >>>>>>
> >>>> be careful
> >>>>
> >>>>
> >>>>>>    * to avoid potential deadlock by delaying execution of
> >>>>>>
> >>>>>>
> >>>> one action
> >>>>
> >>>>
> >>>>>>    * indefinitely until another completes.
> >>>>>>    *
> >>>>>>    * Also, this method itself must not block, because it
> >>>>>>
> >>>>>>
> >>>> may be invoked
> >>>>
> >>>>
> >>>>>>    * by code that is serially processing data to produce
> >>>>>>
> >>>>>>
> >>>> multiple such
> >>>>
> >>>>
> >>>>>>    * arbitrarily-dependent actions that need to be executed.
> >>>>>>    *
> >>>>>>    * @param    runnable the Runnable action to execute
> >>>>>>    *
> >>>>>>    * @param    name string to include in the name of the
> >>>>>>
> >>>>>>
> >>>> thread used
> >>>>
> >>>>
> >>>>>>    * to execute the action
> >>>>>>    */
> >>>>>>   void execute(Runnable runnable, String name); }
> >>>>>>
> >>>>>> Peter Firmstone wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Thanks Brian,
> >>>>>>>
> >>>>>>> Looking at our implementation code, DGC uses an Executor
> >>>>>>>
> >>>>>>>
> >>>> called ThreadPool, it's javadoc states:
> >>>>
> >>>>
> >>>>>>> /**
> >>>>>>> * ThreadPool is a simple thread pool implementation of
> >>>>>>>
> >>>>>>>
> >>>> the Executor
> >>>>
> >>>>
> >>>>>>> * interface.
> >>>>>>> *
> >>>>>>> * A new task is always given to an idle thread, if one is
> >>>>>>>
> >>>>>>>
> >>>> available;
> >>>>
> >>>>
> >>>>>>> * otherwise, a new thread is always created.  There is
> >>>>>>>
> >> no minimum
> >>
> >>>>>>> * warm thread count, nor is there a maximum thread count
> >>>>>>>
> >>>>>>>
> >>>> (tasks are
> >>>>
> >>>>
> >>>>>>> * never queued unless there are sufficient idle threads
> >>>>>>>
> >> to execute
> >>
> >>>>>>> * them).
> >>>>>>> *
> >>>>>>> * New threads are created as daemon threads in the thread
> >>>>>>>
> >>>>>>>
> >>>> group that
> >>>>
> >>>>
> >>>>>>> * was passed to the ThreadPool instance's constructor.  Each
> >>>>>>> thread's
> >>>>>>> * name is the prefix NewThreadAction.NAME_PREFIX
> >>>>>>>
> >> followed by the
> >>
> >>>>>>> name
> >>>>>>> * of the task it is currently executing, or "Idle" if it is
> >>>>>>> currently
> >>>>>>> * idle.
> >>>>>>>
> >>>>>>> ThreadPool predates Java 5, it looks like we can fix this
> >>>>>>>
> >>>>>>>
> >>>> by using an Executor from Java 5, we can look at limiting
> >>>>
> >> the number
> >>
> >>>> of threads created based on available CPU's and a scaling
> >>>>
> >> factor and
> >>
> >>>> place the tasks in a BlockingQueue, so if the queue is
> filled, it
> >>>> blocks.
> >>>>
> >>>>
> >>>>>>> Can you report the issue as a Bug on Jira for me, I'll
> >>>>>>>
> >>>>>>>
> >>>> fix this before the next release.
> >>>>
> >>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Peter.
> >>>>>>>
> >>>>>>> Peter Firmstone wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Hi Peter,
> >>>>>>>>
> >>>>>>>> I was wondering if you had any thoughts on this post
> >>>>>>>>
> >>>>>>>>
> >>>> from Bryan on River users?
> >>>>
> >>>>
> >>>>>>>> Hope you don't mind me asking ;)
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>>
> >>>>>>>> Peter Firmstone.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>
> -------------------------------------------------------------------
> >>>>
> >>>>
> >>>>>>>> -----
> >>>>>>>>
> >>>>>>>> Subject:
> >>>>>>>> Re: DGC threads issue
> >>>>>>>> From:
> >>>>>>>> Tom Hobbs <tv...@googlemail.com>
> >>>>>>>> Date:
> >>>>>>>> Thu, 12 Jan 2012 20:45:01 +0000
> >>>>>>>> To:
> >>>>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>>>
> >>>>>>>> To:
> >>>>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi Bryan,
> >>>>>>>>
> >>>>>>>> Sorry that no one got back to you about this.  I'm
> >>>>>>>>
> >> afraid that I
> >>
> >>>>>>>> don't know the answer to your question, I've copied
> >>>>>>>>
> >> the dev list
> >>
> >>>>>>>> into this email in case someone who monitors that list
> >>>>>>>>
> >> (but not
> >>
> >>>>>>>> this one) has any ideas.
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>>
> >>>>>>>> Tom
> >>>>>>>>
> >>>>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> >>>>>>>>
> >>>>>>>>
> >>>> <br...@systap.com> wrote:
> >>>>
> >>>>
> >>>>>>>>
> >>>>>>>>> Just to follow up on this thread myself.  I modified
> >>>>>>>>>
> >>>>>>>>>
> >>>> the pattern to return a "thick" future rather than a
> proxy for the
> >>>> future.  This caused the RMI call to wait on the server
> until the
> >>>> future was done and then sent back the outcome.  This
> >>>>
> >> "fixed" the DGC
> >>
> >>>> memory/thread leak by reducing the number of exported proxies
> >>>> drammatically.
> >>>>
> >>>>
> >>>>>>>>> In terms of best practices, is distributed DGC simply
> >>>>>>>>>
> >>>>>>>>>
> >>>> not useful for exported objects with short life spans?
> >>>>
> >> Can it only
> >>
> >>>> be used with proxies for relatively long lived services?
> >>>>
> >>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Bryan
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Bryan Thompson
> >>>>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>>>>>>>>> To: user@river.apache.org
> >>>>>>>>>> Subject: DGC threads issue
> >>>>>>>>>>
> >>>>>>>>>> Hello,
> >>>>>>>>>>
> >>>>>>>>>> Background:
> >>>>>>>>>>
> >>>>>>>>>> I am seeing what would appear to be one DGC thread
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> allocated per
> >>>>
> >>>>
> >>>>>>>>>> exported object.  This is using River 2.2 and Sun JDK
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> 1.6.0_17.
> >>>>
> >>>>
> >>>>>>>>>> Relevant configuration parameters are below.
> >>>>>>>>>>
> >>>>>>>>>> I am observing problems with the DGC threads not being
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> retired on
> >>>>
> >>>>
> >>>>>>>>>> a timely basis.  The exported objects are proxies
> >>>>>>>>>>
> >> for Futures
> >>
> >>>>>>>>>> which are being executed on the service.  The code
> >>>>>>>>>>
> >> pattern is
> >>
> >>>>>>>>>> such that the proxied Future goes out of lexical
> scope quite
> >>>>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> >>>>>>>>>>
> >>>>>>>>>> Under a modest load, a large number of such Futures
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> are exported
> >>>>
> >>>>
> >>>>>>>>>> which results in a large number of long lived DGC
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> threads.  This
> >>>>
> >>>>
> >>>>>>>>>> turns into a problem for the JVM due to the stack
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> allocation per
> >>>>
> >>>>
> >>>>>>>>>> thread.  Presumably this is not good for other
> >>>>>>>>>>
> >> reasons as well
> >>
> >>>>>>>>>> (e.g., scheduling).
> >>>>>>>>>>
> >>>>>>>>>> I have tried to override the leaseValue and checkInterval
> >>>>>>>>>> defaults per the configuration options below.  I
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> suspect that the
> >>>>
> >>>>
> >>>>>>>>>> lease interval is somehow not being obeyed, which is
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> presumably a
> >>>>
> >>>>
> >>>>>>>>>> problem on my end.  However, I can verify that the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> configuration
> >>>>
> >>>>
> >>>>>>>>>> values are in fact showing up in
> >>>>>>>>>> System.getProperties() for at least some of the JVMs
> >>>>>>>>>>
> >> involved
> >>
> >>>>>>>>>> (the one which drives the workload and the one that I am
> >>>>>>>>>> monitoring with the large number of DGC lease threads).
> >>>>>>>>>>
> >>>>>>>>>> Some questions:
> >>>>>>>>>>
> >>>>>>>>>> Is this one-thread-per-exported proxy the expected
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> behavior when
> >>>>
> >>>>
> >>>>>>>>>> DGC is requested for the exported object?
> >>>>>>>>>>
> >>>>>>>>>> The DGC lease checker threads appear to expire ~14 -
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> 15 minutes
> >>>>
> >>>>
> >>>>>>>>>> after I terminate the process which was
> originating the RMI
> >>>>>>>>>> requests.  This is close the sum of the default
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> leaseValue (10m)
> >>>>
> >>>>
> >>>>>>>>>> and checkInterval (5m) parameters, but maybe there is
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> some other
> >>>>
> >>>>
> >>>>>>>>>> timeout which is controlling this?  If this is the sum
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> of those
> >>>>
> >>>>
> >>>>>>>>>> parameters, why would the DGC lease threads live until
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> the sum of
> >>>>
> >>>>
> >>>>>>>>>> those values?  I thought that the lease would expire
> >>>>>>>>>>
> >> after the
> >>
> >>>>>>>>>> leaseValue (10m default).
> >>>>>>>>>>
> >>>>>>>>>> Can the issue I am observing be caused by a low heap
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> pressure on
> >>>>
> >>>>
> >>>>>>>>>> the JVM to which the RMI proxies were exported?  If it
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> fails to
> >>>>
> >>>>
> >>>>>>>>>> GC those proxies, even though they are reachable,
> could that
> >>>>>>>>>> cause DGC to continue to retain those proxies on the
> >>>>>>>>>>
> >> JVM which
> >>
> >>>>>>>>>> exported them?
> >>>>>>>>>>
> >>>>>>>>>> Is there any way to configure DGC to use a thread pool
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> or to have
> >>>>
> >>>>
> >>>>>>>>>> the leases managed by a single thread?
> >>>>>>>>>>
> >>>>>>>>>> Is it possible that there is an interaction with the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> useNIO option?
> >>>>
> >>>>
> >>>>>>>>>> Relevant options that I am using include:
> >>>>>>>>>>
> >>>>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
> >>>>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> >>>>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> >>>>>>>>>>
> >>>>>>>>>> Thanks in advance,
> >>>>>>>>>> Bryan
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>
> >
> >
>
>

RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

Peter,

Rather than testing against the trunk, I would prefer to test the change against the 2.2 release.  That will allow me to avoid validating the entire state of the trunk and let me focus on whether the change fixes the specific DGC problem.

It looks like [1] is the commit containing this fix.  Can I simply apply that patch to the clean checkout of 2.2?

Thanks,
Bryan

[1] http://mail-archives.apache.org/mod_mbox/river-commits/201201.mbox/%3C20120114105227.9890723888E7%40eris.apache.org%3E



> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au]
> Sent: Saturday, January 14, 2012 7:04 AM
> To: Bryan Thompson
> Cc: dev@river.apache.org; user@river.apache.org
> Subject: Re: [Fwd: Re: DGC threads issue]
>
> Brian,
>
> First checkout the main trunk (I've only just committed the
> fix so make sure it's updated, if you've checked it out already):
>
> svn checkout http://svn.apache.org/repos/asf/river/jtsk/trunk river
>
>
> Then create a build.properties file in trunk that contains
> something like the following:
>
> java.home=/usr/jdk/jdk1.6.0_25
> river.home=/opt/src/river
> log.config=/home/peter/logging.properties
> debug=true
> debuglevel=lines,source,vars
>
> Then from the command line
>
> $ ant all.clean all.build
>
> If you want to run the qa tests:
>
> $ ant qa.run
>
> Cheers,
>
> Peter.
>
> Bryan Thompson wrote:
> > What about the following property?  Is it still valid?
> >
> >         -Dsun.rmi.transport.tcp.connectionPool=true
> >
> > Thanks,
> > Bryan
> >
> >
> >> -----Original Message-----
> >> From: Peter Firmstone [mailto:jini@zeus.net.au]
> >> Sent: Friday, January 13, 2012 8:41 AM
> >> To: Bryan Thompson
> >> Cc: dev@river.apache.org; user@river.apache.org
> >> Subject: Re: [Fwd: Re: DGC threads issue]
> >>
> >> Brian,
> >>
> >> Do you have a Linux / Unix build environment?
> >>
> >> River isn't building yet on Window's, FreeBSD, or OpenBSD, known
> >> build environments that work are Ubuntu and Solaris.
> >> If not, that's ok, well be able to get the necessary jar's
> off Hudson.
> >>
> >> The DGC lease property values you need to set are:
> >>
> >> com.sun.jini.jeri.dgc.leaseValue=30000
> >> com.sun.jini.jeri.dgc.checkInterval=15000
> >>
> >> RMI (called JRMP) is superseded code now, River doesn't use that
> >> unless you configure your Exporter to be JrmpExporter, but you're
> >> better off sticking with BasicJeriExporter.
> >>
> >> Regards,
> >>
> >> Peter.
> >>
> >> Bryan Thompson wrote:
> >>
> >>> Peter,
> >>>
> >>> I would be happy to test out a fix on this.  Can you point
> >>>
> >> out the modified line and link me to any directions on how
> to rebuild
> >> the necessary jar(s)?
> >>
> >>> Did you already file an issue for this?  If not, I will
> >>>
> >> file an issue and then update it when I test out this fix.
> >>
> >>> Thanks,
> >>> Bryan
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Peter Firmstone [mailto:jini@zeus.net.au]
> >>>> Sent: Friday, January 13, 2012 3:21 AM
> >>>> To: dev@river.apache.org
> >>>> Cc: Bryan Thompson; user@river.apache.org
> >>>> Subject: Re: [Fwd: Re: DGC threads issue]
> >>>>
> >>>> Ok, found the problem, when I fixed River-142, I
> introduced a new
> >>>> bug, after creating a new DGC thread, I didn't set the
> >>>>
> >> running flag
> >>
> >>>> to true, so each time, a new thread is created.
> >>>>
> >>>> Thankfully it's a very easy fix.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Peter.
> >>>>
> >>>> Peter Jones wrote:
> >>>>
> >>>>
> >>>>> Peter,
> >>>>>
> >>>>> That internal Executor interface permits neither
> >>>>>
> >>>>>
> >>>> synchronous execution in the calling thread ("this method
> >>>>
> >> itself must
> >>
> >>>> not block") nor any small bound on the number of Runnables
> >>>>
> >> that can
> >>
> >>>> be executed concurrently ("execution of a given action
> must not be
> >>>> delayed indefinitely in order to complete execution of a
> different
> >>>> action passed to a different invocation of this method").
> >>>>
> >> In other
> >>
> >>>> words, concurrency-wise, it should be equivalent to "new
> >>>> Thread(runnable, name)", or what the result of
> >>>> j.u.c.Executors.newCachedThreadPool() would do.
> >>>>
> >> Otherwise, callers
> >>
> >>>> would not function correctly, because they expect to be
> >>>>
> >> able to pass
> >>
> >>>> Runnables that execute indefinitely, not just short-lived
> >>>>
> >> work tasks.
> >>
> >>>>> The real purpose of this internal Executor/ThreadPool was
> >>>>>
> >>>>>
> >>>> to provide an internal alternative to "new Thread(...)"
> that takes
> >>>> care of certain things in common, like:
> >>>>
> >>>>
> >>>>> - thread reuse instead of creation when reasonable
> (idle timeout)
> >>>>> - but still with threads (re)named for the task being executed
> >>>>> - insulation of any thread creation permission requirement from
> >>>>> context calling execute (NewThreadAction)
> >>>>> - but instead requiring direct user of thread pool to be
> >>>>>
> >>>>>
> >>>> trusted not
> >>>>
> >>>>
> >>>>> to abuse that insulation, with permission requirement
> for access
> >>>>> (ThreadPoolPermission and GetThreadPoolAction)
> >>>>> - common logging of uncaught exceptions
> >>>>>
> >>>>> -- Peter
> >>>>>
> >>>>>
> >>>>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> ThreadPool implements the com.sun.jini.thread.Executor
> interface.
> >>>>>>
> >>>>>> Because the interface states it should not block, I think
> >>>>>>
> >>>>>>
> >>>> the calling thread should execute the task when the thread pool
> >>>> becomes saturated, rather than continue to create new
> >>>>
> >> threads as per
> >>
> >>>> the current implementation.  This will ensure that the task is
> >>>> completed, the calling thread, if it uses a sequence of
> tasks with
> >>>> dependencies will have to ensure that it submits the tasks
> >>>>
> >> in order.
> >>
> >>>> Since only the caller knows the order, it makes sense for
> >>>>
> >> this to be
> >>
> >>>> the callers responsibility.  As a result the executor will
> >>>>
> >> honor the
> >>
> >>>> non blocking contract.  For that reason we'll use a zero length
> >>>> queue, probably a SynchronousQueue.
> >>>>
> >>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Peter.
> >>>>>>
> >>>>>> /**
> >>>>>> * Executor is an abstraction for a thread factory or
> >>>>>>
> >>>>>>
> >>>> thread pool for
> >>>>
> >>>>
> >>>>>> * executing actions asynchronously.
> >>>>>> *
> >>>>>> * @author    Sun Microsystems, Inc.
> >>>>>> *
> >>>>>> */
> >>>>>> public interface Executor {
> >>>>>>
> >>>>>>   /**
> >>>>>>    * Executes the given Runnable action asynchronously in
> >>>>>>
> >>>>>>
> >>>> some thread.
> >>>>
> >>>>
> >>>>>>    *
> >>>>>>    * The implementation may create a new thread to execute
> >>>>>>
> >>>>>>
> >>>> the action,
> >>>>
> >>>>
> >>>>>>    * or it may execute the action in an existing thread.
> >>>>>>    *
> >>>>>>    * The execution of a given action must not be delayed
> >>>>>>
> >>>>>>
> >>>> indefinitely
> >>>>
> >>>>
> >>>>>>    * in order to complete execution of a different action
> >>>>>>
> >>>>>>
> >>>> passed to a
> >>>>
> >>>>
> >>>>>>    * different invocation of this method.  In other words, the
> >>>>>>    * implementation must assume that there may be
> >>>>>>
> >>>>>>
> >>>> arbitrary dependencies
> >>>>
> >>>>
> >>>>>>    * between actions passed to this method, so it needs to
> >>>>>>
> >>>>>>
> >>>> be careful
> >>>>
> >>>>
> >>>>>>    * to avoid potential deadlock by delaying execution of
> >>>>>>
> >>>>>>
> >>>> one action
> >>>>
> >>>>
> >>>>>>    * indefinitely until another completes.
> >>>>>>    *
> >>>>>>    * Also, this method itself must not block, because it
> >>>>>>
> >>>>>>
> >>>> may be invoked
> >>>>
> >>>>
> >>>>>>    * by code that is serially processing data to produce
> >>>>>>
> >>>>>>
> >>>> multiple such
> >>>>
> >>>>
> >>>>>>    * arbitrarily-dependent actions that need to be executed.
> >>>>>>    *
> >>>>>>    * @param    runnable the Runnable action to execute
> >>>>>>    *
> >>>>>>    * @param    name string to include in the name of the
> >>>>>>
> >>>>>>
> >>>> thread used
> >>>>
> >>>>
> >>>>>>    * to execute the action
> >>>>>>    */
> >>>>>>   void execute(Runnable runnable, String name); }
> >>>>>>
> >>>>>> Peter Firmstone wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Thanks Brian,
> >>>>>>>
> >>>>>>> Looking at our implementation code, DGC uses an Executor
> >>>>>>>
> >>>>>>>
> >>>> called ThreadPool, it's javadoc states:
> >>>>
> >>>>
> >>>>>>> /**
> >>>>>>> * ThreadPool is a simple thread pool implementation of
> >>>>>>>
> >>>>>>>
> >>>> the Executor
> >>>>
> >>>>
> >>>>>>> * interface.
> >>>>>>> *
> >>>>>>> * A new task is always given to an idle thread, if one is
> >>>>>>>
> >>>>>>>
> >>>> available;
> >>>>
> >>>>
> >>>>>>> * otherwise, a new thread is always created.  There is
> >>>>>>>
> >> no minimum
> >>
> >>>>>>> * warm thread count, nor is there a maximum thread count
> >>>>>>>
> >>>>>>>
> >>>> (tasks are
> >>>>
> >>>>
> >>>>>>> * never queued unless there are sufficient idle threads
> >>>>>>>
> >> to execute
> >>
> >>>>>>> * them).
> >>>>>>> *
> >>>>>>> * New threads are created as daemon threads in the thread
> >>>>>>>
> >>>>>>>
> >>>> group that
> >>>>
> >>>>
> >>>>>>> * was passed to the ThreadPool instance's constructor.  Each
> >>>>>>> thread's
> >>>>>>> * name is the prefix NewThreadAction.NAME_PREFIX
> >>>>>>>
> >> followed by the
> >>
> >>>>>>> name
> >>>>>>> * of the task it is currently executing, or "Idle" if it is
> >>>>>>> currently
> >>>>>>> * idle.
> >>>>>>>
> >>>>>>> ThreadPool predates Java 5, it looks like we can fix this
> >>>>>>>
> >>>>>>>
> >>>> by using an Executor from Java 5, we can look at limiting
> >>>>
> >> the number
> >>
> >>>> of threads created based on available CPU's and a scaling
> >>>>
> >> factor and
> >>
> >>>> place the tasks in a BlockingQueue, so if the queue is
> filled, it
> >>>> blocks.
> >>>>
> >>>>
> >>>>>>> Can you report the issue as a Bug on Jira for me, I'll
> >>>>>>>
> >>>>>>>
> >>>> fix this before the next release.
> >>>>
> >>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Peter.
> >>>>>>>
> >>>>>>> Peter Firmstone wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Hi Peter,
> >>>>>>>>
> >>>>>>>> I was wondering if you had any thoughts on this post
> >>>>>>>>
> >>>>>>>>
> >>>> from Bryan on River users?
> >>>>
> >>>>
> >>>>>>>> Hope you don't mind me asking ;)
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>>
> >>>>>>>> Peter Firmstone.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>
> -------------------------------------------------------------------
> >>>>
> >>>>
> >>>>>>>> -----
> >>>>>>>>
> >>>>>>>> Subject:
> >>>>>>>> Re: DGC threads issue
> >>>>>>>> From:
> >>>>>>>> Tom Hobbs <tv...@googlemail.com>
> >>>>>>>> Date:
> >>>>>>>> Thu, 12 Jan 2012 20:45:01 +0000
> >>>>>>>> To:
> >>>>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>>>
> >>>>>>>> To:
> >>>>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi Bryan,
> >>>>>>>>
> >>>>>>>> Sorry that no one got back to you about this.  I'm
> >>>>>>>>
> >> afraid that I
> >>
> >>>>>>>> don't know the answer to your question, I've copied
> >>>>>>>>
> >> the dev list
> >>
> >>>>>>>> into this email in case someone who monitors that list
> >>>>>>>>
> >> (but not
> >>
> >>>>>>>> this one) has any ideas.
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>>
> >>>>>>>> Tom
> >>>>>>>>
> >>>>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> >>>>>>>>
> >>>>>>>>
> >>>> <br...@systap.com> wrote:
> >>>>
> >>>>
> >>>>>>>>
> >>>>>>>>> Just to follow up on this thread myself.  I modified
> >>>>>>>>>
> >>>>>>>>>
> >>>> the pattern to return a "thick" future rather than a
> proxy for the
> >>>> future.  This caused the RMI call to wait on the server
> until the
> >>>> future was done and then sent back the outcome.  This
> >>>>
> >> "fixed" the DGC
> >>
> >>>> memory/thread leak by reducing the number of exported proxies
> >>>> drammatically.
> >>>>
> >>>>
> >>>>>>>>> In terms of best practices, is distributed DGC simply
> >>>>>>>>>
> >>>>>>>>>
> >>>> not useful for exported objects with short life spans?
> >>>>
> >> Can it only
> >>
> >>>> be used with proxies for relatively long lived services?
> >>>>
> >>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Bryan
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Bryan Thompson
> >>>>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>>>>>>>>> To: user@river.apache.org
> >>>>>>>>>> Subject: DGC threads issue
> >>>>>>>>>>
> >>>>>>>>>> Hello,
> >>>>>>>>>>
> >>>>>>>>>> Background:
> >>>>>>>>>>
> >>>>>>>>>> I am seeing what would appear to be one DGC thread
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> allocated per
> >>>>
> >>>>
> >>>>>>>>>> exported object.  This is using River 2.2 and Sun JDK
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> 1.6.0_17.
> >>>>
> >>>>
> >>>>>>>>>> Relevant configuration parameters are below.
> >>>>>>>>>>
> >>>>>>>>>> I am observing problems with the DGC threads not being
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> retired on
> >>>>
> >>>>
> >>>>>>>>>> a timely basis.  The exported objects are proxies
> >>>>>>>>>>
> >> for Futures
> >>
> >>>>>>>>>> which are being executed on the service.  The code
> >>>>>>>>>>
> >> pattern is
> >>
> >>>>>>>>>> such that the proxied Future goes out of lexical
> scope quite
> >>>>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> >>>>>>>>>>
> >>>>>>>>>> Under a modest load, a large number of such Futures
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> are exported
> >>>>
> >>>>
> >>>>>>>>>> which results in a large number of long lived DGC
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> threads.  This
> >>>>
> >>>>
> >>>>>>>>>> turns into a problem for the JVM due to the stack
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> allocation per
> >>>>
> >>>>
> >>>>>>>>>> thread.  Presumably this is not good for other
> >>>>>>>>>>
> >> reasons as well
> >>
> >>>>>>>>>> (e.g., scheduling).
> >>>>>>>>>>
> >>>>>>>>>> I have tried to override the leaseValue and checkInterval
> >>>>>>>>>> defaults per the configuration options below.  I
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> suspect that the
> >>>>
> >>>>
> >>>>>>>>>> lease interval is somehow not being obeyed, which is
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> presumably a
> >>>>
> >>>>
> >>>>>>>>>> problem on my end.  However, I can verify that the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> configuration
> >>>>
> >>>>
> >>>>>>>>>> values are in fact showing up in
> >>>>>>>>>> System.getProperties() for at least some of the JVMs
> >>>>>>>>>>
> >> involved
> >>
> >>>>>>>>>> (the one which drives the workload and the one that I am
> >>>>>>>>>> monitoring with the large number of DGC lease threads).
> >>>>>>>>>>
> >>>>>>>>>> Some questions:
> >>>>>>>>>>
> >>>>>>>>>> Is this one-thread-per-exported proxy the expected
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> behavior when
> >>>>
> >>>>
> >>>>>>>>>> DGC is requested for the exported object?
> >>>>>>>>>>
> >>>>>>>>>> The DGC lease checker threads appear to expire ~14 -
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> 15 minutes
> >>>>
> >>>>
> >>>>>>>>>> after I terminate the process which was
> originating the RMI
> >>>>>>>>>> requests.  This is close the sum of the default
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> leaseValue (10m)
> >>>>
> >>>>
> >>>>>>>>>> and checkInterval (5m) parameters, but maybe there is
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> some other
> >>>>
> >>>>
> >>>>>>>>>> timeout which is controlling this?  If this is the sum
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> of those
> >>>>
> >>>>
> >>>>>>>>>> parameters, why would the DGC lease threads live until
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> the sum of
> >>>>
> >>>>
> >>>>>>>>>> those values?  I thought that the lease would expire
> >>>>>>>>>>
> >> after the
> >>
> >>>>>>>>>> leaseValue (10m default).
> >>>>>>>>>>
> >>>>>>>>>> Can the issue I am observing be caused by a low heap
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> pressure on
> >>>>
> >>>>
> >>>>>>>>>> the JVM to which the RMI proxies were exported?  If it
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> fails to
> >>>>
> >>>>
> >>>>>>>>>> GC those proxies, even though they are reachable,
> could that
> >>>>>>>>>> cause DGC to continue to retain those proxies on the
> >>>>>>>>>>
> >> JVM which
> >>
> >>>>>>>>>> exported them?
> >>>>>>>>>>
> >>>>>>>>>> Is there any way to configure DGC to use a thread pool
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> or to have
> >>>>
> >>>>
> >>>>>>>>>> the leases managed by a single thread?
> >>>>>>>>>>
> >>>>>>>>>> Is it possible that there is an interaction with the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> useNIO option?
> >>>>
> >>>>
> >>>>>>>>>> Relevant options that I am using include:
> >>>>>>>>>>
> >>>>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
> >>>>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> >>>>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> >>>>>>>>>>
> >>>>>>>>>> Thanks in advance,
> >>>>>>>>>> Bryan
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>
> >
> >
>
>

Re: [Fwd: Re: DGC threads issue]

[Peter Firmstone <ji...@zeus.net.au>]

Brian,

First checkout the main trunk (I've only just committed the fix so make 
sure it's updated, if you've checked it out already):

svn checkout http://svn.apache.org/repos/asf/river/jtsk/trunk river


Then create a build.properties file in trunk that contains something like the following:

java.home=/usr/jdk/jdk1.6.0_25
river.home=/opt/src/river
log.config=/home/peter/logging.properties
debug=true
debuglevel=lines,source,vars

Then from the command line

$ ant all.clean all.build

If you want to run the qa tests:

$ ant qa.run

Cheers,

Peter.

Bryan Thompson wrote:
> What about the following property?  Is it still valid?
>
>         -Dsun.rmi.transport.tcp.connectionPool=true
>
> Thanks,
> Bryan
>
>   
>> -----Original Message-----
>> From: Peter Firmstone [mailto:jini@zeus.net.au]
>> Sent: Friday, January 13, 2012 8:41 AM
>> To: Bryan Thompson
>> Cc: dev@river.apache.org; user@river.apache.org
>> Subject: Re: [Fwd: Re: DGC threads issue]
>>
>> Brian,
>>
>> Do you have a Linux / Unix build environment?
>>
>> River isn't building yet on Window's, FreeBSD, or OpenBSD,
>> known build environments that work are Ubuntu and Solaris.
>> If not, that's ok, well be able to get the necessary jar's off Hudson.
>>
>> The DGC lease property values you need to set are:
>>
>> com.sun.jini.jeri.dgc.leaseValue=30000
>> com.sun.jini.jeri.dgc.checkInterval=15000
>>
>> RMI (called JRMP) is superseded code now, River doesn't use
>> that unless you configure your Exporter to be JrmpExporter,
>> but you're better off sticking with BasicJeriExporter.
>>
>> Regards,
>>
>> Peter.
>>
>> Bryan Thompson wrote:
>>     
>>> Peter,
>>>
>>> I would be happy to test out a fix on this.  Can you point
>>>       
>> out the modified line and link me to any directions on how to
>> rebuild the necessary jar(s)?
>>     
>>> Did you already file an issue for this?  If not, I will
>>>       
>> file an issue and then update it when I test out this fix.
>>     
>>> Thanks,
>>> Bryan
>>>
>>>
>>>       
>>>> -----Original Message-----
>>>> From: Peter Firmstone [mailto:jini@zeus.net.au]
>>>> Sent: Friday, January 13, 2012 3:21 AM
>>>> To: dev@river.apache.org
>>>> Cc: Bryan Thompson; user@river.apache.org
>>>> Subject: Re: [Fwd: Re: DGC threads issue]
>>>>
>>>> Ok, found the problem, when I fixed River-142, I introduced a new
>>>> bug, after creating a new DGC thread, I didn't set the
>>>>         
>> running flag
>>     
>>>> to true, so each time, a new thread is created.
>>>>
>>>> Thankfully it's a very easy fix.
>>>>
>>>> Cheers,
>>>>
>>>> Peter.
>>>>
>>>> Peter Jones wrote:
>>>>
>>>>         
>>>>> Peter,
>>>>>
>>>>> That internal Executor interface permits neither
>>>>>
>>>>>           
>>>> synchronous execution in the calling thread ("this method
>>>>         
>> itself must
>>     
>>>> not block") nor any small bound on the number of Runnables
>>>>         
>> that can
>>     
>>>> be executed concurrently ("execution of a given action must not be
>>>> delayed indefinitely in order to complete execution of a different
>>>> action passed to a different invocation of this method").
>>>>         
>> In other
>>     
>>>> words, concurrency-wise, it should be equivalent to "new
>>>> Thread(runnable, name)", or what the result of
>>>> j.u.c.Executors.newCachedThreadPool() would do.
>>>>         
>> Otherwise, callers
>>     
>>>> would not function correctly, because they expect to be
>>>>         
>> able to pass
>>     
>>>> Runnables that execute indefinitely, not just short-lived
>>>>         
>> work tasks.
>>     
>>>>> The real purpose of this internal Executor/ThreadPool was
>>>>>
>>>>>           
>>>> to provide an internal alternative to "new Thread(...)" that takes
>>>> care of certain things in common, like:
>>>>
>>>>         
>>>>> - thread reuse instead of creation when reasonable (idle timeout)
>>>>> - but still with threads (re)named for the task being executed
>>>>> - insulation of any thread creation permission requirement from
>>>>> context calling execute (NewThreadAction)
>>>>> - but instead requiring direct user of thread pool to be
>>>>>
>>>>>           
>>>> trusted not
>>>>
>>>>         
>>>>> to abuse that insulation, with permission requirement for access
>>>>> (ThreadPoolPermission and GetThreadPoolAction)
>>>>> - common logging of uncaught exceptions
>>>>>
>>>>> -- Peter
>>>>>
>>>>>
>>>>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> ThreadPool implements the com.sun.jini.thread.Executor interface.
>>>>>>
>>>>>> Because the interface states it should not block, I think
>>>>>>
>>>>>>             
>>>> the calling thread should execute the task when the thread pool
>>>> becomes saturated, rather than continue to create new
>>>>         
>> threads as per
>>     
>>>> the current implementation.  This will ensure that the task is
>>>> completed, the calling thread, if it uses a sequence of tasks with
>>>> dependencies will have to ensure that it submits the tasks
>>>>         
>> in order.
>>     
>>>> Since only the caller knows the order, it makes sense for
>>>>         
>> this to be
>>     
>>>> the callers responsibility.  As a result the executor will
>>>>         
>> honor the
>>     
>>>> non blocking contract.  For that reason we'll use a zero length
>>>> queue, probably a SynchronousQueue.
>>>>
>>>>         
>>>>>> Regards,
>>>>>>
>>>>>> Peter.
>>>>>>
>>>>>> /**
>>>>>> * Executor is an abstraction for a thread factory or
>>>>>>
>>>>>>             
>>>> thread pool for
>>>>
>>>>         
>>>>>> * executing actions asynchronously.
>>>>>> *
>>>>>> * @author    Sun Microsystems, Inc.
>>>>>> *
>>>>>> */
>>>>>> public interface Executor {
>>>>>>
>>>>>>   /**
>>>>>>    * Executes the given Runnable action asynchronously in
>>>>>>
>>>>>>             
>>>> some thread.
>>>>
>>>>         
>>>>>>    *
>>>>>>    * The implementation may create a new thread to execute
>>>>>>
>>>>>>             
>>>> the action,
>>>>
>>>>         
>>>>>>    * or it may execute the action in an existing thread.
>>>>>>    *
>>>>>>    * The execution of a given action must not be delayed
>>>>>>
>>>>>>             
>>>> indefinitely
>>>>
>>>>         
>>>>>>    * in order to complete execution of a different action
>>>>>>
>>>>>>             
>>>> passed to a
>>>>
>>>>         
>>>>>>    * different invocation of this method.  In other words, the
>>>>>>    * implementation must assume that there may be
>>>>>>
>>>>>>             
>>>> arbitrary dependencies
>>>>
>>>>         
>>>>>>    * between actions passed to this method, so it needs to
>>>>>>
>>>>>>             
>>>> be careful
>>>>
>>>>         
>>>>>>    * to avoid potential deadlock by delaying execution of
>>>>>>
>>>>>>             
>>>> one action
>>>>
>>>>         
>>>>>>    * indefinitely until another completes.
>>>>>>    *
>>>>>>    * Also, this method itself must not block, because it
>>>>>>
>>>>>>             
>>>> may be invoked
>>>>
>>>>         
>>>>>>    * by code that is serially processing data to produce
>>>>>>
>>>>>>             
>>>> multiple such
>>>>
>>>>         
>>>>>>    * arbitrarily-dependent actions that need to be executed.
>>>>>>    *
>>>>>>    * @param    runnable the Runnable action to execute
>>>>>>    *
>>>>>>    * @param    name string to include in the name of the
>>>>>>
>>>>>>             
>>>> thread used
>>>>
>>>>         
>>>>>>    * to execute the action
>>>>>>    */
>>>>>>   void execute(Runnable runnable, String name); }
>>>>>>
>>>>>> Peter Firmstone wrote:
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Thanks Brian,
>>>>>>>
>>>>>>> Looking at our implementation code, DGC uses an Executor
>>>>>>>
>>>>>>>               
>>>> called ThreadPool, it's javadoc states:
>>>>
>>>>         
>>>>>>> /**
>>>>>>> * ThreadPool is a simple thread pool implementation of
>>>>>>>
>>>>>>>               
>>>> the Executor
>>>>
>>>>         
>>>>>>> * interface.
>>>>>>> *
>>>>>>> * A new task is always given to an idle thread, if one is
>>>>>>>
>>>>>>>               
>>>> available;
>>>>
>>>>         
>>>>>>> * otherwise, a new thread is always created.  There is
>>>>>>>               
>> no minimum
>>     
>>>>>>> * warm thread count, nor is there a maximum thread count
>>>>>>>
>>>>>>>               
>>>> (tasks are
>>>>
>>>>         
>>>>>>> * never queued unless there are sufficient idle threads
>>>>>>>               
>> to execute
>>     
>>>>>>> * them).
>>>>>>> *
>>>>>>> * New threads are created as daemon threads in the thread
>>>>>>>
>>>>>>>               
>>>> group that
>>>>
>>>>         
>>>>>>> * was passed to the ThreadPool instance's constructor.  Each
>>>>>>> thread's
>>>>>>> * name is the prefix NewThreadAction.NAME_PREFIX
>>>>>>>               
>> followed by the
>>     
>>>>>>> name
>>>>>>> * of the task it is currently executing, or "Idle" if it is
>>>>>>> currently
>>>>>>> * idle.
>>>>>>>
>>>>>>> ThreadPool predates Java 5, it looks like we can fix this
>>>>>>>
>>>>>>>               
>>>> by using an Executor from Java 5, we can look at limiting
>>>>         
>> the number
>>     
>>>> of threads created based on available CPU's and a scaling
>>>>         
>> factor and
>>     
>>>> place the tasks in a BlockingQueue, so if the queue is filled, it
>>>> blocks.
>>>>
>>>>         
>>>>>>> Can you report the issue as a Bug on Jira for me, I'll
>>>>>>>
>>>>>>>               
>>>> fix this before the next release.
>>>>
>>>>         
>>>>>>> Regards,
>>>>>>>
>>>>>>> Peter.
>>>>>>>
>>>>>>> Peter Firmstone wrote:
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Hi Peter,
>>>>>>>>
>>>>>>>> I was wondering if you had any thoughts on this post
>>>>>>>>
>>>>>>>>                 
>>>> from Bryan on River users?
>>>>
>>>>         
>>>>>>>> Hope you don't mind me asking ;)
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>>
>>>>>>>> Peter Firmstone.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>> -------------------------------------------------------------------
>>>>
>>>>         
>>>>>>>> -----
>>>>>>>>
>>>>>>>> Subject:
>>>>>>>> Re: DGC threads issue
>>>>>>>> From:
>>>>>>>> Tom Hobbs <tv...@googlemail.com>
>>>>>>>> Date:
>>>>>>>> Thu, 12 Jan 2012 20:45:01 +0000
>>>>>>>> To:
>>>>>>>> user@river.apache.org, dev@river.apache.org
>>>>>>>>
>>>>>>>> To:
>>>>>>>> user@river.apache.org, dev@river.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Bryan,
>>>>>>>>
>>>>>>>> Sorry that no one got back to you about this.  I'm
>>>>>>>>                 
>> afraid that I
>>     
>>>>>>>> don't know the answer to your question, I've copied
>>>>>>>>                 
>> the dev list
>>     
>>>>>>>> into this email in case someone who monitors that list
>>>>>>>>                 
>> (but not
>>     
>>>>>>>> this one) has any ideas.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Tom
>>>>>>>>
>>>>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
>>>>>>>>
>>>>>>>>                 
>>>> <br...@systap.com> wrote:
>>>>
>>>>         
>>>>>>>>                 
>>>>>>>>> Just to follow up on this thread myself.  I modified
>>>>>>>>>
>>>>>>>>>                   
>>>> the pattern to return a "thick" future rather than a proxy for the
>>>> future.  This caused the RMI call to wait on the server until the
>>>> future was done and then sent back the outcome.  This
>>>>         
>> "fixed" the DGC
>>     
>>>> memory/thread leak by reducing the number of exported proxies
>>>> drammatically.
>>>>
>>>>         
>>>>>>>>> In terms of best practices, is distributed DGC simply
>>>>>>>>>
>>>>>>>>>                   
>>>> not useful for exported objects with short life spans?
>>>>         
>> Can it only
>>     
>>>> be used with proxies for relatively long lived services?
>>>>
>>>>         
>>>>>>>>> Thanks,
>>>>>>>>> Bryan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Bryan Thompson
>>>>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
>>>>>>>>>> To: user@river.apache.org
>>>>>>>>>> Subject: DGC threads issue
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> Background:
>>>>>>>>>>
>>>>>>>>>> I am seeing what would appear to be one DGC thread
>>>>>>>>>>
>>>>>>>>>>                     
>>>> allocated per
>>>>
>>>>         
>>>>>>>>>> exported object.  This is using River 2.2 and Sun JDK
>>>>>>>>>>
>>>>>>>>>>                     
>>>> 1.6.0_17.
>>>>
>>>>         
>>>>>>>>>> Relevant configuration parameters are below.
>>>>>>>>>>
>>>>>>>>>> I am observing problems with the DGC threads not being
>>>>>>>>>>
>>>>>>>>>>                     
>>>> retired on
>>>>
>>>>         
>>>>>>>>>> a timely basis.  The exported objects are proxies
>>>>>>>>>>                     
>> for Futures
>>     
>>>>>>>>>> which are being executed on the service.  The code
>>>>>>>>>>                     
>> pattern is
>>     
>>>>>>>>>> such that the proxied Future goes out of lexical scope quite
>>>>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
>>>>>>>>>>
>>>>>>>>>> Under a modest load, a large number of such Futures
>>>>>>>>>>
>>>>>>>>>>                     
>>>> are exported
>>>>
>>>>         
>>>>>>>>>> which results in a large number of long lived DGC
>>>>>>>>>>
>>>>>>>>>>                     
>>>> threads.  This
>>>>
>>>>         
>>>>>>>>>> turns into a problem for the JVM due to the stack
>>>>>>>>>>
>>>>>>>>>>                     
>>>> allocation per
>>>>
>>>>         
>>>>>>>>>> thread.  Presumably this is not good for other
>>>>>>>>>>                     
>> reasons as well
>>     
>>>>>>>>>> (e.g., scheduling).
>>>>>>>>>>
>>>>>>>>>> I have tried to override the leaseValue and checkInterval
>>>>>>>>>> defaults per the configuration options below.  I
>>>>>>>>>>
>>>>>>>>>>                     
>>>> suspect that the
>>>>
>>>>         
>>>>>>>>>> lease interval is somehow not being obeyed, which is
>>>>>>>>>>
>>>>>>>>>>                     
>>>> presumably a
>>>>
>>>>         
>>>>>>>>>> problem on my end.  However, I can verify that the
>>>>>>>>>>
>>>>>>>>>>                     
>>>> configuration
>>>>
>>>>         
>>>>>>>>>> values are in fact showing up in
>>>>>>>>>> System.getProperties() for at least some of the JVMs
>>>>>>>>>>                     
>> involved
>>     
>>>>>>>>>> (the one which drives the workload and the one that I am
>>>>>>>>>> monitoring with the large number of DGC lease threads).
>>>>>>>>>>
>>>>>>>>>> Some questions:
>>>>>>>>>>
>>>>>>>>>> Is this one-thread-per-exported proxy the expected
>>>>>>>>>>
>>>>>>>>>>                     
>>>> behavior when
>>>>
>>>>         
>>>>>>>>>> DGC is requested for the exported object?
>>>>>>>>>>
>>>>>>>>>> The DGC lease checker threads appear to expire ~14 -
>>>>>>>>>>
>>>>>>>>>>                     
>>>> 15 minutes
>>>>
>>>>         
>>>>>>>>>> after I terminate the process which was originating the RMI
>>>>>>>>>> requests.  This is close the sum of the default
>>>>>>>>>>
>>>>>>>>>>                     
>>>> leaseValue (10m)
>>>>
>>>>         
>>>>>>>>>> and checkInterval (5m) parameters, but maybe there is
>>>>>>>>>>
>>>>>>>>>>                     
>>>> some other
>>>>
>>>>         
>>>>>>>>>> timeout which is controlling this?  If this is the sum
>>>>>>>>>>
>>>>>>>>>>                     
>>>> of those
>>>>
>>>>         
>>>>>>>>>> parameters, why would the DGC lease threads live until
>>>>>>>>>>
>>>>>>>>>>                     
>>>> the sum of
>>>>
>>>>         
>>>>>>>>>> those values?  I thought that the lease would expire
>>>>>>>>>>                     
>> after the
>>     
>>>>>>>>>> leaseValue (10m default).
>>>>>>>>>>
>>>>>>>>>> Can the issue I am observing be caused by a low heap
>>>>>>>>>>
>>>>>>>>>>                     
>>>> pressure on
>>>>
>>>>         
>>>>>>>>>> the JVM to which the RMI proxies were exported?  If it
>>>>>>>>>>
>>>>>>>>>>                     
>>>> fails to
>>>>
>>>>         
>>>>>>>>>> GC those proxies, even though they are reachable, could that
>>>>>>>>>> cause DGC to continue to retain those proxies on the
>>>>>>>>>>                     
>> JVM which
>>     
>>>>>>>>>> exported them?
>>>>>>>>>>
>>>>>>>>>> Is there any way to configure DGC to use a thread pool
>>>>>>>>>>
>>>>>>>>>>                     
>>>> or to have
>>>>
>>>>         
>>>>>>>>>> the leases managed by a single thread?
>>>>>>>>>>
>>>>>>>>>> Is it possible that there is an interaction with the
>>>>>>>>>>
>>>>>>>>>>                     
>>>> useNIO option?
>>>>
>>>>         
>>>>>>>>>> Relevant options that I am using include:
>>>>>>>>>>
>>>>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
>>>>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
>>>>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
>>>>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
>>>>>>>>>>
>>>>>>>>>> Thanks in advance,
>>>>>>>>>> Bryan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>
>>>>>>>>                 
>>>>>>>               
>>>>>           
>>     
>
>   

Re: [Fwd: Re: DGC threads issue]

[Peter Firmstone <ji...@zeus.net.au>]

Only if you use JrmpExporter.

I'll get back to shortly with some build instructions.

Cheers,

Peter.

Bryan Thompson wrote:
> What about the following property?  Is it still valid?
>
>         -Dsun.rmi.transport.tcp.connectionPool=true
>
> Thanks,
> Bryan
>
>   
>> -----Original Message-----
>> From: Peter Firmstone [mailto:jini@zeus.net.au]
>> Sent: Friday, January 13, 2012 8:41 AM
>> To: Bryan Thompson
>> Cc: dev@river.apache.org; user@river.apache.org
>> Subject: Re: [Fwd: Re: DGC threads issue]
>>
>> Brian,
>>
>> Do you have a Linux / Unix build environment?
>>
>> River isn't building yet on Window's, FreeBSD, or OpenBSD,
>> known build environments that work are Ubuntu and Solaris.
>> If not, that's ok, well be able to get the necessary jar's off Hudson.
>>
>> The DGC lease property values you need to set are:
>>
>> com.sun.jini.jeri.dgc.leaseValue=30000
>> com.sun.jini.jeri.dgc.checkInterval=15000
>>
>> RMI (called JRMP) is superseded code now, River doesn't use
>> that unless you configure your Exporter to be JrmpExporter,
>> but you're better off sticking with BasicJeriExporter.
>>
>> Regards,
>>
>> Peter.
>>
>> Bryan Thompson wrote:
>>     
>>> Peter,
>>>
>>> I would be happy to test out a fix on this.  Can you point
>>>       
>> out the modified line and link me to any directions on how to
>> rebuild the necessary jar(s)?
>>     
>>> Did you already file an issue for this?  If not, I will
>>>       
>> file an issue and then update it when I test out this fix.
>>     
>>> Thanks,
>>> Bryan
>>>
>>>
>>>       
>>>> -----Original Message-----
>>>> From: Peter Firmstone [mailto:jini@zeus.net.au]
>>>> Sent: Friday, January 13, 2012 3:21 AM
>>>> To: dev@river.apache.org
>>>> Cc: Bryan Thompson; user@river.apache.org
>>>> Subject: Re: [Fwd: Re: DGC threads issue]
>>>>
>>>> Ok, found the problem, when I fixed River-142, I introduced a new
>>>> bug, after creating a new DGC thread, I didn't set the
>>>>         
>> running flag
>>     
>>>> to true, so each time, a new thread is created.
>>>>
>>>> Thankfully it's a very easy fix.
>>>>
>>>> Cheers,
>>>>
>>>> Peter.
>>>>
>>>> Peter Jones wrote:
>>>>
>>>>         
>>>>> Peter,
>>>>>
>>>>> That internal Executor interface permits neither
>>>>>
>>>>>           
>>>> synchronous execution in the calling thread ("this method
>>>>         
>> itself must
>>     
>>>> not block") nor any small bound on the number of Runnables
>>>>         
>> that can
>>     
>>>> be executed concurrently ("execution of a given action must not be
>>>> delayed indefinitely in order to complete execution of a different
>>>> action passed to a different invocation of this method").
>>>>         
>> In other
>>     
>>>> words, concurrency-wise, it should be equivalent to "new
>>>> Thread(runnable, name)", or what the result of
>>>> j.u.c.Executors.newCachedThreadPool() would do.
>>>>         
>> Otherwise, callers
>>     
>>>> would not function correctly, because they expect to be
>>>>         
>> able to pass
>>     
>>>> Runnables that execute indefinitely, not just short-lived
>>>>         
>> work tasks.
>>     
>>>>> The real purpose of this internal Executor/ThreadPool was
>>>>>
>>>>>           
>>>> to provide an internal alternative to "new Thread(...)" that takes
>>>> care of certain things in common, like:
>>>>
>>>>         
>>>>> - thread reuse instead of creation when reasonable (idle timeout)
>>>>> - but still with threads (re)named for the task being executed
>>>>> - insulation of any thread creation permission requirement from
>>>>> context calling execute (NewThreadAction)
>>>>> - but instead requiring direct user of thread pool to be
>>>>>
>>>>>           
>>>> trusted not
>>>>
>>>>         
>>>>> to abuse that insulation, with permission requirement for access
>>>>> (ThreadPoolPermission and GetThreadPoolAction)
>>>>> - common logging of uncaught exceptions
>>>>>
>>>>> -- Peter
>>>>>
>>>>>
>>>>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> ThreadPool implements the com.sun.jini.thread.Executor interface.
>>>>>>
>>>>>> Because the interface states it should not block, I think
>>>>>>
>>>>>>             
>>>> the calling thread should execute the task when the thread pool
>>>> becomes saturated, rather than continue to create new
>>>>         
>> threads as per
>>     
>>>> the current implementation.  This will ensure that the task is
>>>> completed, the calling thread, if it uses a sequence of tasks with
>>>> dependencies will have to ensure that it submits the tasks
>>>>         
>> in order.
>>     
>>>> Since only the caller knows the order, it makes sense for
>>>>         
>> this to be
>>     
>>>> the callers responsibility.  As a result the executor will
>>>>         
>> honor the
>>     
>>>> non blocking contract.  For that reason we'll use a zero length
>>>> queue, probably a SynchronousQueue.
>>>>
>>>>         
>>>>>> Regards,
>>>>>>
>>>>>> Peter.
>>>>>>
>>>>>> /**
>>>>>> * Executor is an abstraction for a thread factory or
>>>>>>
>>>>>>             
>>>> thread pool for
>>>>
>>>>         
>>>>>> * executing actions asynchronously.
>>>>>> *
>>>>>> * @author    Sun Microsystems, Inc.
>>>>>> *
>>>>>> */
>>>>>> public interface Executor {
>>>>>>
>>>>>>   /**
>>>>>>    * Executes the given Runnable action asynchronously in
>>>>>>
>>>>>>             
>>>> some thread.
>>>>
>>>>         
>>>>>>    *
>>>>>>    * The implementation may create a new thread to execute
>>>>>>
>>>>>>             
>>>> the action,
>>>>
>>>>         
>>>>>>    * or it may execute the action in an existing thread.
>>>>>>    *
>>>>>>    * The execution of a given action must not be delayed
>>>>>>
>>>>>>             
>>>> indefinitely
>>>>
>>>>         
>>>>>>    * in order to complete execution of a different action
>>>>>>
>>>>>>             
>>>> passed to a
>>>>
>>>>         
>>>>>>    * different invocation of this method.  In other words, the
>>>>>>    * implementation must assume that there may be
>>>>>>
>>>>>>             
>>>> arbitrary dependencies
>>>>
>>>>         
>>>>>>    * between actions passed to this method, so it needs to
>>>>>>
>>>>>>             
>>>> be careful
>>>>
>>>>         
>>>>>>    * to avoid potential deadlock by delaying execution of
>>>>>>
>>>>>>             
>>>> one action
>>>>
>>>>         
>>>>>>    * indefinitely until another completes.
>>>>>>    *
>>>>>>    * Also, this method itself must not block, because it
>>>>>>
>>>>>>             
>>>> may be invoked
>>>>
>>>>         
>>>>>>    * by code that is serially processing data to produce
>>>>>>
>>>>>>             
>>>> multiple such
>>>>
>>>>         
>>>>>>    * arbitrarily-dependent actions that need to be executed.
>>>>>>    *
>>>>>>    * @param    runnable the Runnable action to execute
>>>>>>    *
>>>>>>    * @param    name string to include in the name of the
>>>>>>
>>>>>>             
>>>> thread used
>>>>
>>>>         
>>>>>>    * to execute the action
>>>>>>    */
>>>>>>   void execute(Runnable runnable, String name); }
>>>>>>
>>>>>> Peter Firmstone wrote:
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Thanks Brian,
>>>>>>>
>>>>>>> Looking at our implementation code, DGC uses an Executor
>>>>>>>
>>>>>>>               
>>>> called ThreadPool, it's javadoc states:
>>>>
>>>>         
>>>>>>> /**
>>>>>>> * ThreadPool is a simple thread pool implementation of
>>>>>>>
>>>>>>>               
>>>> the Executor
>>>>
>>>>         
>>>>>>> * interface.
>>>>>>> *
>>>>>>> * A new task is always given to an idle thread, if one is
>>>>>>>
>>>>>>>               
>>>> available;
>>>>
>>>>         
>>>>>>> * otherwise, a new thread is always created.  There is
>>>>>>>               
>> no minimum
>>     
>>>>>>> * warm thread count, nor is there a maximum thread count
>>>>>>>
>>>>>>>               
>>>> (tasks are
>>>>
>>>>         
>>>>>>> * never queued unless there are sufficient idle threads
>>>>>>>               
>> to execute
>>     
>>>>>>> * them).
>>>>>>> *
>>>>>>> * New threads are created as daemon threads in the thread
>>>>>>>
>>>>>>>               
>>>> group that
>>>>
>>>>         
>>>>>>> * was passed to the ThreadPool instance's constructor.  Each
>>>>>>> thread's
>>>>>>> * name is the prefix NewThreadAction.NAME_PREFIX
>>>>>>>               
>> followed by the
>>     
>>>>>>> name
>>>>>>> * of the task it is currently executing, or "Idle" if it is
>>>>>>> currently
>>>>>>> * idle.
>>>>>>>
>>>>>>> ThreadPool predates Java 5, it looks like we can fix this
>>>>>>>
>>>>>>>               
>>>> by using an Executor from Java 5, we can look at limiting
>>>>         
>> the number
>>     
>>>> of threads created based on available CPU's and a scaling
>>>>         
>> factor and
>>     
>>>> place the tasks in a BlockingQueue, so if the queue is filled, it
>>>> blocks.
>>>>
>>>>         
>>>>>>> Can you report the issue as a Bug on Jira for me, I'll
>>>>>>>
>>>>>>>               
>>>> fix this before the next release.
>>>>
>>>>         
>>>>>>> Regards,
>>>>>>>
>>>>>>> Peter.
>>>>>>>
>>>>>>> Peter Firmstone wrote:
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Hi Peter,
>>>>>>>>
>>>>>>>> I was wondering if you had any thoughts on this post
>>>>>>>>
>>>>>>>>                 
>>>> from Bryan on River users?
>>>>
>>>>         
>>>>>>>> Hope you don't mind me asking ;)
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>>
>>>>>>>> Peter Firmstone.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>> -------------------------------------------------------------------
>>>>
>>>>         
>>>>>>>> -----
>>>>>>>>
>>>>>>>> Subject:
>>>>>>>> Re: DGC threads issue
>>>>>>>> From:
>>>>>>>> Tom Hobbs <tv...@googlemail.com>
>>>>>>>> Date:
>>>>>>>> Thu, 12 Jan 2012 20:45:01 +0000
>>>>>>>> To:
>>>>>>>> user@river.apache.org, dev@river.apache.org
>>>>>>>>
>>>>>>>> To:
>>>>>>>> user@river.apache.org, dev@river.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Bryan,
>>>>>>>>
>>>>>>>> Sorry that no one got back to you about this.  I'm
>>>>>>>>                 
>> afraid that I
>>     
>>>>>>>> don't know the answer to your question, I've copied
>>>>>>>>                 
>> the dev list
>>     
>>>>>>>> into this email in case someone who monitors that list
>>>>>>>>                 
>> (but not
>>     
>>>>>>>> this one) has any ideas.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Tom
>>>>>>>>
>>>>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
>>>>>>>>
>>>>>>>>                 
>>>> <br...@systap.com> wrote:
>>>>
>>>>         
>>>>>>>>                 
>>>>>>>>> Just to follow up on this thread myself.  I modified
>>>>>>>>>
>>>>>>>>>                   
>>>> the pattern to return a "thick" future rather than a proxy for the
>>>> future.  This caused the RMI call to wait on the server until the
>>>> future was done and then sent back the outcome.  This
>>>>         
>> "fixed" the DGC
>>     
>>>> memory/thread leak by reducing the number of exported proxies
>>>> drammatically.
>>>>
>>>>         
>>>>>>>>> In terms of best practices, is distributed DGC simply
>>>>>>>>>
>>>>>>>>>                   
>>>> not useful for exported objects with short life spans?
>>>>         
>> Can it only
>>     
>>>> be used with proxies for relatively long lived services?
>>>>
>>>>         
>>>>>>>>> Thanks,
>>>>>>>>> Bryan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Bryan Thompson
>>>>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
>>>>>>>>>> To: user@river.apache.org
>>>>>>>>>> Subject: DGC threads issue
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> Background:
>>>>>>>>>>
>>>>>>>>>> I am seeing what would appear to be one DGC thread
>>>>>>>>>>
>>>>>>>>>>                     
>>>> allocated per
>>>>
>>>>         
>>>>>>>>>> exported object.  This is using River 2.2 and Sun JDK
>>>>>>>>>>
>>>>>>>>>>                     
>>>> 1.6.0_17.
>>>>
>>>>         
>>>>>>>>>> Relevant configuration parameters are below.
>>>>>>>>>>
>>>>>>>>>> I am observing problems with the DGC threads not being
>>>>>>>>>>
>>>>>>>>>>                     
>>>> retired on
>>>>
>>>>         
>>>>>>>>>> a timely basis.  The exported objects are proxies
>>>>>>>>>>                     
>> for Futures
>>     
>>>>>>>>>> which are being executed on the service.  The code
>>>>>>>>>>                     
>> pattern is
>>     
>>>>>>>>>> such that the proxied Future goes out of lexical scope quite
>>>>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
>>>>>>>>>>
>>>>>>>>>> Under a modest load, a large number of such Futures
>>>>>>>>>>
>>>>>>>>>>                     
>>>> are exported
>>>>
>>>>         
>>>>>>>>>> which results in a large number of long lived DGC
>>>>>>>>>>
>>>>>>>>>>                     
>>>> threads.  This
>>>>
>>>>         
>>>>>>>>>> turns into a problem for the JVM due to the stack
>>>>>>>>>>
>>>>>>>>>>                     
>>>> allocation per
>>>>
>>>>         
>>>>>>>>>> thread.  Presumably this is not good for other
>>>>>>>>>>                     
>> reasons as well
>>     
>>>>>>>>>> (e.g., scheduling).
>>>>>>>>>>
>>>>>>>>>> I have tried to override the leaseValue and checkInterval
>>>>>>>>>> defaults per the configuration options below.  I
>>>>>>>>>>
>>>>>>>>>>                     
>>>> suspect that the
>>>>
>>>>         
>>>>>>>>>> lease interval is somehow not being obeyed, which is
>>>>>>>>>>
>>>>>>>>>>                     
>>>> presumably a
>>>>
>>>>         
>>>>>>>>>> problem on my end.  However, I can verify that the
>>>>>>>>>>
>>>>>>>>>>                     
>>>> configuration
>>>>
>>>>         
>>>>>>>>>> values are in fact showing up in
>>>>>>>>>> System.getProperties() for at least some of the JVMs
>>>>>>>>>>                     
>> involved
>>     
>>>>>>>>>> (the one which drives the workload and the one that I am
>>>>>>>>>> monitoring with the large number of DGC lease threads).
>>>>>>>>>>
>>>>>>>>>> Some questions:
>>>>>>>>>>
>>>>>>>>>> Is this one-thread-per-exported proxy the expected
>>>>>>>>>>
>>>>>>>>>>                     
>>>> behavior when
>>>>
>>>>         
>>>>>>>>>> DGC is requested for the exported object?
>>>>>>>>>>
>>>>>>>>>> The DGC lease checker threads appear to expire ~14 -
>>>>>>>>>>
>>>>>>>>>>                     
>>>> 15 minutes
>>>>
>>>>         
>>>>>>>>>> after I terminate the process which was originating the RMI
>>>>>>>>>> requests.  This is close the sum of the default
>>>>>>>>>>
>>>>>>>>>>                     
>>>> leaseValue (10m)
>>>>
>>>>         
>>>>>>>>>> and checkInterval (5m) parameters, but maybe there is
>>>>>>>>>>
>>>>>>>>>>                     
>>>> some other
>>>>
>>>>         
>>>>>>>>>> timeout which is controlling this?  If this is the sum
>>>>>>>>>>
>>>>>>>>>>                     
>>>> of those
>>>>
>>>>         
>>>>>>>>>> parameters, why would the DGC lease threads live until
>>>>>>>>>>
>>>>>>>>>>                     
>>>> the sum of
>>>>
>>>>         
>>>>>>>>>> those values?  I thought that the lease would expire
>>>>>>>>>>                     
>> after the
>>     
>>>>>>>>>> leaseValue (10m default).
>>>>>>>>>>
>>>>>>>>>> Can the issue I am observing be caused by a low heap
>>>>>>>>>>
>>>>>>>>>>                     
>>>> pressure on
>>>>
>>>>         
>>>>>>>>>> the JVM to which the RMI proxies were exported?  If it
>>>>>>>>>>
>>>>>>>>>>                     
>>>> fails to
>>>>
>>>>         
>>>>>>>>>> GC those proxies, even though they are reachable, could that
>>>>>>>>>> cause DGC to continue to retain those proxies on the
>>>>>>>>>>                     
>> JVM which
>>     
>>>>>>>>>> exported them?
>>>>>>>>>>
>>>>>>>>>> Is there any way to configure DGC to use a thread pool
>>>>>>>>>>
>>>>>>>>>>                     
>>>> or to have
>>>>
>>>>         
>>>>>>>>>> the leases managed by a single thread?
>>>>>>>>>>
>>>>>>>>>> Is it possible that there is an interaction with the
>>>>>>>>>>
>>>>>>>>>>                     
>>>> useNIO option?
>>>>
>>>>         
>>>>>>>>>> Relevant options that I am using include:
>>>>>>>>>>
>>>>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
>>>>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
>>>>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
>>>>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
>>>>>>>>>>
>>>>>>>>>> Thanks in advance,
>>>>>>>>>> Bryan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>
>>>>>>>>                 
>>>>>>>               
>>>>>           
>>     
>
>   

RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

What about the following property?  Is it still valid?

        -Dsun.rmi.transport.tcp.connectionPool=true

Thanks,
Bryan

> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au]
> Sent: Friday, January 13, 2012 8:41 AM
> To: Bryan Thompson
> Cc: dev@river.apache.org; user@river.apache.org
> Subject: Re: [Fwd: Re: DGC threads issue]
>
> Brian,
>
> Do you have a Linux / Unix build environment?
>
> River isn't building yet on Window's, FreeBSD, or OpenBSD,
> known build environments that work are Ubuntu and Solaris.
> If not, that's ok, well be able to get the necessary jar's off Hudson.
>
> The DGC lease property values you need to set are:
>
> com.sun.jini.jeri.dgc.leaseValue=30000
> com.sun.jini.jeri.dgc.checkInterval=15000
>
> RMI (called JRMP) is superseded code now, River doesn't use
> that unless you configure your Exporter to be JrmpExporter,
> but you're better off sticking with BasicJeriExporter.
>
> Regards,
>
> Peter.
>
> Bryan Thompson wrote:
> > Peter,
> >
> > I would be happy to test out a fix on this.  Can you point
> out the modified line and link me to any directions on how to
> rebuild the necessary jar(s)?
> >
> > Did you already file an issue for this?  If not, I will
> file an issue and then update it when I test out this fix.
> >
> > Thanks,
> > Bryan
> >
> >
> >> -----Original Message-----
> >> From: Peter Firmstone [mailto:jini@zeus.net.au]
> >> Sent: Friday, January 13, 2012 3:21 AM
> >> To: dev@river.apache.org
> >> Cc: Bryan Thompson; user@river.apache.org
> >> Subject: Re: [Fwd: Re: DGC threads issue]
> >>
> >> Ok, found the problem, when I fixed River-142, I introduced a new
> >> bug, after creating a new DGC thread, I didn't set the
> running flag
> >> to true, so each time, a new thread is created.
> >>
> >> Thankfully it's a very easy fix.
> >>
> >> Cheers,
> >>
> >> Peter.
> >>
> >> Peter Jones wrote:
> >>
> >>> Peter,
> >>>
> >>> That internal Executor interface permits neither
> >>>
> >> synchronous execution in the calling thread ("this method
> itself must
> >> not block") nor any small bound on the number of Runnables
> that can
> >> be executed concurrently ("execution of a given action must not be
> >> delayed indefinitely in order to complete execution of a different
> >> action passed to a different invocation of this method").
> In other
> >> words, concurrency-wise, it should be equivalent to "new
> >> Thread(runnable, name)", or what the result of
> >> j.u.c.Executors.newCachedThreadPool() would do.
> Otherwise, callers
> >> would not function correctly, because they expect to be
> able to pass
> >> Runnables that execute indefinitely, not just short-lived
> work tasks.
> >>
> >>> The real purpose of this internal Executor/ThreadPool was
> >>>
> >> to provide an internal alternative to "new Thread(...)" that takes
> >> care of certain things in common, like:
> >>
> >>> - thread reuse instead of creation when reasonable (idle timeout)
> >>> - but still with threads (re)named for the task being executed
> >>> - insulation of any thread creation permission requirement from
> >>> context calling execute (NewThreadAction)
> >>> - but instead requiring direct user of thread pool to be
> >>>
> >> trusted not
> >>
> >>> to abuse that insulation, with permission requirement for access
> >>> (ThreadPoolPermission and GetThreadPoolAction)
> >>> - common logging of uncaught exceptions
> >>>
> >>> -- Peter
> >>>
> >>>
> >>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> >>>
> >>>
> >>>
> >>>> ThreadPool implements the com.sun.jini.thread.Executor interface.
> >>>>
> >>>> Because the interface states it should not block, I think
> >>>>
> >> the calling thread should execute the task when the thread pool
> >> becomes saturated, rather than continue to create new
> threads as per
> >> the current implementation.  This will ensure that the task is
> >> completed, the calling thread, if it uses a sequence of tasks with
> >> dependencies will have to ensure that it submits the tasks
> in order.
> >> Since only the caller knows the order, it makes sense for
> this to be
> >> the callers responsibility.  As a result the executor will
> honor the
> >> non blocking contract.  For that reason we'll use a zero length
> >> queue, probably a SynchronousQueue.
> >>
> >>>> Regards,
> >>>>
> >>>> Peter.
> >>>>
> >>>> /**
> >>>> * Executor is an abstraction for a thread factory or
> >>>>
> >> thread pool for
> >>
> >>>> * executing actions asynchronously.
> >>>> *
> >>>> * @author    Sun Microsystems, Inc.
> >>>> *
> >>>> */
> >>>> public interface Executor {
> >>>>
> >>>>   /**
> >>>>    * Executes the given Runnable action asynchronously in
> >>>>
> >> some thread.
> >>
> >>>>    *
> >>>>    * The implementation may create a new thread to execute
> >>>>
> >> the action,
> >>
> >>>>    * or it may execute the action in an existing thread.
> >>>>    *
> >>>>    * The execution of a given action must not be delayed
> >>>>
> >> indefinitely
> >>
> >>>>    * in order to complete execution of a different action
> >>>>
> >> passed to a
> >>
> >>>>    * different invocation of this method.  In other words, the
> >>>>    * implementation must assume that there may be
> >>>>
> >> arbitrary dependencies
> >>
> >>>>    * between actions passed to this method, so it needs to
> >>>>
> >> be careful
> >>
> >>>>    * to avoid potential deadlock by delaying execution of
> >>>>
> >> one action
> >>
> >>>>    * indefinitely until another completes.
> >>>>    *
> >>>>    * Also, this method itself must not block, because it
> >>>>
> >> may be invoked
> >>
> >>>>    * by code that is serially processing data to produce
> >>>>
> >> multiple such
> >>
> >>>>    * arbitrarily-dependent actions that need to be executed.
> >>>>    *
> >>>>    * @param    runnable the Runnable action to execute
> >>>>    *
> >>>>    * @param    name string to include in the name of the
> >>>>
> >> thread used
> >>
> >>>>    * to execute the action
> >>>>    */
> >>>>   void execute(Runnable runnable, String name); }
> >>>>
> >>>> Peter Firmstone wrote:
> >>>>
> >>>>
> >>>>> Thanks Brian,
> >>>>>
> >>>>> Looking at our implementation code, DGC uses an Executor
> >>>>>
> >> called ThreadPool, it's javadoc states:
> >>
> >>>>> /**
> >>>>> * ThreadPool is a simple thread pool implementation of
> >>>>>
> >> the Executor
> >>
> >>>>> * interface.
> >>>>> *
> >>>>> * A new task is always given to an idle thread, if one is
> >>>>>
> >> available;
> >>
> >>>>> * otherwise, a new thread is always created.  There is
> no minimum
> >>>>> * warm thread count, nor is there a maximum thread count
> >>>>>
> >> (tasks are
> >>
> >>>>> * never queued unless there are sufficient idle threads
> to execute
> >>>>> * them).
> >>>>> *
> >>>>> * New threads are created as daemon threads in the thread
> >>>>>
> >> group that
> >>
> >>>>> * was passed to the ThreadPool instance's constructor.  Each
> >>>>> thread's
> >>>>> * name is the prefix NewThreadAction.NAME_PREFIX
> followed by the
> >>>>> name
> >>>>> * of the task it is currently executing, or "Idle" if it is
> >>>>> currently
> >>>>> * idle.
> >>>>>
> >>>>> ThreadPool predates Java 5, it looks like we can fix this
> >>>>>
> >> by using an Executor from Java 5, we can look at limiting
> the number
> >> of threads created based on available CPU's and a scaling
> factor and
> >> place the tasks in a BlockingQueue, so if the queue is filled, it
> >> blocks.
> >>
> >>>>> Can you report the issue as a Bug on Jira for me, I'll
> >>>>>
> >> fix this before the next release.
> >>
> >>>>> Regards,
> >>>>>
> >>>>> Peter.
> >>>>>
> >>>>> Peter Firmstone wrote:
> >>>>>
> >>>>>
> >>>>>> Hi Peter,
> >>>>>>
> >>>>>> I was wondering if you had any thoughts on this post
> >>>>>>
> >> from Bryan on River users?
> >>
> >>>>>> Hope you don't mind me asking ;)
> >>>>>>
> >>>>>> Best Regards,
> >>>>>>
> >>>>>> Peter Firmstone.
> >>>>>>
> >>>>>>
> >>>>>>
> >> -------------------------------------------------------------------
> >>
> >>>>>> -----
> >>>>>>
> >>>>>> Subject:
> >>>>>> Re: DGC threads issue
> >>>>>> From:
> >>>>>> Tom Hobbs <tv...@googlemail.com>
> >>>>>> Date:
> >>>>>> Thu, 12 Jan 2012 20:45:01 +0000
> >>>>>> To:
> >>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>
> >>>>>> To:
> >>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>
> >>>>>>
> >>>>>> Hi Bryan,
> >>>>>>
> >>>>>> Sorry that no one got back to you about this.  I'm
> afraid that I
> >>>>>> don't know the answer to your question, I've copied
> the dev list
> >>>>>> into this email in case someone who monitors that list
> (but not
> >>>>>> this one) has any ideas.
> >>>>>>
> >>>>>> Best regards,
> >>>>>>
> >>>>>> Tom
> >>>>>>
> >>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> >>>>>>
> >> <br...@systap.com> wrote:
> >>
> >>>>>>
> >>>>>>
> >>>>>>> Just to follow up on this thread myself.  I modified
> >>>>>>>
> >> the pattern to return a "thick" future rather than a proxy for the
> >> future.  This caused the RMI call to wait on the server until the
> >> future was done and then sent back the outcome.  This
> "fixed" the DGC
> >> memory/thread leak by reducing the number of exported proxies
> >> drammatically.
> >>
> >>>>>>> In terms of best practices, is distributed DGC simply
> >>>>>>>
> >> not useful for exported objects with short life spans?
> Can it only
> >> be used with proxies for relatively long lived services?
> >>
> >>>>>>> Thanks,
> >>>>>>> Bryan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Bryan Thompson
> >>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>>>>>>> To: user@river.apache.org
> >>>>>>>> Subject: DGC threads issue
> >>>>>>>>
> >>>>>>>> Hello,
> >>>>>>>>
> >>>>>>>> Background:
> >>>>>>>>
> >>>>>>>> I am seeing what would appear to be one DGC thread
> >>>>>>>>
> >> allocated per
> >>
> >>>>>>>> exported object.  This is using River 2.2 and Sun JDK
> >>>>>>>>
> >> 1.6.0_17.
> >>
> >>>>>>>> Relevant configuration parameters are below.
> >>>>>>>>
> >>>>>>>> I am observing problems with the DGC threads not being
> >>>>>>>>
> >> retired on
> >>
> >>>>>>>> a timely basis.  The exported objects are proxies
> for Futures
> >>>>>>>> which are being executed on the service.  The code
> pattern is
> >>>>>>>> such that the proxied Future goes out of lexical scope quite
> >>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> >>>>>>>>
> >>>>>>>> Under a modest load, a large number of such Futures
> >>>>>>>>
> >> are exported
> >>
> >>>>>>>> which results in a large number of long lived DGC
> >>>>>>>>
> >> threads.  This
> >>
> >>>>>>>> turns into a problem for the JVM due to the stack
> >>>>>>>>
> >> allocation per
> >>
> >>>>>>>> thread.  Presumably this is not good for other
> reasons as well
> >>>>>>>> (e.g., scheduling).
> >>>>>>>>
> >>>>>>>> I have tried to override the leaseValue and checkInterval
> >>>>>>>> defaults per the configuration options below.  I
> >>>>>>>>
> >> suspect that the
> >>
> >>>>>>>> lease interval is somehow not being obeyed, which is
> >>>>>>>>
> >> presumably a
> >>
> >>>>>>>> problem on my end.  However, I can verify that the
> >>>>>>>>
> >> configuration
> >>
> >>>>>>>> values are in fact showing up in
> >>>>>>>> System.getProperties() for at least some of the JVMs
> involved
> >>>>>>>> (the one which drives the workload and the one that I am
> >>>>>>>> monitoring with the large number of DGC lease threads).
> >>>>>>>>
> >>>>>>>> Some questions:
> >>>>>>>>
> >>>>>>>> Is this one-thread-per-exported proxy the expected
> >>>>>>>>
> >> behavior when
> >>
> >>>>>>>> DGC is requested for the exported object?
> >>>>>>>>
> >>>>>>>> The DGC lease checker threads appear to expire ~14 -
> >>>>>>>>
> >> 15 minutes
> >>
> >>>>>>>> after I terminate the process which was originating the RMI
> >>>>>>>> requests.  This is close the sum of the default
> >>>>>>>>
> >> leaseValue (10m)
> >>
> >>>>>>>> and checkInterval (5m) parameters, but maybe there is
> >>>>>>>>
> >> some other
> >>
> >>>>>>>> timeout which is controlling this?  If this is the sum
> >>>>>>>>
> >> of those
> >>
> >>>>>>>> parameters, why would the DGC lease threads live until
> >>>>>>>>
> >> the sum of
> >>
> >>>>>>>> those values?  I thought that the lease would expire
> after the
> >>>>>>>> leaseValue (10m default).
> >>>>>>>>
> >>>>>>>> Can the issue I am observing be caused by a low heap
> >>>>>>>>
> >> pressure on
> >>
> >>>>>>>> the JVM to which the RMI proxies were exported?  If it
> >>>>>>>>
> >> fails to
> >>
> >>>>>>>> GC those proxies, even though they are reachable, could that
> >>>>>>>> cause DGC to continue to retain those proxies on the
> JVM which
> >>>>>>>> exported them?
> >>>>>>>>
> >>>>>>>> Is there any way to configure DGC to use a thread pool
> >>>>>>>>
> >> or to have
> >>
> >>>>>>>> the leases managed by a single thread?
> >>>>>>>>
> >>>>>>>> Is it possible that there is an interaction with the
> >>>>>>>>
> >> useNIO option?
> >>
> >>>>>>>> Relevant options that I am using include:
> >>>>>>>>
> >>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
> >>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> >>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> >>>>>>>>
> >>>>>>>> Thanks in advance,
> >>>>>>>> Bryan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

What about the following property?  Is it still valid?

        -Dsun.rmi.transport.tcp.connectionPool=true

Thanks,
Bryan

> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au]
> Sent: Friday, January 13, 2012 8:41 AM
> To: Bryan Thompson
> Cc: dev@river.apache.org; user@river.apache.org
> Subject: Re: [Fwd: Re: DGC threads issue]
>
> Brian,
>
> Do you have a Linux / Unix build environment?
>
> River isn't building yet on Window's, FreeBSD, or OpenBSD,
> known build environments that work are Ubuntu and Solaris.
> If not, that's ok, well be able to get the necessary jar's off Hudson.
>
> The DGC lease property values you need to set are:
>
> com.sun.jini.jeri.dgc.leaseValue=30000
> com.sun.jini.jeri.dgc.checkInterval=15000
>
> RMI (called JRMP) is superseded code now, River doesn't use
> that unless you configure your Exporter to be JrmpExporter,
> but you're better off sticking with BasicJeriExporter.
>
> Regards,
>
> Peter.
>
> Bryan Thompson wrote:
> > Peter,
> >
> > I would be happy to test out a fix on this.  Can you point
> out the modified line and link me to any directions on how to
> rebuild the necessary jar(s)?
> >
> > Did you already file an issue for this?  If not, I will
> file an issue and then update it when I test out this fix.
> >
> > Thanks,
> > Bryan
> >
> >
> >> -----Original Message-----
> >> From: Peter Firmstone [mailto:jini@zeus.net.au]
> >> Sent: Friday, January 13, 2012 3:21 AM
> >> To: dev@river.apache.org
> >> Cc: Bryan Thompson; user@river.apache.org
> >> Subject: Re: [Fwd: Re: DGC threads issue]
> >>
> >> Ok, found the problem, when I fixed River-142, I introduced a new
> >> bug, after creating a new DGC thread, I didn't set the
> running flag
> >> to true, so each time, a new thread is created.
> >>
> >> Thankfully it's a very easy fix.
> >>
> >> Cheers,
> >>
> >> Peter.
> >>
> >> Peter Jones wrote:
> >>
> >>> Peter,
> >>>
> >>> That internal Executor interface permits neither
> >>>
> >> synchronous execution in the calling thread ("this method
> itself must
> >> not block") nor any small bound on the number of Runnables
> that can
> >> be executed concurrently ("execution of a given action must not be
> >> delayed indefinitely in order to complete execution of a different
> >> action passed to a different invocation of this method").
> In other
> >> words, concurrency-wise, it should be equivalent to "new
> >> Thread(runnable, name)", or what the result of
> >> j.u.c.Executors.newCachedThreadPool() would do.
> Otherwise, callers
> >> would not function correctly, because they expect to be
> able to pass
> >> Runnables that execute indefinitely, not just short-lived
> work tasks.
> >>
> >>> The real purpose of this internal Executor/ThreadPool was
> >>>
> >> to provide an internal alternative to "new Thread(...)" that takes
> >> care of certain things in common, like:
> >>
> >>> - thread reuse instead of creation when reasonable (idle timeout)
> >>> - but still with threads (re)named for the task being executed
> >>> - insulation of any thread creation permission requirement from
> >>> context calling execute (NewThreadAction)
> >>> - but instead requiring direct user of thread pool to be
> >>>
> >> trusted not
> >>
> >>> to abuse that insulation, with permission requirement for access
> >>> (ThreadPoolPermission and GetThreadPoolAction)
> >>> - common logging of uncaught exceptions
> >>>
> >>> -- Peter
> >>>
> >>>
> >>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> >>>
> >>>
> >>>
> >>>> ThreadPool implements the com.sun.jini.thread.Executor interface.
> >>>>
> >>>> Because the interface states it should not block, I think
> >>>>
> >> the calling thread should execute the task when the thread pool
> >> becomes saturated, rather than continue to create new
> threads as per
> >> the current implementation.  This will ensure that the task is
> >> completed, the calling thread, if it uses a sequence of tasks with
> >> dependencies will have to ensure that it submits the tasks
> in order.
> >> Since only the caller knows the order, it makes sense for
> this to be
> >> the callers responsibility.  As a result the executor will
> honor the
> >> non blocking contract.  For that reason we'll use a zero length
> >> queue, probably a SynchronousQueue.
> >>
> >>>> Regards,
> >>>>
> >>>> Peter.
> >>>>
> >>>> /**
> >>>> * Executor is an abstraction for a thread factory or
> >>>>
> >> thread pool for
> >>
> >>>> * executing actions asynchronously.
> >>>> *
> >>>> * @author    Sun Microsystems, Inc.
> >>>> *
> >>>> */
> >>>> public interface Executor {
> >>>>
> >>>>   /**
> >>>>    * Executes the given Runnable action asynchronously in
> >>>>
> >> some thread.
> >>
> >>>>    *
> >>>>    * The implementation may create a new thread to execute
> >>>>
> >> the action,
> >>
> >>>>    * or it may execute the action in an existing thread.
> >>>>    *
> >>>>    * The execution of a given action must not be delayed
> >>>>
> >> indefinitely
> >>
> >>>>    * in order to complete execution of a different action
> >>>>
> >> passed to a
> >>
> >>>>    * different invocation of this method.  In other words, the
> >>>>    * implementation must assume that there may be
> >>>>
> >> arbitrary dependencies
> >>
> >>>>    * between actions passed to this method, so it needs to
> >>>>
> >> be careful
> >>
> >>>>    * to avoid potential deadlock by delaying execution of
> >>>>
> >> one action
> >>
> >>>>    * indefinitely until another completes.
> >>>>    *
> >>>>    * Also, this method itself must not block, because it
> >>>>
> >> may be invoked
> >>
> >>>>    * by code that is serially processing data to produce
> >>>>
> >> multiple such
> >>
> >>>>    * arbitrarily-dependent actions that need to be executed.
> >>>>    *
> >>>>    * @param    runnable the Runnable action to execute
> >>>>    *
> >>>>    * @param    name string to include in the name of the
> >>>>
> >> thread used
> >>
> >>>>    * to execute the action
> >>>>    */
> >>>>   void execute(Runnable runnable, String name); }
> >>>>
> >>>> Peter Firmstone wrote:
> >>>>
> >>>>
> >>>>> Thanks Brian,
> >>>>>
> >>>>> Looking at our implementation code, DGC uses an Executor
> >>>>>
> >> called ThreadPool, it's javadoc states:
> >>
> >>>>> /**
> >>>>> * ThreadPool is a simple thread pool implementation of
> >>>>>
> >> the Executor
> >>
> >>>>> * interface.
> >>>>> *
> >>>>> * A new task is always given to an idle thread, if one is
> >>>>>
> >> available;
> >>
> >>>>> * otherwise, a new thread is always created.  There is
> no minimum
> >>>>> * warm thread count, nor is there a maximum thread count
> >>>>>
> >> (tasks are
> >>
> >>>>> * never queued unless there are sufficient idle threads
> to execute
> >>>>> * them).
> >>>>> *
> >>>>> * New threads are created as daemon threads in the thread
> >>>>>
> >> group that
> >>
> >>>>> * was passed to the ThreadPool instance's constructor.  Each
> >>>>> thread's
> >>>>> * name is the prefix NewThreadAction.NAME_PREFIX
> followed by the
> >>>>> name
> >>>>> * of the task it is currently executing, or "Idle" if it is
> >>>>> currently
> >>>>> * idle.
> >>>>>
> >>>>> ThreadPool predates Java 5, it looks like we can fix this
> >>>>>
> >> by using an Executor from Java 5, we can look at limiting
> the number
> >> of threads created based on available CPU's and a scaling
> factor and
> >> place the tasks in a BlockingQueue, so if the queue is filled, it
> >> blocks.
> >>
> >>>>> Can you report the issue as a Bug on Jira for me, I'll
> >>>>>
> >> fix this before the next release.
> >>
> >>>>> Regards,
> >>>>>
> >>>>> Peter.
> >>>>>
> >>>>> Peter Firmstone wrote:
> >>>>>
> >>>>>
> >>>>>> Hi Peter,
> >>>>>>
> >>>>>> I was wondering if you had any thoughts on this post
> >>>>>>
> >> from Bryan on River users?
> >>
> >>>>>> Hope you don't mind me asking ;)
> >>>>>>
> >>>>>> Best Regards,
> >>>>>>
> >>>>>> Peter Firmstone.
> >>>>>>
> >>>>>>
> >>>>>>
> >> -------------------------------------------------------------------
> >>
> >>>>>> -----
> >>>>>>
> >>>>>> Subject:
> >>>>>> Re: DGC threads issue
> >>>>>> From:
> >>>>>> Tom Hobbs <tv...@googlemail.com>
> >>>>>> Date:
> >>>>>> Thu, 12 Jan 2012 20:45:01 +0000
> >>>>>> To:
> >>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>
> >>>>>> To:
> >>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>
> >>>>>>
> >>>>>> Hi Bryan,
> >>>>>>
> >>>>>> Sorry that no one got back to you about this.  I'm
> afraid that I
> >>>>>> don't know the answer to your question, I've copied
> the dev list
> >>>>>> into this email in case someone who monitors that list
> (but not
> >>>>>> this one) has any ideas.
> >>>>>>
> >>>>>> Best regards,
> >>>>>>
> >>>>>> Tom
> >>>>>>
> >>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> >>>>>>
> >> <br...@systap.com> wrote:
> >>
> >>>>>>
> >>>>>>
> >>>>>>> Just to follow up on this thread myself.  I modified
> >>>>>>>
> >> the pattern to return a "thick" future rather than a proxy for the
> >> future.  This caused the RMI call to wait on the server until the
> >> future was done and then sent back the outcome.  This
> "fixed" the DGC
> >> memory/thread leak by reducing the number of exported proxies
> >> drammatically.
> >>
> >>>>>>> In terms of best practices, is distributed DGC simply
> >>>>>>>
> >> not useful for exported objects with short life spans?
> Can it only
> >> be used with proxies for relatively long lived services?
> >>
> >>>>>>> Thanks,
> >>>>>>> Bryan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Bryan Thompson
> >>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>>>>>>> To: user@river.apache.org
> >>>>>>>> Subject: DGC threads issue
> >>>>>>>>
> >>>>>>>> Hello,
> >>>>>>>>
> >>>>>>>> Background:
> >>>>>>>>
> >>>>>>>> I am seeing what would appear to be one DGC thread
> >>>>>>>>
> >> allocated per
> >>
> >>>>>>>> exported object.  This is using River 2.2 and Sun JDK
> >>>>>>>>
> >> 1.6.0_17.
> >>
> >>>>>>>> Relevant configuration parameters are below.
> >>>>>>>>
> >>>>>>>> I am observing problems with the DGC threads not being
> >>>>>>>>
> >> retired on
> >>
> >>>>>>>> a timely basis.  The exported objects are proxies
> for Futures
> >>>>>>>> which are being executed on the service.  The code
> pattern is
> >>>>>>>> such that the proxied Future goes out of lexical scope quite
> >>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> >>>>>>>>
> >>>>>>>> Under a modest load, a large number of such Futures
> >>>>>>>>
> >> are exported
> >>
> >>>>>>>> which results in a large number of long lived DGC
> >>>>>>>>
> >> threads.  This
> >>
> >>>>>>>> turns into a problem for the JVM due to the stack
> >>>>>>>>
> >> allocation per
> >>
> >>>>>>>> thread.  Presumably this is not good for other
> reasons as well
> >>>>>>>> (e.g., scheduling).
> >>>>>>>>
> >>>>>>>> I have tried to override the leaseValue and checkInterval
> >>>>>>>> defaults per the configuration options below.  I
> >>>>>>>>
> >> suspect that the
> >>
> >>>>>>>> lease interval is somehow not being obeyed, which is
> >>>>>>>>
> >> presumably a
> >>
> >>>>>>>> problem on my end.  However, I can verify that the
> >>>>>>>>
> >> configuration
> >>
> >>>>>>>> values are in fact showing up in
> >>>>>>>> System.getProperties() for at least some of the JVMs
> involved
> >>>>>>>> (the one which drives the workload and the one that I am
> >>>>>>>> monitoring with the large number of DGC lease threads).
> >>>>>>>>
> >>>>>>>> Some questions:
> >>>>>>>>
> >>>>>>>> Is this one-thread-per-exported proxy the expected
> >>>>>>>>
> >> behavior when
> >>
> >>>>>>>> DGC is requested for the exported object?
> >>>>>>>>
> >>>>>>>> The DGC lease checker threads appear to expire ~14 -
> >>>>>>>>
> >> 15 minutes
> >>
> >>>>>>>> after I terminate the process which was originating the RMI
> >>>>>>>> requests.  This is close the sum of the default
> >>>>>>>>
> >> leaseValue (10m)
> >>
> >>>>>>>> and checkInterval (5m) parameters, but maybe there is
> >>>>>>>>
> >> some other
> >>
> >>>>>>>> timeout which is controlling this?  If this is the sum
> >>>>>>>>
> >> of those
> >>
> >>>>>>>> parameters, why would the DGC lease threads live until
> >>>>>>>>
> >> the sum of
> >>
> >>>>>>>> those values?  I thought that the lease would expire
> after the
> >>>>>>>> leaseValue (10m default).
> >>>>>>>>
> >>>>>>>> Can the issue I am observing be caused by a low heap
> >>>>>>>>
> >> pressure on
> >>
> >>>>>>>> the JVM to which the RMI proxies were exported?  If it
> >>>>>>>>
> >> fails to
> >>
> >>>>>>>> GC those proxies, even though they are reachable, could that
> >>>>>>>> cause DGC to continue to retain those proxies on the
> JVM which
> >>>>>>>> exported them?
> >>>>>>>>
> >>>>>>>> Is there any way to configure DGC to use a thread pool
> >>>>>>>>
> >> or to have
> >>
> >>>>>>>> the leases managed by a single thread?
> >>>>>>>>
> >>>>>>>> Is it possible that there is an interaction with the
> >>>>>>>>
> >> useNIO option?
> >>
> >>>>>>>> Relevant options that I am using include:
> >>>>>>>>
> >>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
> >>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> >>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> >>>>>>>>
> >>>>>>>> Thanks in advance,
> >>>>>>>> Bryan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

Yes, I can build under linux (Centos or Ubuntu).

That new property value names would explain why DGC lease timeouts were not being effected by the configuration.

Thanks,
Bryan

> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au]
> Sent: Friday, January 13, 2012 8:41 AM
> To: Bryan Thompson
> Cc: dev@river.apache.org; user@river.apache.org
> Subject: Re: [Fwd: Re: DGC threads issue]
>
> Brian,
>
> Do you have a Linux / Unix build environment?
>
> River isn't building yet on Window's, FreeBSD, or OpenBSD,
> known build environments that work are Ubuntu and Solaris.
> If not, that's ok, well be able to get the necessary jar's off Hudson.
>
> The DGC lease property values you need to set are:
>
> com.sun.jini.jeri.dgc.leaseValue=30000
> com.sun.jini.jeri.dgc.checkInterval=15000
>
> RMI (called JRMP) is superseded code now, River doesn't use
> that unless you configure your Exporter to be JrmpExporter,
> but you're better off sticking with BasicJeriExporter.
>
> Regards,
>
> Peter.
>
> Bryan Thompson wrote:
> > Peter,
> >
> > I would be happy to test out a fix on this.  Can you point
> out the modified line and link me to any directions on how to
> rebuild the necessary jar(s)?
> >
> > Did you already file an issue for this?  If not, I will
> file an issue and then update it when I test out this fix.
> >
> > Thanks,
> > Bryan
> >
> >
> >> -----Original Message-----
> >> From: Peter Firmstone [mailto:jini@zeus.net.au]
> >> Sent: Friday, January 13, 2012 3:21 AM
> >> To: dev@river.apache.org
> >> Cc: Bryan Thompson; user@river.apache.org
> >> Subject: Re: [Fwd: Re: DGC threads issue]
> >>
> >> Ok, found the problem, when I fixed River-142, I introduced a new
> >> bug, after creating a new DGC thread, I didn't set the
> running flag
> >> to true, so each time, a new thread is created.
> >>
> >> Thankfully it's a very easy fix.
> >>
> >> Cheers,
> >>
> >> Peter.
> >>
> >> Peter Jones wrote:
> >>
> >>> Peter,
> >>>
> >>> That internal Executor interface permits neither
> >>>
> >> synchronous execution in the calling thread ("this method
> itself must
> >> not block") nor any small bound on the number of Runnables
> that can
> >> be executed concurrently ("execution of a given action must not be
> >> delayed indefinitely in order to complete execution of a different
> >> action passed to a different invocation of this method").
> In other
> >> words, concurrency-wise, it should be equivalent to "new
> >> Thread(runnable, name)", or what the result of
> >> j.u.c.Executors.newCachedThreadPool() would do.
> Otherwise, callers
> >> would not function correctly, because they expect to be
> able to pass
> >> Runnables that execute indefinitely, not just short-lived
> work tasks.
> >>
> >>> The real purpose of this internal Executor/ThreadPool was
> >>>
> >> to provide an internal alternative to "new Thread(...)" that takes
> >> care of certain things in common, like:
> >>
> >>> - thread reuse instead of creation when reasonable (idle timeout)
> >>> - but still with threads (re)named for the task being executed
> >>> - insulation of any thread creation permission requirement from
> >>> context calling execute (NewThreadAction)
> >>> - but instead requiring direct user of thread pool to be
> >>>
> >> trusted not
> >>
> >>> to abuse that insulation, with permission requirement for access
> >>> (ThreadPoolPermission and GetThreadPoolAction)
> >>> - common logging of uncaught exceptions
> >>>
> >>> -- Peter
> >>>
> >>>
> >>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> >>>
> >>>
> >>>
> >>>> ThreadPool implements the com.sun.jini.thread.Executor interface.
> >>>>
> >>>> Because the interface states it should not block, I think
> >>>>
> >> the calling thread should execute the task when the thread pool
> >> becomes saturated, rather than continue to create new
> threads as per
> >> the current implementation.  This will ensure that the task is
> >> completed, the calling thread, if it uses a sequence of tasks with
> >> dependencies will have to ensure that it submits the tasks
> in order.
> >> Since only the caller knows the order, it makes sense for
> this to be
> >> the callers responsibility.  As a result the executor will
> honor the
> >> non blocking contract.  For that reason we'll use a zero length
> >> queue, probably a SynchronousQueue.
> >>
> >>>> Regards,
> >>>>
> >>>> Peter.
> >>>>
> >>>> /**
> >>>> * Executor is an abstraction for a thread factory or
> >>>>
> >> thread pool for
> >>
> >>>> * executing actions asynchronously.
> >>>> *
> >>>> * @author    Sun Microsystems, Inc.
> >>>> *
> >>>> */
> >>>> public interface Executor {
> >>>>
> >>>>   /**
> >>>>    * Executes the given Runnable action asynchronously in
> >>>>
> >> some thread.
> >>
> >>>>    *
> >>>>    * The implementation may create a new thread to execute
> >>>>
> >> the action,
> >>
> >>>>    * or it may execute the action in an existing thread.
> >>>>    *
> >>>>    * The execution of a given action must not be delayed
> >>>>
> >> indefinitely
> >>
> >>>>    * in order to complete execution of a different action
> >>>>
> >> passed to a
> >>
> >>>>    * different invocation of this method.  In other words, the
> >>>>    * implementation must assume that there may be
> >>>>
> >> arbitrary dependencies
> >>
> >>>>    * between actions passed to this method, so it needs to
> >>>>
> >> be careful
> >>
> >>>>    * to avoid potential deadlock by delaying execution of
> >>>>
> >> one action
> >>
> >>>>    * indefinitely until another completes.
> >>>>    *
> >>>>    * Also, this method itself must not block, because it
> >>>>
> >> may be invoked
> >>
> >>>>    * by code that is serially processing data to produce
> >>>>
> >> multiple such
> >>
> >>>>    * arbitrarily-dependent actions that need to be executed.
> >>>>    *
> >>>>    * @param    runnable the Runnable action to execute
> >>>>    *
> >>>>    * @param    name string to include in the name of the
> >>>>
> >> thread used
> >>
> >>>>    * to execute the action
> >>>>    */
> >>>>   void execute(Runnable runnable, String name); }
> >>>>
> >>>> Peter Firmstone wrote:
> >>>>
> >>>>
> >>>>> Thanks Brian,
> >>>>>
> >>>>> Looking at our implementation code, DGC uses an Executor
> >>>>>
> >> called ThreadPool, it's javadoc states:
> >>
> >>>>> /**
> >>>>> * ThreadPool is a simple thread pool implementation of
> >>>>>
> >> the Executor
> >>
> >>>>> * interface.
> >>>>> *
> >>>>> * A new task is always given to an idle thread, if one is
> >>>>>
> >> available;
> >>
> >>>>> * otherwise, a new thread is always created.  There is
> no minimum
> >>>>> * warm thread count, nor is there a maximum thread count
> >>>>>
> >> (tasks are
> >>
> >>>>> * never queued unless there are sufficient idle threads
> to execute
> >>>>> * them).
> >>>>> *
> >>>>> * New threads are created as daemon threads in the thread
> >>>>>
> >> group that
> >>
> >>>>> * was passed to the ThreadPool instance's constructor.  Each
> >>>>> thread's
> >>>>> * name is the prefix NewThreadAction.NAME_PREFIX
> followed by the
> >>>>> name
> >>>>> * of the task it is currently executing, or "Idle" if it is
> >>>>> currently
> >>>>> * idle.
> >>>>>
> >>>>> ThreadPool predates Java 5, it looks like we can fix this
> >>>>>
> >> by using an Executor from Java 5, we can look at limiting
> the number
> >> of threads created based on available CPU's and a scaling
> factor and
> >> place the tasks in a BlockingQueue, so if the queue is filled, it
> >> blocks.
> >>
> >>>>> Can you report the issue as a Bug on Jira for me, I'll
> >>>>>
> >> fix this before the next release.
> >>
> >>>>> Regards,
> >>>>>
> >>>>> Peter.
> >>>>>
> >>>>> Peter Firmstone wrote:
> >>>>>
> >>>>>
> >>>>>> Hi Peter,
> >>>>>>
> >>>>>> I was wondering if you had any thoughts on this post
> >>>>>>
> >> from Bryan on River users?
> >>
> >>>>>> Hope you don't mind me asking ;)
> >>>>>>
> >>>>>> Best Regards,
> >>>>>>
> >>>>>> Peter Firmstone.
> >>>>>>
> >>>>>>
> >>>>>>
> >> -------------------------------------------------------------------
> >>
> >>>>>> -----
> >>>>>>
> >>>>>> Subject:
> >>>>>> Re: DGC threads issue
> >>>>>> From:
> >>>>>> Tom Hobbs <tv...@googlemail.com>
> >>>>>> Date:
> >>>>>> Thu, 12 Jan 2012 20:45:01 +0000
> >>>>>> To:
> >>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>
> >>>>>> To:
> >>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>
> >>>>>>
> >>>>>> Hi Bryan,
> >>>>>>
> >>>>>> Sorry that no one got back to you about this.  I'm
> afraid that I
> >>>>>> don't know the answer to your question, I've copied
> the dev list
> >>>>>> into this email in case someone who monitors that list
> (but not
> >>>>>> this one) has any ideas.
> >>>>>>
> >>>>>> Best regards,
> >>>>>>
> >>>>>> Tom
> >>>>>>
> >>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> >>>>>>
> >> <br...@systap.com> wrote:
> >>
> >>>>>>
> >>>>>>
> >>>>>>> Just to follow up on this thread myself.  I modified
> >>>>>>>
> >> the pattern to return a "thick" future rather than a proxy for the
> >> future.  This caused the RMI call to wait on the server until the
> >> future was done and then sent back the outcome.  This
> "fixed" the DGC
> >> memory/thread leak by reducing the number of exported proxies
> >> drammatically.
> >>
> >>>>>>> In terms of best practices, is distributed DGC simply
> >>>>>>>
> >> not useful for exported objects with short life spans?
> Can it only
> >> be used with proxies for relatively long lived services?
> >>
> >>>>>>> Thanks,
> >>>>>>> Bryan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Bryan Thompson
> >>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>>>>>>> To: user@river.apache.org
> >>>>>>>> Subject: DGC threads issue
> >>>>>>>>
> >>>>>>>> Hello,
> >>>>>>>>
> >>>>>>>> Background:
> >>>>>>>>
> >>>>>>>> I am seeing what would appear to be one DGC thread
> >>>>>>>>
> >> allocated per
> >>
> >>>>>>>> exported object.  This is using River 2.2 and Sun JDK
> >>>>>>>>
> >> 1.6.0_17.
> >>
> >>>>>>>> Relevant configuration parameters are below.
> >>>>>>>>
> >>>>>>>> I am observing problems with the DGC threads not being
> >>>>>>>>
> >> retired on
> >>
> >>>>>>>> a timely basis.  The exported objects are proxies
> for Futures
> >>>>>>>> which are being executed on the service.  The code
> pattern is
> >>>>>>>> such that the proxied Future goes out of lexical scope quite
> >>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> >>>>>>>>
> >>>>>>>> Under a modest load, a large number of such Futures
> >>>>>>>>
> >> are exported
> >>
> >>>>>>>> which results in a large number of long lived DGC
> >>>>>>>>
> >> threads.  This
> >>
> >>>>>>>> turns into a problem for the JVM due to the stack
> >>>>>>>>
> >> allocation per
> >>
> >>>>>>>> thread.  Presumably this is not good for other
> reasons as well
> >>>>>>>> (e.g., scheduling).
> >>>>>>>>
> >>>>>>>> I have tried to override the leaseValue and checkInterval
> >>>>>>>> defaults per the configuration options below.  I
> >>>>>>>>
> >> suspect that the
> >>
> >>>>>>>> lease interval is somehow not being obeyed, which is
> >>>>>>>>
> >> presumably a
> >>
> >>>>>>>> problem on my end.  However, I can verify that the
> >>>>>>>>
> >> configuration
> >>
> >>>>>>>> values are in fact showing up in
> >>>>>>>> System.getProperties() for at least some of the JVMs
> involved
> >>>>>>>> (the one which drives the workload and the one that I am
> >>>>>>>> monitoring with the large number of DGC lease threads).
> >>>>>>>>
> >>>>>>>> Some questions:
> >>>>>>>>
> >>>>>>>> Is this one-thread-per-exported proxy the expected
> >>>>>>>>
> >> behavior when
> >>
> >>>>>>>> DGC is requested for the exported object?
> >>>>>>>>
> >>>>>>>> The DGC lease checker threads appear to expire ~14 -
> >>>>>>>>
> >> 15 minutes
> >>
> >>>>>>>> after I terminate the process which was originating the RMI
> >>>>>>>> requests.  This is close the sum of the default
> >>>>>>>>
> >> leaseValue (10m)
> >>
> >>>>>>>> and checkInterval (5m) parameters, but maybe there is
> >>>>>>>>
> >> some other
> >>
> >>>>>>>> timeout which is controlling this?  If this is the sum
> >>>>>>>>
> >> of those
> >>
> >>>>>>>> parameters, why would the DGC lease threads live until
> >>>>>>>>
> >> the sum of
> >>
> >>>>>>>> those values?  I thought that the lease would expire
> after the
> >>>>>>>> leaseValue (10m default).
> >>>>>>>>
> >>>>>>>> Can the issue I am observing be caused by a low heap
> >>>>>>>>
> >> pressure on
> >>
> >>>>>>>> the JVM to which the RMI proxies were exported?  If it
> >>>>>>>>
> >> fails to
> >>
> >>>>>>>> GC those proxies, even though they are reachable, could that
> >>>>>>>> cause DGC to continue to retain those proxies on the
> JVM which
> >>>>>>>> exported them?
> >>>>>>>>
> >>>>>>>> Is there any way to configure DGC to use a thread pool
> >>>>>>>>
> >> or to have
> >>
> >>>>>>>> the leases managed by a single thread?
> >>>>>>>>
> >>>>>>>> Is it possible that there is an interaction with the
> >>>>>>>>
> >> useNIO option?
> >>
> >>>>>>>> Relevant options that I am using include:
> >>>>>>>>
> >>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
> >>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> >>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> >>>>>>>>
> >>>>>>>> Thanks in advance,
> >>>>>>>> Bryan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

Yes, I can build under linux (Centos or Ubuntu).

That new property value names would explain why DGC lease timeouts were not being effected by the configuration.

Thanks,
Bryan

> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au]
> Sent: Friday, January 13, 2012 8:41 AM
> To: Bryan Thompson
> Cc: dev@river.apache.org; user@river.apache.org
> Subject: Re: [Fwd: Re: DGC threads issue]
>
> Brian,
>
> Do you have a Linux / Unix build environment?
>
> River isn't building yet on Window's, FreeBSD, or OpenBSD,
> known build environments that work are Ubuntu and Solaris.
> If not, that's ok, well be able to get the necessary jar's off Hudson.
>
> The DGC lease property values you need to set are:
>
> com.sun.jini.jeri.dgc.leaseValue=30000
> com.sun.jini.jeri.dgc.checkInterval=15000
>
> RMI (called JRMP) is superseded code now, River doesn't use
> that unless you configure your Exporter to be JrmpExporter,
> but you're better off sticking with BasicJeriExporter.
>
> Regards,
>
> Peter.
>
> Bryan Thompson wrote:
> > Peter,
> >
> > I would be happy to test out a fix on this.  Can you point
> out the modified line and link me to any directions on how to
> rebuild the necessary jar(s)?
> >
> > Did you already file an issue for this?  If not, I will
> file an issue and then update it when I test out this fix.
> >
> > Thanks,
> > Bryan
> >
> >
> >> -----Original Message-----
> >> From: Peter Firmstone [mailto:jini@zeus.net.au]
> >> Sent: Friday, January 13, 2012 3:21 AM
> >> To: dev@river.apache.org
> >> Cc: Bryan Thompson; user@river.apache.org
> >> Subject: Re: [Fwd: Re: DGC threads issue]
> >>
> >> Ok, found the problem, when I fixed River-142, I introduced a new
> >> bug, after creating a new DGC thread, I didn't set the
> running flag
> >> to true, so each time, a new thread is created.
> >>
> >> Thankfully it's a very easy fix.
> >>
> >> Cheers,
> >>
> >> Peter.
> >>
> >> Peter Jones wrote:
> >>
> >>> Peter,
> >>>
> >>> That internal Executor interface permits neither
> >>>
> >> synchronous execution in the calling thread ("this method
> itself must
> >> not block") nor any small bound on the number of Runnables
> that can
> >> be executed concurrently ("execution of a given action must not be
> >> delayed indefinitely in order to complete execution of a different
> >> action passed to a different invocation of this method").
> In other
> >> words, concurrency-wise, it should be equivalent to "new
> >> Thread(runnable, name)", or what the result of
> >> j.u.c.Executors.newCachedThreadPool() would do.
> Otherwise, callers
> >> would not function correctly, because they expect to be
> able to pass
> >> Runnables that execute indefinitely, not just short-lived
> work tasks.
> >>
> >>> The real purpose of this internal Executor/ThreadPool was
> >>>
> >> to provide an internal alternative to "new Thread(...)" that takes
> >> care of certain things in common, like:
> >>
> >>> - thread reuse instead of creation when reasonable (idle timeout)
> >>> - but still with threads (re)named for the task being executed
> >>> - insulation of any thread creation permission requirement from
> >>> context calling execute (NewThreadAction)
> >>> - but instead requiring direct user of thread pool to be
> >>>
> >> trusted not
> >>
> >>> to abuse that insulation, with permission requirement for access
> >>> (ThreadPoolPermission and GetThreadPoolAction)
> >>> - common logging of uncaught exceptions
> >>>
> >>> -- Peter
> >>>
> >>>
> >>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> >>>
> >>>
> >>>
> >>>> ThreadPool implements the com.sun.jini.thread.Executor interface.
> >>>>
> >>>> Because the interface states it should not block, I think
> >>>>
> >> the calling thread should execute the task when the thread pool
> >> becomes saturated, rather than continue to create new
> threads as per
> >> the current implementation.  This will ensure that the task is
> >> completed, the calling thread, if it uses a sequence of tasks with
> >> dependencies will have to ensure that it submits the tasks
> in order.
> >> Since only the caller knows the order, it makes sense for
> this to be
> >> the callers responsibility.  As a result the executor will
> honor the
> >> non blocking contract.  For that reason we'll use a zero length
> >> queue, probably a SynchronousQueue.
> >>
> >>>> Regards,
> >>>>
> >>>> Peter.
> >>>>
> >>>> /**
> >>>> * Executor is an abstraction for a thread factory or
> >>>>
> >> thread pool for
> >>
> >>>> * executing actions asynchronously.
> >>>> *
> >>>> * @author    Sun Microsystems, Inc.
> >>>> *
> >>>> */
> >>>> public interface Executor {
> >>>>
> >>>>   /**
> >>>>    * Executes the given Runnable action asynchronously in
> >>>>
> >> some thread.
> >>
> >>>>    *
> >>>>    * The implementation may create a new thread to execute
> >>>>
> >> the action,
> >>
> >>>>    * or it may execute the action in an existing thread.
> >>>>    *
> >>>>    * The execution of a given action must not be delayed
> >>>>
> >> indefinitely
> >>
> >>>>    * in order to complete execution of a different action
> >>>>
> >> passed to a
> >>
> >>>>    * different invocation of this method.  In other words, the
> >>>>    * implementation must assume that there may be
> >>>>
> >> arbitrary dependencies
> >>
> >>>>    * between actions passed to this method, so it needs to
> >>>>
> >> be careful
> >>
> >>>>    * to avoid potential deadlock by delaying execution of
> >>>>
> >> one action
> >>
> >>>>    * indefinitely until another completes.
> >>>>    *
> >>>>    * Also, this method itself must not block, because it
> >>>>
> >> may be invoked
> >>
> >>>>    * by code that is serially processing data to produce
> >>>>
> >> multiple such
> >>
> >>>>    * arbitrarily-dependent actions that need to be executed.
> >>>>    *
> >>>>    * @param    runnable the Runnable action to execute
> >>>>    *
> >>>>    * @param    name string to include in the name of the
> >>>>
> >> thread used
> >>
> >>>>    * to execute the action
> >>>>    */
> >>>>   void execute(Runnable runnable, String name); }
> >>>>
> >>>> Peter Firmstone wrote:
> >>>>
> >>>>
> >>>>> Thanks Brian,
> >>>>>
> >>>>> Looking at our implementation code, DGC uses an Executor
> >>>>>
> >> called ThreadPool, it's javadoc states:
> >>
> >>>>> /**
> >>>>> * ThreadPool is a simple thread pool implementation of
> >>>>>
> >> the Executor
> >>
> >>>>> * interface.
> >>>>> *
> >>>>> * A new task is always given to an idle thread, if one is
> >>>>>
> >> available;
> >>
> >>>>> * otherwise, a new thread is always created.  There is
> no minimum
> >>>>> * warm thread count, nor is there a maximum thread count
> >>>>>
> >> (tasks are
> >>
> >>>>> * never queued unless there are sufficient idle threads
> to execute
> >>>>> * them).
> >>>>> *
> >>>>> * New threads are created as daemon threads in the thread
> >>>>>
> >> group that
> >>
> >>>>> * was passed to the ThreadPool instance's constructor.  Each
> >>>>> thread's
> >>>>> * name is the prefix NewThreadAction.NAME_PREFIX
> followed by the
> >>>>> name
> >>>>> * of the task it is currently executing, or "Idle" if it is
> >>>>> currently
> >>>>> * idle.
> >>>>>
> >>>>> ThreadPool predates Java 5, it looks like we can fix this
> >>>>>
> >> by using an Executor from Java 5, we can look at limiting
> the number
> >> of threads created based on available CPU's and a scaling
> factor and
> >> place the tasks in a BlockingQueue, so if the queue is filled, it
> >> blocks.
> >>
> >>>>> Can you report the issue as a Bug on Jira for me, I'll
> >>>>>
> >> fix this before the next release.
> >>
> >>>>> Regards,
> >>>>>
> >>>>> Peter.
> >>>>>
> >>>>> Peter Firmstone wrote:
> >>>>>
> >>>>>
> >>>>>> Hi Peter,
> >>>>>>
> >>>>>> I was wondering if you had any thoughts on this post
> >>>>>>
> >> from Bryan on River users?
> >>
> >>>>>> Hope you don't mind me asking ;)
> >>>>>>
> >>>>>> Best Regards,
> >>>>>>
> >>>>>> Peter Firmstone.
> >>>>>>
> >>>>>>
> >>>>>>
> >> -------------------------------------------------------------------
> >>
> >>>>>> -----
> >>>>>>
> >>>>>> Subject:
> >>>>>> Re: DGC threads issue
> >>>>>> From:
> >>>>>> Tom Hobbs <tv...@googlemail.com>
> >>>>>> Date:
> >>>>>> Thu, 12 Jan 2012 20:45:01 +0000
> >>>>>> To:
> >>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>
> >>>>>> To:
> >>>>>> user@river.apache.org, dev@river.apache.org
> >>>>>>
> >>>>>>
> >>>>>> Hi Bryan,
> >>>>>>
> >>>>>> Sorry that no one got back to you about this.  I'm
> afraid that I
> >>>>>> don't know the answer to your question, I've copied
> the dev list
> >>>>>> into this email in case someone who monitors that list
> (but not
> >>>>>> this one) has any ideas.
> >>>>>>
> >>>>>> Best regards,
> >>>>>>
> >>>>>> Tom
> >>>>>>
> >>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> >>>>>>
> >> <br...@systap.com> wrote:
> >>
> >>>>>>
> >>>>>>
> >>>>>>> Just to follow up on this thread myself.  I modified
> >>>>>>>
> >> the pattern to return a "thick" future rather than a proxy for the
> >> future.  This caused the RMI call to wait on the server until the
> >> future was done and then sent back the outcome.  This
> "fixed" the DGC
> >> memory/thread leak by reducing the number of exported proxies
> >> drammatically.
> >>
> >>>>>>> In terms of best practices, is distributed DGC simply
> >>>>>>>
> >> not useful for exported objects with short life spans?
> Can it only
> >> be used with proxies for relatively long lived services?
> >>
> >>>>>>> Thanks,
> >>>>>>> Bryan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Bryan Thompson
> >>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>>>>>>> To: user@river.apache.org
> >>>>>>>> Subject: DGC threads issue
> >>>>>>>>
> >>>>>>>> Hello,
> >>>>>>>>
> >>>>>>>> Background:
> >>>>>>>>
> >>>>>>>> I am seeing what would appear to be one DGC thread
> >>>>>>>>
> >> allocated per
> >>
> >>>>>>>> exported object.  This is using River 2.2 and Sun JDK
> >>>>>>>>
> >> 1.6.0_17.
> >>
> >>>>>>>> Relevant configuration parameters are below.
> >>>>>>>>
> >>>>>>>> I am observing problems with the DGC threads not being
> >>>>>>>>
> >> retired on
> >>
> >>>>>>>> a timely basis.  The exported objects are proxies
> for Futures
> >>>>>>>> which are being executed on the service.  The code
> pattern is
> >>>>>>>> such that the proxied Future goes out of lexical scope quite
> >>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> >>>>>>>>
> >>>>>>>> Under a modest load, a large number of such Futures
> >>>>>>>>
> >> are exported
> >>
> >>>>>>>> which results in a large number of long lived DGC
> >>>>>>>>
> >> threads.  This
> >>
> >>>>>>>> turns into a problem for the JVM due to the stack
> >>>>>>>>
> >> allocation per
> >>
> >>>>>>>> thread.  Presumably this is not good for other
> reasons as well
> >>>>>>>> (e.g., scheduling).
> >>>>>>>>
> >>>>>>>> I have tried to override the leaseValue and checkInterval
> >>>>>>>> defaults per the configuration options below.  I
> >>>>>>>>
> >> suspect that the
> >>
> >>>>>>>> lease interval is somehow not being obeyed, which is
> >>>>>>>>
> >> presumably a
> >>
> >>>>>>>> problem on my end.  However, I can verify that the
> >>>>>>>>
> >> configuration
> >>
> >>>>>>>> values are in fact showing up in
> >>>>>>>> System.getProperties() for at least some of the JVMs
> involved
> >>>>>>>> (the one which drives the workload and the one that I am
> >>>>>>>> monitoring with the large number of DGC lease threads).
> >>>>>>>>
> >>>>>>>> Some questions:
> >>>>>>>>
> >>>>>>>> Is this one-thread-per-exported proxy the expected
> >>>>>>>>
> >> behavior when
> >>
> >>>>>>>> DGC is requested for the exported object?
> >>>>>>>>
> >>>>>>>> The DGC lease checker threads appear to expire ~14 -
> >>>>>>>>
> >> 15 minutes
> >>
> >>>>>>>> after I terminate the process which was originating the RMI
> >>>>>>>> requests.  This is close the sum of the default
> >>>>>>>>
> >> leaseValue (10m)
> >>
> >>>>>>>> and checkInterval (5m) parameters, but maybe there is
> >>>>>>>>
> >> some other
> >>
> >>>>>>>> timeout which is controlling this?  If this is the sum
> >>>>>>>>
> >> of those
> >>
> >>>>>>>> parameters, why would the DGC lease threads live until
> >>>>>>>>
> >> the sum of
> >>
> >>>>>>>> those values?  I thought that the lease would expire
> after the
> >>>>>>>> leaseValue (10m default).
> >>>>>>>>
> >>>>>>>> Can the issue I am observing be caused by a low heap
> >>>>>>>>
> >> pressure on
> >>
> >>>>>>>> the JVM to which the RMI proxies were exported?  If it
> >>>>>>>>
> >> fails to
> >>
> >>>>>>>> GC those proxies, even though they are reachable, could that
> >>>>>>>> cause DGC to continue to retain those proxies on the
> JVM which
> >>>>>>>> exported them?
> >>>>>>>>
> >>>>>>>> Is there any way to configure DGC to use a thread pool
> >>>>>>>>
> >> or to have
> >>
> >>>>>>>> the leases managed by a single thread?
> >>>>>>>>
> >>>>>>>> Is it possible that there is an interaction with the
> >>>>>>>>
> >> useNIO option?
> >>
> >>>>>>>> Relevant options that I am using include:
> >>>>>>>>
> >>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
> >>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> >>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> >>>>>>>>
> >>>>>>>> Thanks in advance,
> >>>>>>>> Bryan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: [Fwd: Re: DGC threads issue]

[Peter Firmstone <ji...@zeus.net.au>]

Brian,

Do you have a Linux / Unix build environment? 

River isn't building yet on Window's, FreeBSD, or OpenBSD, known build 
environments that work are Ubuntu and Solaris.  If not, that's ok, well 
be able to get the necessary jar's off Hudson.

The DGC lease property values you need to set are:

com.sun.jini.jeri.dgc.leaseValue=30000
com.sun.jini.jeri.dgc.checkInterval=15000

RMI (called JRMP) is superseded code now, River doesn't use that unless 
you configure your Exporter to be JrmpExporter, but you're better off 
sticking with BasicJeriExporter.

Regards,

Peter.

Bryan Thompson wrote:
> Peter,
>
> I would be happy to test out a fix on this.  Can you point out the modified line and link me to any directions on how to rebuild the necessary jar(s)?
>
> Did you already file an issue for this?  If not, I will file an issue and then update it when I test out this fix.
>
> Thanks,
> Bryan 
>
>   
>> -----Original Message-----
>> From: Peter Firmstone [mailto:jini@zeus.net.au] 
>> Sent: Friday, January 13, 2012 3:21 AM
>> To: dev@river.apache.org
>> Cc: Bryan Thompson; user@river.apache.org
>> Subject: Re: [Fwd: Re: DGC threads issue]
>>
>> Ok, found the problem, when I fixed River-142, I introduced a 
>> new bug, after creating a new DGC thread, I didn't set the 
>> running flag to true, so each time, a new thread is created.
>>
>> Thankfully it's a very easy fix.
>>
>> Cheers,
>>
>> Peter.
>>
>> Peter Jones wrote:
>>     
>>> Peter,
>>>
>>> That internal Executor interface permits neither 
>>>       
>> synchronous execution in the calling thread ("this method 
>> itself must not block") nor any small bound on the number of 
>> Runnables that can be executed concurrently ("execution of a 
>> given action must not be delayed indefinitely in order to 
>> complete execution of a different action passed to a 
>> different invocation of this method").  In other words, 
>> concurrency-wise, it should be equivalent to "new 
>> Thread(runnable, name)", or what the result of 
>> j.u.c.Executors.newCachedThreadPool() would do.  Otherwise, 
>> callers would not function correctly, because they expect to 
>> be able to pass Runnables that execute indefinitely, not just 
>> short-lived work tasks.
>>     
>>> The real purpose of this internal Executor/ThreadPool was 
>>>       
>> to provide an internal alternative to "new Thread(...)" that 
>> takes care of certain things in common, like:
>>     
>>> - thread reuse instead of creation when reasonable (idle timeout)
>>> - but still with threads (re)named for the task being executed
>>> - insulation of any thread creation permission requirement from 
>>> context calling execute (NewThreadAction)
>>> - but instead requiring direct user of thread pool to be 
>>>       
>> trusted not 
>>     
>>> to abuse that insulation, with permission requirement for access 
>>> (ThreadPoolPermission and GetThreadPoolAction)
>>> - common logging of uncaught exceptions
>>>
>>> -- Peter
>>>
>>>
>>> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
>>>
>>>   
>>>       
>>>> ThreadPool implements the com.sun.jini.thread.Executor interface.
>>>>
>>>> Because the interface states it should not block, I think 
>>>>         
>> the calling thread should execute the task when the thread 
>> pool becomes saturated, rather than continue to create new 
>> threads as per the current implementation.  This will ensure 
>> that the task is completed, the calling thread, if it uses a 
>> sequence of tasks with dependencies will have to ensure that 
>> it submits the tasks in order.  Since only the caller knows 
>> the order, it makes sense for this to be the callers 
>> responsibility.  As a result the executor will honor the non 
>> blocking contract.  For that reason we'll use a zero length 
>> queue, probably a SynchronousQueue.
>>     
>>>> Regards,
>>>>
>>>> Peter.
>>>>
>>>> /**
>>>> * Executor is an abstraction for a thread factory or 
>>>>         
>> thread pool for
>>     
>>>> * executing actions asynchronously.
>>>> *
>>>> * @author    Sun Microsystems, Inc.
>>>> *
>>>> */
>>>> public interface Executor {
>>>>
>>>>   /**
>>>>    * Executes the given Runnable action asynchronously in 
>>>>         
>> some thread.
>>     
>>>>    *
>>>>    * The implementation may create a new thread to execute 
>>>>         
>> the action,
>>     
>>>>    * or it may execute the action in an existing thread.
>>>>    *
>>>>    * The execution of a given action must not be delayed 
>>>>         
>> indefinitely
>>     
>>>>    * in order to complete execution of a different action 
>>>>         
>> passed to a
>>     
>>>>    * different invocation of this method.  In other words, the
>>>>    * implementation must assume that there may be 
>>>>         
>> arbitrary dependencies
>>     
>>>>    * between actions passed to this method, so it needs to 
>>>>         
>> be careful
>>     
>>>>    * to avoid potential deadlock by delaying execution of 
>>>>         
>> one action
>>     
>>>>    * indefinitely until another completes.
>>>>    *
>>>>    * Also, this method itself must not block, because it 
>>>>         
>> may be invoked
>>     
>>>>    * by code that is serially processing data to produce 
>>>>         
>> multiple such
>>     
>>>>    * arbitrarily-dependent actions that need to be executed.
>>>>    *
>>>>    * @param    runnable the Runnable action to execute
>>>>    *
>>>>    * @param    name string to include in the name of the 
>>>>         
>> thread used
>>     
>>>>    * to execute the action
>>>>    */
>>>>   void execute(Runnable runnable, String name); }
>>>>
>>>> Peter Firmstone wrote:
>>>>     
>>>>         
>>>>> Thanks Brian,
>>>>>
>>>>> Looking at our implementation code, DGC uses an Executor 
>>>>>           
>> called ThreadPool, it's javadoc states:
>>     
>>>>> /**
>>>>> * ThreadPool is a simple thread pool implementation of 
>>>>>           
>> the Executor
>>     
>>>>> * interface.
>>>>> *
>>>>> * A new task is always given to an idle thread, if one is 
>>>>>           
>> available;
>>     
>>>>> * otherwise, a new thread is always created.  There is no minimum
>>>>> * warm thread count, nor is there a maximum thread count 
>>>>>           
>> (tasks are
>>     
>>>>> * never queued unless there are sufficient idle threads to execute
>>>>> * them).
>>>>> *
>>>>> * New threads are created as daemon threads in the thread 
>>>>>           
>> group that
>>     
>>>>> * was passed to the ThreadPool instance's constructor.  Each 
>>>>> thread's
>>>>> * name is the prefix NewThreadAction.NAME_PREFIX followed by the 
>>>>> name
>>>>> * of the task it is currently executing, or "Idle" if it is 
>>>>> currently
>>>>> * idle.
>>>>>
>>>>> ThreadPool predates Java 5, it looks like we can fix this 
>>>>>           
>> by using an Executor from Java 5, we can look at limiting the 
>> number of threads created based on available CPU's and a 
>> scaling factor and place the tasks in a BlockingQueue, so if 
>> the queue is filled, it blocks.
>>     
>>>>> Can you report the issue as a Bug on Jira for me, I'll 
>>>>>           
>> fix this before the next release.
>>     
>>>>> Regards,
>>>>>
>>>>> Peter.
>>>>>
>>>>> Peter Firmstone wrote:
>>>>>       
>>>>>           
>>>>>> Hi Peter,
>>>>>>
>>>>>> I was wondering if you had any thoughts on this post 
>>>>>>             
>> from Bryan on River users?
>>     
>>>>>> Hope you don't mind me asking ;)
>>>>>>
>>>>>> Best Regards,
>>>>>>
>>>>>> Peter Firmstone.
>>>>>>
>>>>>>
>>>>>>             
>> -------------------------------------------------------------------
>>     
>>>>>> -----
>>>>>>
>>>>>> Subject:
>>>>>> Re: DGC threads issue
>>>>>> From:
>>>>>> Tom Hobbs <tv...@googlemail.com>
>>>>>> Date:
>>>>>> Thu, 12 Jan 2012 20:45:01 +0000
>>>>>> To:
>>>>>> user@river.apache.org, dev@river.apache.org
>>>>>>
>>>>>> To:
>>>>>> user@river.apache.org, dev@river.apache.org
>>>>>>
>>>>>>
>>>>>> Hi Bryan,
>>>>>>
>>>>>> Sorry that no one got back to you about this.  I'm afraid that I 
>>>>>> don't know the answer to your question, I've copied the dev list 
>>>>>> into this email in case someone who monitors that list (but not 
>>>>>> this one) has any ideas.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Tom
>>>>>>
>>>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson 
>>>>>>             
>> <br...@systap.com> wrote:
>>     
>>>>>>         
>>>>>>             
>>>>>>> Just to follow up on this thread myself.  I modified 
>>>>>>>               
>> the pattern to return a "thick" future rather than a proxy 
>> for the future.  This caused the RMI call to wait on the 
>> server until the future was done and then sent back the 
>> outcome.  This "fixed" the DGC memory/thread leak by reducing 
>> the number of exported proxies drammatically.
>>     
>>>>>>> In terms of best practices, is distributed DGC simply 
>>>>>>>               
>> not useful for exported objects with short life spans?  Can 
>> it only be used with proxies for relatively long lived services?
>>     
>>>>>>> Thanks,
>>>>>>> Bryan
>>>>>>>
>>>>>>>   
>>>>>>>           
>>>>>>>               
>>>>>>>> -----Original Message-----
>>>>>>>> From: Bryan Thompson
>>>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
>>>>>>>> To: user@river.apache.org
>>>>>>>> Subject: DGC threads issue
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Background:
>>>>>>>>
>>>>>>>> I am seeing what would appear to be one DGC thread 
>>>>>>>>                 
>> allocated per 
>>     
>>>>>>>> exported object.  This is using River 2.2 and Sun JDK 
>>>>>>>>                 
>> 1.6.0_17.  
>>     
>>>>>>>> Relevant configuration parameters are below.
>>>>>>>>
>>>>>>>> I am observing problems with the DGC threads not being 
>>>>>>>>                 
>> retired on 
>>     
>>>>>>>> a timely basis.  The exported objects are proxies for Futures 
>>>>>>>> which are being executed on the service.  The code pattern is 
>>>>>>>> such that the proxied Future goes out of lexical scope quite 
>>>>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
>>>>>>>>
>>>>>>>> Under a modest load, a large number of such Futures 
>>>>>>>>                 
>> are exported 
>>     
>>>>>>>> which results in a large number of long lived DGC 
>>>>>>>>                 
>> threads.  This 
>>     
>>>>>>>> turns into a problem for the JVM due to the stack 
>>>>>>>>                 
>> allocation per 
>>     
>>>>>>>> thread.  Presumably this is not good for other reasons as well 
>>>>>>>> (e.g., scheduling).
>>>>>>>>
>>>>>>>> I have tried to override the leaseValue and checkInterval 
>>>>>>>> defaults per the configuration options below.  I 
>>>>>>>>                 
>> suspect that the 
>>     
>>>>>>>> lease interval is somehow not being obeyed, which is 
>>>>>>>>                 
>> presumably a 
>>     
>>>>>>>> problem on my end.  However, I can verify that the 
>>>>>>>>                 
>> configuration 
>>     
>>>>>>>> values are in fact showing up in
>>>>>>>> System.getProperties() for at least some of the JVMs involved 
>>>>>>>> (the one which drives the workload and the one that I am 
>>>>>>>> monitoring with the large number of DGC lease threads).
>>>>>>>>
>>>>>>>> Some questions:
>>>>>>>>
>>>>>>>> Is this one-thread-per-exported proxy the expected 
>>>>>>>>                 
>> behavior when 
>>     
>>>>>>>> DGC is requested for the exported object?
>>>>>>>>
>>>>>>>> The DGC lease checker threads appear to expire ~14 - 
>>>>>>>>                 
>> 15 minutes 
>>     
>>>>>>>> after I terminate the process which was originating the RMI 
>>>>>>>> requests.  This is close the sum of the default 
>>>>>>>>                 
>> leaseValue (10m) 
>>     
>>>>>>>> and checkInterval (5m) parameters, but maybe there is 
>>>>>>>>                 
>> some other 
>>     
>>>>>>>> timeout which is controlling this?  If this is the sum 
>>>>>>>>                 
>> of those 
>>     
>>>>>>>> parameters, why would the DGC lease threads live until 
>>>>>>>>                 
>> the sum of 
>>     
>>>>>>>> those values?  I thought that the lease would expire after the 
>>>>>>>> leaseValue (10m default).
>>>>>>>>
>>>>>>>> Can the issue I am observing be caused by a low heap 
>>>>>>>>                 
>> pressure on 
>>     
>>>>>>>> the JVM to which the RMI proxies were exported?  If it 
>>>>>>>>                 
>> fails to 
>>     
>>>>>>>> GC those proxies, even though they are reachable, could that 
>>>>>>>> cause DGC to continue to retain those proxies on the JVM which 
>>>>>>>> exported them?
>>>>>>>>
>>>>>>>> Is there any way to configure DGC to use a thread pool 
>>>>>>>>                 
>> or to have 
>>     
>>>>>>>> the leases managed by a single thread?
>>>>>>>>
>>>>>>>> Is it possible that there is an interaction with the 
>>>>>>>>                 
>> useNIO option?
>>     
>>>>>>>> Relevant options that I am using include:
>>>>>>>>
>>>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
>>>>>>>>   -Djava.rmi.dgc.leaseValue=30000
>>>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
>>>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
>>>>>>>>
>>>>>>>> Thanks in advance,
>>>>>>>> Bryan
>>>>>>>>      
>>>>>>>>             
>>>>>>>>                 
>>>>>>  
>>>>>>         
>>>>>>             
>>>>>       
>>>>>           
>>>   
>>>       
>>     

RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

Peter,

I would be happy to test out a fix on this.  Can you point out the modified line and link me to any directions on how to rebuild the necessary jar(s)?

Did you already file an issue for this?  If not, I will file an issue and then update it when I test out this fix.

Thanks,
Bryan 

> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au] 
> Sent: Friday, January 13, 2012 3:21 AM
> To: dev@river.apache.org
> Cc: Bryan Thompson; user@river.apache.org
> Subject: Re: [Fwd: Re: DGC threads issue]
> 
> Ok, found the problem, when I fixed River-142, I introduced a 
> new bug, after creating a new DGC thread, I didn't set the 
> running flag to true, so each time, a new thread is created.
> 
> Thankfully it's a very easy fix.
> 
> Cheers,
> 
> Peter.
> 
> Peter Jones wrote:
> > Peter,
> >
> > That internal Executor interface permits neither 
> synchronous execution in the calling thread ("this method 
> itself must not block") nor any small bound on the number of 
> Runnables that can be executed concurrently ("execution of a 
> given action must not be delayed indefinitely in order to 
> complete execution of a different action passed to a 
> different invocation of this method").  In other words, 
> concurrency-wise, it should be equivalent to "new 
> Thread(runnable, name)", or what the result of 
> j.u.c.Executors.newCachedThreadPool() would do.  Otherwise, 
> callers would not function correctly, because they expect to 
> be able to pass Runnables that execute indefinitely, not just 
> short-lived work tasks.
> >
> > The real purpose of this internal Executor/ThreadPool was 
> to provide an internal alternative to "new Thread(...)" that 
> takes care of certain things in common, like:
> >
> > - thread reuse instead of creation when reasonable (idle timeout)
> > - but still with threads (re)named for the task being executed
> > - insulation of any thread creation permission requirement from 
> > context calling execute (NewThreadAction)
> > - but instead requiring direct user of thread pool to be 
> trusted not 
> > to abuse that insulation, with permission requirement for access 
> > (ThreadPoolPermission and GetThreadPoolAction)
> > - common logging of uncaught exceptions
> >
> > -- Peter
> >
> >
> > On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> >
> >   
> >> ThreadPool implements the com.sun.jini.thread.Executor interface.
> >>
> >> Because the interface states it should not block, I think 
> the calling thread should execute the task when the thread 
> pool becomes saturated, rather than continue to create new 
> threads as per the current implementation.  This will ensure 
> that the task is completed, the calling thread, if it uses a 
> sequence of tasks with dependencies will have to ensure that 
> it submits the tasks in order.  Since only the caller knows 
> the order, it makes sense for this to be the callers 
> responsibility.  As a result the executor will honor the non 
> blocking contract.  For that reason we'll use a zero length 
> queue, probably a SynchronousQueue.
> >>
> >> Regards,
> >>
> >> Peter.
> >>
> >> /**
> >> * Executor is an abstraction for a thread factory or 
> thread pool for
> >> * executing actions asynchronously.
> >> *
> >> * @author    Sun Microsystems, Inc.
> >> *
> >> */
> >> public interface Executor {
> >>
> >>   /**
> >>    * Executes the given Runnable action asynchronously in 
> some thread.
> >>    *
> >>    * The implementation may create a new thread to execute 
> the action,
> >>    * or it may execute the action in an existing thread.
> >>    *
> >>    * The execution of a given action must not be delayed 
> indefinitely
> >>    * in order to complete execution of a different action 
> passed to a
> >>    * different invocation of this method.  In other words, the
> >>    * implementation must assume that there may be 
> arbitrary dependencies
> >>    * between actions passed to this method, so it needs to 
> be careful
> >>    * to avoid potential deadlock by delaying execution of 
> one action
> >>    * indefinitely until another completes.
> >>    *
> >>    * Also, this method itself must not block, because it 
> may be invoked
> >>    * by code that is serially processing data to produce 
> multiple such
> >>    * arbitrarily-dependent actions that need to be executed.
> >>    *
> >>    * @param    runnable the Runnable action to execute
> >>    *
> >>    * @param    name string to include in the name of the 
> thread used
> >>    * to execute the action
> >>    */
> >>   void execute(Runnable runnable, String name); }
> >>
> >> Peter Firmstone wrote:
> >>     
> >>> Thanks Brian,
> >>>
> >>> Looking at our implementation code, DGC uses an Executor 
> called ThreadPool, it's javadoc states:
> >>>
> >>> /**
> >>> * ThreadPool is a simple thread pool implementation of 
> the Executor
> >>> * interface.
> >>> *
> >>> * A new task is always given to an idle thread, if one is 
> available;
> >>> * otherwise, a new thread is always created.  There is no minimum
> >>> * warm thread count, nor is there a maximum thread count 
> (tasks are
> >>> * never queued unless there are sufficient idle threads to execute
> >>> * them).
> >>> *
> >>> * New threads are created as daemon threads in the thread 
> group that
> >>> * was passed to the ThreadPool instance's constructor.  Each 
> >>> thread's
> >>> * name is the prefix NewThreadAction.NAME_PREFIX followed by the 
> >>> name
> >>> * of the task it is currently executing, or "Idle" if it is 
> >>> currently
> >>> * idle.
> >>>
> >>> ThreadPool predates Java 5, it looks like we can fix this 
> by using an Executor from Java 5, we can look at limiting the 
> number of threads created based on available CPU's and a 
> scaling factor and place the tasks in a BlockingQueue, so if 
> the queue is filled, it blocks.
> >>>
> >>> Can you report the issue as a Bug on Jira for me, I'll 
> fix this before the next release.
> >>>
> >>> Regards,
> >>>
> >>> Peter.
> >>>
> >>> Peter Firmstone wrote:
> >>>       
> >>>> Hi Peter,
> >>>>
> >>>> I was wondering if you had any thoughts on this post 
> from Bryan on River users?
> >>>>
> >>>> Hope you don't mind me asking ;)
> >>>>
> >>>> Best Regards,
> >>>>
> >>>> Peter Firmstone.
> >>>>
> >>>> 
> -------------------------------------------------------------------
> >>>> -----
> >>>>
> >>>> Subject:
> >>>> Re: DGC threads issue
> >>>> From:
> >>>> Tom Hobbs <tv...@googlemail.com>
> >>>> Date:
> >>>> Thu, 12 Jan 2012 20:45:01 +0000
> >>>> To:
> >>>> user@river.apache.org, dev@river.apache.org
> >>>>
> >>>> To:
> >>>> user@river.apache.org, dev@river.apache.org
> >>>>
> >>>>
> >>>> Hi Bryan,
> >>>>
> >>>> Sorry that no one got back to you about this.  I'm afraid that I 
> >>>> don't know the answer to your question, I've copied the dev list 
> >>>> into this email in case someone who monitors that list (but not 
> >>>> this one) has any ideas.
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Tom
> >>>>
> >>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson 
> <br...@systap.com> wrote:
> >>>>
> >>>>         
> >>>>> Just to follow up on this thread myself.  I modified 
> the pattern to return a "thick" future rather than a proxy 
> for the future.  This caused the RMI call to wait on the 
> server until the future was done and then sent back the 
> outcome.  This "fixed" the DGC memory/thread leak by reducing 
> the number of exported proxies drammatically.
> >>>>>
> >>>>> In terms of best practices, is distributed DGC simply 
> not useful for exported objects with short life spans?  Can 
> it only be used with proxies for relatively long lived services?
> >>>>>
> >>>>> Thanks,
> >>>>> Bryan
> >>>>>
> >>>>>   
> >>>>>           
> >>>>>> -----Original Message-----
> >>>>>> From: Bryan Thompson
> >>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>>>>> To: user@river.apache.org
> >>>>>> Subject: DGC threads issue
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> Background:
> >>>>>>
> >>>>>> I am seeing what would appear to be one DGC thread 
> allocated per 
> >>>>>> exported object.  This is using River 2.2 and Sun JDK 
> 1.6.0_17.  
> >>>>>> Relevant configuration parameters are below.
> >>>>>>
> >>>>>> I am observing problems with the DGC threads not being 
> retired on 
> >>>>>> a timely basis.  The exported objects are proxies for Futures 
> >>>>>> which are being executed on the service.  The code pattern is 
> >>>>>> such that the proxied Future goes out of lexical scope quite 
> >>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> >>>>>>
> >>>>>> Under a modest load, a large number of such Futures 
> are exported 
> >>>>>> which results in a large number of long lived DGC 
> threads.  This 
> >>>>>> turns into a problem for the JVM due to the stack 
> allocation per 
> >>>>>> thread.  Presumably this is not good for other reasons as well 
> >>>>>> (e.g., scheduling).
> >>>>>>
> >>>>>> I have tried to override the leaseValue and checkInterval 
> >>>>>> defaults per the configuration options below.  I 
> suspect that the 
> >>>>>> lease interval is somehow not being obeyed, which is 
> presumably a 
> >>>>>> problem on my end.  However, I can verify that the 
> configuration 
> >>>>>> values are in fact showing up in
> >>>>>> System.getProperties() for at least some of the JVMs involved 
> >>>>>> (the one which drives the workload and the one that I am 
> >>>>>> monitoring with the large number of DGC lease threads).
> >>>>>>
> >>>>>> Some questions:
> >>>>>>
> >>>>>> Is this one-thread-per-exported proxy the expected 
> behavior when 
> >>>>>> DGC is requested for the exported object?
> >>>>>>
> >>>>>> The DGC lease checker threads appear to expire ~14 - 
> 15 minutes 
> >>>>>> after I terminate the process which was originating the RMI 
> >>>>>> requests.  This is close the sum of the default 
> leaseValue (10m) 
> >>>>>> and checkInterval (5m) parameters, but maybe there is 
> some other 
> >>>>>> timeout which is controlling this?  If this is the sum 
> of those 
> >>>>>> parameters, why would the DGC lease threads live until 
> the sum of 
> >>>>>> those values?  I thought that the lease would expire after the 
> >>>>>> leaseValue (10m default).
> >>>>>>
> >>>>>> Can the issue I am observing be caused by a low heap 
> pressure on 
> >>>>>> the JVM to which the RMI proxies were exported?  If it 
> fails to 
> >>>>>> GC those proxies, even though they are reachable, could that 
> >>>>>> cause DGC to continue to retain those proxies on the JVM which 
> >>>>>> exported them?
> >>>>>>
> >>>>>> Is there any way to configure DGC to use a thread pool 
> or to have 
> >>>>>> the leases managed by a single thread?
> >>>>>>
> >>>>>> Is it possible that there is an interaction with the 
> useNIO option?
> >>>>>>
> >>>>>> Relevant options that I am using include:
> >>>>>>
> >>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>>>>   -Djava.rmi.dgc.leaseValue=30000
> >>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> >>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> >>>>>>
> >>>>>> Thanks in advance,
> >>>>>> Bryan
> >>>>>>      
> >>>>>>             
> >>>>  
> >>>>         
> >>>       
> >
> >
> >   
> 
> 

RE: [Fwd: Re: DGC threads issue]

[Bryan Thompson <br...@systap.com>]

Peter,

I would be happy to test out a fix on this.  Can you point out the modified line and link me to any directions on how to rebuild the necessary jar(s)?

Did you already file an issue for this?  If not, I will file an issue and then update it when I test out this fix.

Thanks,
Bryan 

> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au] 
> Sent: Friday, January 13, 2012 3:21 AM
> To: dev@river.apache.org
> Cc: Bryan Thompson; user@river.apache.org
> Subject: Re: [Fwd: Re: DGC threads issue]
> 
> Ok, found the problem, when I fixed River-142, I introduced a 
> new bug, after creating a new DGC thread, I didn't set the 
> running flag to true, so each time, a new thread is created.
> 
> Thankfully it's a very easy fix.
> 
> Cheers,
> 
> Peter.
> 
> Peter Jones wrote:
> > Peter,
> >
> > That internal Executor interface permits neither 
> synchronous execution in the calling thread ("this method 
> itself must not block") nor any small bound on the number of 
> Runnables that can be executed concurrently ("execution of a 
> given action must not be delayed indefinitely in order to 
> complete execution of a different action passed to a 
> different invocation of this method").  In other words, 
> concurrency-wise, it should be equivalent to "new 
> Thread(runnable, name)", or what the result of 
> j.u.c.Executors.newCachedThreadPool() would do.  Otherwise, 
> callers would not function correctly, because they expect to 
> be able to pass Runnables that execute indefinitely, not just 
> short-lived work tasks.
> >
> > The real purpose of this internal Executor/ThreadPool was 
> to provide an internal alternative to "new Thread(...)" that 
> takes care of certain things in common, like:
> >
> > - thread reuse instead of creation when reasonable (idle timeout)
> > - but still with threads (re)named for the task being executed
> > - insulation of any thread creation permission requirement from 
> > context calling execute (NewThreadAction)
> > - but instead requiring direct user of thread pool to be 
> trusted not 
> > to abuse that insulation, with permission requirement for access 
> > (ThreadPoolPermission and GetThreadPoolAction)
> > - common logging of uncaught exceptions
> >
> > -- Peter
> >
> >
> > On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> >
> >   
> >> ThreadPool implements the com.sun.jini.thread.Executor interface.
> >>
> >> Because the interface states it should not block, I think 
> the calling thread should execute the task when the thread 
> pool becomes saturated, rather than continue to create new 
> threads as per the current implementation.  This will ensure 
> that the task is completed, the calling thread, if it uses a 
> sequence of tasks with dependencies will have to ensure that 
> it submits the tasks in order.  Since only the caller knows 
> the order, it makes sense for this to be the callers 
> responsibility.  As a result the executor will honor the non 
> blocking contract.  For that reason we'll use a zero length 
> queue, probably a SynchronousQueue.
> >>
> >> Regards,
> >>
> >> Peter.
> >>
> >> /**
> >> * Executor is an abstraction for a thread factory or 
> thread pool for
> >> * executing actions asynchronously.
> >> *
> >> * @author    Sun Microsystems, Inc.
> >> *
> >> */
> >> public interface Executor {
> >>
> >>   /**
> >>    * Executes the given Runnable action asynchronously in 
> some thread.
> >>    *
> >>    * The implementation may create a new thread to execute 
> the action,
> >>    * or it may execute the action in an existing thread.
> >>    *
> >>    * The execution of a given action must not be delayed 
> indefinitely
> >>    * in order to complete execution of a different action 
> passed to a
> >>    * different invocation of this method.  In other words, the
> >>    * implementation must assume that there may be 
> arbitrary dependencies
> >>    * between actions passed to this method, so it needs to 
> be careful
> >>    * to avoid potential deadlock by delaying execution of 
> one action
> >>    * indefinitely until another completes.
> >>    *
> >>    * Also, this method itself must not block, because it 
> may be invoked
> >>    * by code that is serially processing data to produce 
> multiple such
> >>    * arbitrarily-dependent actions that need to be executed.
> >>    *
> >>    * @param    runnable the Runnable action to execute
> >>    *
> >>    * @param    name string to include in the name of the 
> thread used
> >>    * to execute the action
> >>    */
> >>   void execute(Runnable runnable, String name); }
> >>
> >> Peter Firmstone wrote:
> >>     
> >>> Thanks Brian,
> >>>
> >>> Looking at our implementation code, DGC uses an Executor 
> called ThreadPool, it's javadoc states:
> >>>
> >>> /**
> >>> * ThreadPool is a simple thread pool implementation of 
> the Executor
> >>> * interface.
> >>> *
> >>> * A new task is always given to an idle thread, if one is 
> available;
> >>> * otherwise, a new thread is always created.  There is no minimum
> >>> * warm thread count, nor is there a maximum thread count 
> (tasks are
> >>> * never queued unless there are sufficient idle threads to execute
> >>> * them).
> >>> *
> >>> * New threads are created as daemon threads in the thread 
> group that
> >>> * was passed to the ThreadPool instance's constructor.  Each 
> >>> thread's
> >>> * name is the prefix NewThreadAction.NAME_PREFIX followed by the 
> >>> name
> >>> * of the task it is currently executing, or "Idle" if it is 
> >>> currently
> >>> * idle.
> >>>
> >>> ThreadPool predates Java 5, it looks like we can fix this 
> by using an Executor from Java 5, we can look at limiting the 
> number of threads created based on available CPU's and a 
> scaling factor and place the tasks in a BlockingQueue, so if 
> the queue is filled, it blocks.
> >>>
> >>> Can you report the issue as a Bug on Jira for me, I'll 
> fix this before the next release.
> >>>
> >>> Regards,
> >>>
> >>> Peter.
> >>>
> >>> Peter Firmstone wrote:
> >>>       
> >>>> Hi Peter,
> >>>>
> >>>> I was wondering if you had any thoughts on this post 
> from Bryan on River users?
> >>>>
> >>>> Hope you don't mind me asking ;)
> >>>>
> >>>> Best Regards,
> >>>>
> >>>> Peter Firmstone.
> >>>>
> >>>> 
> -------------------------------------------------------------------
> >>>> -----
> >>>>
> >>>> Subject:
> >>>> Re: DGC threads issue
> >>>> From:
> >>>> Tom Hobbs <tv...@googlemail.com>
> >>>> Date:
> >>>> Thu, 12 Jan 2012 20:45:01 +0000
> >>>> To:
> >>>> user@river.apache.org, dev@river.apache.org
> >>>>
> >>>> To:
> >>>> user@river.apache.org, dev@river.apache.org
> >>>>
> >>>>
> >>>> Hi Bryan,
> >>>>
> >>>> Sorry that no one got back to you about this.  I'm afraid that I 
> >>>> don't know the answer to your question, I've copied the dev list 
> >>>> into this email in case someone who monitors that list (but not 
> >>>> this one) has any ideas.
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Tom
> >>>>
> >>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson 
> <br...@systap.com> wrote:
> >>>>
> >>>>         
> >>>>> Just to follow up on this thread myself.  I modified 
> the pattern to return a "thick" future rather than a proxy 
> for the future.  This caused the RMI call to wait on the 
> server until the future was done and then sent back the 
> outcome.  This "fixed" the DGC memory/thread leak by reducing 
> the number of exported proxies drammatically.
> >>>>>
> >>>>> In terms of best practices, is distributed DGC simply 
> not useful for exported objects with short life spans?  Can 
> it only be used with proxies for relatively long lived services?
> >>>>>
> >>>>> Thanks,
> >>>>> Bryan
> >>>>>
> >>>>>   
> >>>>>           
> >>>>>> -----Original Message-----
> >>>>>> From: Bryan Thompson
> >>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>>>>> To: user@river.apache.org
> >>>>>> Subject: DGC threads issue
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> Background:
> >>>>>>
> >>>>>> I am seeing what would appear to be one DGC thread 
> allocated per 
> >>>>>> exported object.  This is using River 2.2 and Sun JDK 
> 1.6.0_17.  
> >>>>>> Relevant configuration parameters are below.
> >>>>>>
> >>>>>> I am observing problems with the DGC threads not being 
> retired on 
> >>>>>> a timely basis.  The exported objects are proxies for Futures 
> >>>>>> which are being executed on the service.  The code pattern is 
> >>>>>> such that the proxied Future goes out of lexical scope quite 
> >>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> >>>>>>
> >>>>>> Under a modest load, a large number of such Futures 
> are exported 
> >>>>>> which results in a large number of long lived DGC 
> threads.  This 
> >>>>>> turns into a problem for the JVM due to the stack 
> allocation per 
> >>>>>> thread.  Presumably this is not good for other reasons as well 
> >>>>>> (e.g., scheduling).
> >>>>>>
> >>>>>> I have tried to override the leaseValue and checkInterval 
> >>>>>> defaults per the configuration options below.  I 
> suspect that the 
> >>>>>> lease interval is somehow not being obeyed, which is 
> presumably a 
> >>>>>> problem on my end.  However, I can verify that the 
> configuration 
> >>>>>> values are in fact showing up in
> >>>>>> System.getProperties() for at least some of the JVMs involved 
> >>>>>> (the one which drives the workload and the one that I am 
> >>>>>> monitoring with the large number of DGC lease threads).
> >>>>>>
> >>>>>> Some questions:
> >>>>>>
> >>>>>> Is this one-thread-per-exported proxy the expected 
> behavior when 
> >>>>>> DGC is requested for the exported object?
> >>>>>>
> >>>>>> The DGC lease checker threads appear to expire ~14 - 
> 15 minutes 
> >>>>>> after I terminate the process which was originating the RMI 
> >>>>>> requests.  This is close the sum of the default 
> leaseValue (10m) 
> >>>>>> and checkInterval (5m) parameters, but maybe there is 
> some other 
> >>>>>> timeout which is controlling this?  If this is the sum 
> of those 
> >>>>>> parameters, why would the DGC lease threads live until 
> the sum of 
> >>>>>> those values?  I thought that the lease would expire after the 
> >>>>>> leaseValue (10m default).
> >>>>>>
> >>>>>> Can the issue I am observing be caused by a low heap 
> pressure on 
> >>>>>> the JVM to which the RMI proxies were exported?  If it 
> fails to 
> >>>>>> GC those proxies, even though they are reachable, could that 
> >>>>>> cause DGC to continue to retain those proxies on the JVM which 
> >>>>>> exported them?
> >>>>>>
> >>>>>> Is there any way to configure DGC to use a thread pool 
> or to have 
> >>>>>> the leases managed by a single thread?
> >>>>>>
> >>>>>> Is it possible that there is an interaction with the 
> useNIO option?
> >>>>>>
> >>>>>> Relevant options that I am using include:
> >>>>>>
> >>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>>>>   -Djava.rmi.dgc.leaseValue=30000
> >>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> >>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> >>>>>>
> >>>>>> Thanks in advance,
> >>>>>> Bryan
> >>>>>>      
> >>>>>>             
> >>>>  
> >>>>         
> >>>       
> >
> >
> >   
> 
> 

Re: [Fwd: Re: DGC threads issue]

[Peter Firmstone <ji...@zeus.net.au>]

Ok, found the problem, when I fixed River-142, I introduced a new bug,  
after creating a new DGC thread, I didn't set the running flag to true, 
so each time, a new thread is created.

Thankfully it's a very easy fix.

Cheers,

Peter.

Peter Jones wrote:
> Peter,
>
> That internal Executor interface permits neither synchronous execution in the calling thread ("this method itself must not block") nor any small bound on the number of Runnables that can be executed concurrently ("execution of a given action must not be delayed indefinitely in order to complete execution of a different action passed to a different invocation of this method").  In other words, concurrency-wise, it should be equivalent to "new Thread(runnable, name)", or what the result of j.u.c.Executors.newCachedThreadPool() would do.  Otherwise, callers would not function correctly, because they expect to be able to pass Runnables that execute indefinitely, not just short-lived work tasks.
>
> The real purpose of this internal Executor/ThreadPool was to provide an internal alternative to "new Thread(...)" that takes care of certain things in common, like:
>
> - thread reuse instead of creation when reasonable (idle timeout)
> - but still with threads (re)named for the task being executed
> - insulation of any thread creation permission requirement from context calling execute (NewThreadAction)
> - but instead requiring direct user of thread pool to be trusted not to abuse that insulation, with permission requirement for access (ThreadPoolPermission and GetThreadPoolAction)
> - common logging of uncaught exceptions
>
> -- Peter
>
>
> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
>
>   
>> ThreadPool implements the com.sun.jini.thread.Executor interface.
>>
>> Because the interface states it should not block, I think the calling thread should execute the task when the thread pool becomes saturated, rather than continue to create new threads as per the current implementation.  This will ensure that the task is completed, the calling thread, if it uses a sequence of tasks with dependencies will have to ensure that it submits the tasks in order.  Since only the caller knows the order, it makes sense for this to be the callers responsibility.  As a result the executor will honor the non blocking contract.  For that reason we'll use a zero length queue, probably a SynchronousQueue.
>>
>> Regards,
>>
>> Peter.
>>
>> /**
>> * Executor is an abstraction for a thread factory or thread pool for
>> * executing actions asynchronously.
>> *
>> * @author    Sun Microsystems, Inc.
>> *
>> */
>> public interface Executor {
>>
>>   /**
>>    * Executes the given Runnable action asynchronously in some thread.
>>    *
>>    * The implementation may create a new thread to execute the action,
>>    * or it may execute the action in an existing thread.
>>    *
>>    * The execution of a given action must not be delayed indefinitely
>>    * in order to complete execution of a different action passed to a
>>    * different invocation of this method.  In other words, the
>>    * implementation must assume that there may be arbitrary dependencies
>>    * between actions passed to this method, so it needs to be careful
>>    * to avoid potential deadlock by delaying execution of one action
>>    * indefinitely until another completes.
>>    *
>>    * Also, this method itself must not block, because it may be invoked
>>    * by code that is serially processing data to produce multiple such
>>    * arbitrarily-dependent actions that need to be executed.
>>    *
>>    * @param    runnable the Runnable action to execute
>>    *
>>    * @param    name string to include in the name of the thread used
>>    * to execute the action
>>    */
>>   void execute(Runnable runnable, String name);
>> }
>>
>> Peter Firmstone wrote:
>>     
>>> Thanks Brian,
>>>
>>> Looking at our implementation code, DGC uses an Executor called ThreadPool, it's javadoc states:
>>>
>>> /**
>>> * ThreadPool is a simple thread pool implementation of the Executor
>>> * interface.
>>> *
>>> * A new task is always given to an idle thread, if one is available;
>>> * otherwise, a new thread is always created.  There is no minimum
>>> * warm thread count, nor is there a maximum thread count (tasks are
>>> * never queued unless there are sufficient idle threads to execute
>>> * them).
>>> *
>>> * New threads are created as daemon threads in the thread group that
>>> * was passed to the ThreadPool instance's constructor.  Each thread's
>>> * name is the prefix NewThreadAction.NAME_PREFIX followed by the name
>>> * of the task it is currently executing, or "Idle" if it is currently
>>> * idle.
>>>
>>> ThreadPool predates Java 5, it looks like we can fix this by using an Executor from Java 5, we can look at limiting the number of threads created based on available CPU's and a scaling factor and place the tasks in a BlockingQueue, so if the queue is filled, it blocks.
>>>
>>> Can you report the issue as a Bug on Jira for me, I'll fix this before the next release.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> Peter Firmstone wrote:
>>>       
>>>> Hi Peter,
>>>>
>>>> I was wondering if you had any thoughts on this post from Bryan on River users?
>>>>
>>>> Hope you don't mind me asking ;)
>>>>
>>>> Best Regards,
>>>>
>>>> Peter Firmstone.
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> Subject:
>>>> Re: DGC threads issue
>>>> From:
>>>> Tom Hobbs <tv...@googlemail.com>
>>>> Date:
>>>> Thu, 12 Jan 2012 20:45:01 +0000
>>>> To:
>>>> user@river.apache.org, dev@river.apache.org
>>>>
>>>> To:
>>>> user@river.apache.org, dev@river.apache.org
>>>>
>>>>
>>>> Hi Bryan,
>>>>
>>>> Sorry that no one got back to you about this.  I'm afraid that I don't
>>>> know the answer to your question, I've copied the dev list into this
>>>> email in case someone who monitors that list (but not this one) has
>>>> any ideas.
>>>>
>>>> Best regards,
>>>>
>>>> Tom
>>>>
>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson <br...@systap.com> wrote:
>>>>
>>>>         
>>>>> Just to follow up on this thread myself.  I modified the pattern to return a "thick" future rather than a proxy for the future.  This caused the RMI call to wait on the server until the future was done and then sent back the outcome.  This "fixed" the DGC memory/thread leak by reducing the number of exported proxies drammatically.
>>>>>
>>>>> In terms of best practices, is distributed DGC simply not useful for exported objects with short life spans?  Can it only be used with proxies for relatively long lived services?
>>>>>
>>>>> Thanks,
>>>>> Bryan
>>>>>
>>>>>   
>>>>>           
>>>>>> -----Original Message-----
>>>>>> From: Bryan Thompson
>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
>>>>>> To: user@river.apache.org
>>>>>> Subject: DGC threads issue
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Background:
>>>>>>
>>>>>> I am seeing what would appear to be one DGC thread allocated
>>>>>> per exported object.  This is using River 2.2 and Sun JDK
>>>>>> 1.6.0_17.  Relevant configuration parameters are below.
>>>>>>
>>>>>> I am observing problems with the DGC threads not being
>>>>>> retired on a timely basis.  The exported objects are proxies
>>>>>> for Futures which are being executed on the service.  The
>>>>>> code pattern is such that the proxied Future goes out of
>>>>>> lexical scope quite quickly.  E.g.,
>>>>>> rmiCallReturningProxyForFuture().get().
>>>>>>
>>>>>> Under a modest load, a large number of such Futures are
>>>>>> exported which results in a large number of long lived DGC
>>>>>> threads.  This turns into a problem for the JVM due to the
>>>>>> stack allocation per thread.  Presumably this is not good for
>>>>>> other reasons as well (e.g., scheduling).
>>>>>>
>>>>>> I have tried to override the leaseValue and checkInterval
>>>>>> defaults per the configuration options below.  I suspect that
>>>>>> the lease interval is somehow not being obeyed, which is
>>>>>> presumably a problem on my end.  However, I can verify that
>>>>>> the configuration values are in fact showing up in
>>>>>> System.getProperties() for at least some of the JVMs involved
>>>>>> (the one which drives the workload and the one that I am
>>>>>> monitoring with the large number of DGC lease threads).
>>>>>>
>>>>>> Some questions:
>>>>>>
>>>>>> Is this one-thread-per-exported proxy the expected behavior
>>>>>> when DGC is requested for the exported object?
>>>>>>
>>>>>> The DGC lease checker threads appear to expire ~14 - 15
>>>>>> minutes after I terminate the process which was originating
>>>>>> the RMI requests.  This is close the sum of the default
>>>>>> leaseValue (10m) and checkInterval (5m) parameters, but maybe
>>>>>> there is some other timeout which is controlling this?  If
>>>>>> this is the sum of those parameters, why would the DGC lease
>>>>>> threads live until the sum of those values?  I thought that
>>>>>> the lease would expire after the leaseValue (10m default).
>>>>>>
>>>>>> Can the issue I am observing be caused by a low heap pressure
>>>>>> on the JVM to which the RMI proxies were exported?  If it
>>>>>> fails to GC those proxies, even though they are reachable,
>>>>>> could that cause DGC to continue to retain those proxies on
>>>>>> the JVM which exported them?
>>>>>>
>>>>>> Is there any way to configure DGC to use a thread pool or to
>>>>>> have the leases managed by a single thread?
>>>>>>
>>>>>> Is it possible that there is an interaction with the useNIO option?
>>>>>>
>>>>>> Relevant options that I am using include:
>>>>>>
>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
>>>>>>   -Djava.rmi.dgc.leaseValue=30000
>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Bryan
>>>>>>      
>>>>>>             
>>>>  
>>>>         
>>>       
>
>
>   

Re: [Fwd: Re: DGC threads issue]

[Peter Firmstone <ji...@zeus.net.au>]

Ok, found the problem, when I fixed River-142, I introduced a new bug,  
after creating a new DGC thread, I didn't set the running flag to true, 
so each time, a new thread is created.

Thankfully it's a very easy fix.

Cheers,

Peter.

Peter Jones wrote:
> Peter,
>
> That internal Executor interface permits neither synchronous execution in the calling thread ("this method itself must not block") nor any small bound on the number of Runnables that can be executed concurrently ("execution of a given action must not be delayed indefinitely in order to complete execution of a different action passed to a different invocation of this method").  In other words, concurrency-wise, it should be equivalent to "new Thread(runnable, name)", or what the result of j.u.c.Executors.newCachedThreadPool() would do.  Otherwise, callers would not function correctly, because they expect to be able to pass Runnables that execute indefinitely, not just short-lived work tasks.
>
> The real purpose of this internal Executor/ThreadPool was to provide an internal alternative to "new Thread(...)" that takes care of certain things in common, like:
>
> - thread reuse instead of creation when reasonable (idle timeout)
> - but still with threads (re)named for the task being executed
> - insulation of any thread creation permission requirement from context calling execute (NewThreadAction)
> - but instead requiring direct user of thread pool to be trusted not to abuse that insulation, with permission requirement for access (ThreadPoolPermission and GetThreadPoolAction)
> - common logging of uncaught exceptions
>
> -- Peter
>
>
> On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
>
>   
>> ThreadPool implements the com.sun.jini.thread.Executor interface.
>>
>> Because the interface states it should not block, I think the calling thread should execute the task when the thread pool becomes saturated, rather than continue to create new threads as per the current implementation.  This will ensure that the task is completed, the calling thread, if it uses a sequence of tasks with dependencies will have to ensure that it submits the tasks in order.  Since only the caller knows the order, it makes sense for this to be the callers responsibility.  As a result the executor will honor the non blocking contract.  For that reason we'll use a zero length queue, probably a SynchronousQueue.
>>
>> Regards,
>>
>> Peter.
>>
>> /**
>> * Executor is an abstraction for a thread factory or thread pool for
>> * executing actions asynchronously.
>> *
>> * @author    Sun Microsystems, Inc.
>> *
>> */
>> public interface Executor {
>>
>>   /**
>>    * Executes the given Runnable action asynchronously in some thread.
>>    *
>>    * The implementation may create a new thread to execute the action,
>>    * or it may execute the action in an existing thread.
>>    *
>>    * The execution of a given action must not be delayed indefinitely
>>    * in order to complete execution of a different action passed to a
>>    * different invocation of this method.  In other words, the
>>    * implementation must assume that there may be arbitrary dependencies
>>    * between actions passed to this method, so it needs to be careful
>>    * to avoid potential deadlock by delaying execution of one action
>>    * indefinitely until another completes.
>>    *
>>    * Also, this method itself must not block, because it may be invoked
>>    * by code that is serially processing data to produce multiple such
>>    * arbitrarily-dependent actions that need to be executed.
>>    *
>>    * @param    runnable the Runnable action to execute
>>    *
>>    * @param    name string to include in the name of the thread used
>>    * to execute the action
>>    */
>>   void execute(Runnable runnable, String name);
>> }
>>
>> Peter Firmstone wrote:
>>     
>>> Thanks Brian,
>>>
>>> Looking at our implementation code, DGC uses an Executor called ThreadPool, it's javadoc states:
>>>
>>> /**
>>> * ThreadPool is a simple thread pool implementation of the Executor
>>> * interface.
>>> *
>>> * A new task is always given to an idle thread, if one is available;
>>> * otherwise, a new thread is always created.  There is no minimum
>>> * warm thread count, nor is there a maximum thread count (tasks are
>>> * never queued unless there are sufficient idle threads to execute
>>> * them).
>>> *
>>> * New threads are created as daemon threads in the thread group that
>>> * was passed to the ThreadPool instance's constructor.  Each thread's
>>> * name is the prefix NewThreadAction.NAME_PREFIX followed by the name
>>> * of the task it is currently executing, or "Idle" if it is currently
>>> * idle.
>>>
>>> ThreadPool predates Java 5, it looks like we can fix this by using an Executor from Java 5, we can look at limiting the number of threads created based on available CPU's and a scaling factor and place the tasks in a BlockingQueue, so if the queue is filled, it blocks.
>>>
>>> Can you report the issue as a Bug on Jira for me, I'll fix this before the next release.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> Peter Firmstone wrote:
>>>       
>>>> Hi Peter,
>>>>
>>>> I was wondering if you had any thoughts on this post from Bryan on River users?
>>>>
>>>> Hope you don't mind me asking ;)
>>>>
>>>> Best Regards,
>>>>
>>>> Peter Firmstone.
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> Subject:
>>>> Re: DGC threads issue
>>>> From:
>>>> Tom Hobbs <tv...@googlemail.com>
>>>> Date:
>>>> Thu, 12 Jan 2012 20:45:01 +0000
>>>> To:
>>>> user@river.apache.org, dev@river.apache.org
>>>>
>>>> To:
>>>> user@river.apache.org, dev@river.apache.org
>>>>
>>>>
>>>> Hi Bryan,
>>>>
>>>> Sorry that no one got back to you about this.  I'm afraid that I don't
>>>> know the answer to your question, I've copied the dev list into this
>>>> email in case someone who monitors that list (but not this one) has
>>>> any ideas.
>>>>
>>>> Best regards,
>>>>
>>>> Tom
>>>>
>>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson <br...@systap.com> wrote:
>>>>
>>>>         
>>>>> Just to follow up on this thread myself.  I modified the pattern to return a "thick" future rather than a proxy for the future.  This caused the RMI call to wait on the server until the future was done and then sent back the outcome.  This "fixed" the DGC memory/thread leak by reducing the number of exported proxies drammatically.
>>>>>
>>>>> In terms of best practices, is distributed DGC simply not useful for exported objects with short life spans?  Can it only be used with proxies for relatively long lived services?
>>>>>
>>>>> Thanks,
>>>>> Bryan
>>>>>
>>>>>   
>>>>>           
>>>>>> -----Original Message-----
>>>>>> From: Bryan Thompson
>>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
>>>>>> To: user@river.apache.org
>>>>>> Subject: DGC threads issue
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Background:
>>>>>>
>>>>>> I am seeing what would appear to be one DGC thread allocated
>>>>>> per exported object.  This is using River 2.2 and Sun JDK
>>>>>> 1.6.0_17.  Relevant configuration parameters are below.
>>>>>>
>>>>>> I am observing problems with the DGC threads not being
>>>>>> retired on a timely basis.  The exported objects are proxies
>>>>>> for Futures which are being executed on the service.  The
>>>>>> code pattern is such that the proxied Future goes out of
>>>>>> lexical scope quite quickly.  E.g.,
>>>>>> rmiCallReturningProxyForFuture().get().
>>>>>>
>>>>>> Under a modest load, a large number of such Futures are
>>>>>> exported which results in a large number of long lived DGC
>>>>>> threads.  This turns into a problem for the JVM due to the
>>>>>> stack allocation per thread.  Presumably this is not good for
>>>>>> other reasons as well (e.g., scheduling).
>>>>>>
>>>>>> I have tried to override the leaseValue and checkInterval
>>>>>> defaults per the configuration options below.  I suspect that
>>>>>> the lease interval is somehow not being obeyed, which is
>>>>>> presumably a problem on my end.  However, I can verify that
>>>>>> the configuration values are in fact showing up in
>>>>>> System.getProperties() for at least some of the JVMs involved
>>>>>> (the one which drives the workload and the one that I am
>>>>>> monitoring with the large number of DGC lease threads).
>>>>>>
>>>>>> Some questions:
>>>>>>
>>>>>> Is this one-thread-per-exported proxy the expected behavior
>>>>>> when DGC is requested for the exported object?
>>>>>>
>>>>>> The DGC lease checker threads appear to expire ~14 - 15
>>>>>> minutes after I terminate the process which was originating
>>>>>> the RMI requests.  This is close the sum of the default
>>>>>> leaseValue (10m) and checkInterval (5m) parameters, but maybe
>>>>>> there is some other timeout which is controlling this?  If
>>>>>> this is the sum of those parameters, why would the DGC lease
>>>>>> threads live until the sum of those values?  I thought that
>>>>>> the lease would expire after the leaseValue (10m default).
>>>>>>
>>>>>> Can the issue I am observing be caused by a low heap pressure
>>>>>> on the JVM to which the RMI proxies were exported?  If it
>>>>>> fails to GC those proxies, even though they are reachable,
>>>>>> could that cause DGC to continue to retain those proxies on
>>>>>> the JVM which exported them?
>>>>>>
>>>>>> Is there any way to configure DGC to use a thread pool or to
>>>>>> have the leases managed by a single thread?
>>>>>>
>>>>>> Is it possible that there is an interaction with the useNIO option?
>>>>>>
>>>>>> Relevant options that I am using include:
>>>>>>
>>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
>>>>>>   -Djava.rmi.dgc.leaseValue=30000
>>>>>>   -Dsun.rmi.dgc.checkInterval=15000
>>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Bryan
>>>>>>      
>>>>>>             
>>>>  
>>>>         
>>>       
>
>
>