You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Gregg Wonderly <gr...@cytetech.com> on 2010/06/04 17:00:04 UTC

com.sun.jini.thread lock contention

The TaskManager class has a subclass of Thread that it uses to run tasks.  The 
problem with subclassing Thread, is that there is a lock that is used in Thread 
to see if some methods are overridden so that security can be maintained.  So, 
in an environment where Thread is used with a SecurityManager, it is better to 
use a Runnable to encapsulate the custom code, instead of a Thread.

The TaskThread class should be a Runnable instead of a Thread, and then there 
are just 3 other places where it is referenced that need some adjustment.  The 
thread that is running inside of the runnable needs to be put into a per 
instance variable for one place where interrupt processing is handled.

Gregg Wonderly

RE: com.sun.jini.thread lock contention

Posted by Christopher Dolan <ch...@avid.com>.
Attached is a minimalist initial patch.  I'm not happy that I had to
store a reference to the thread in the Runnable...  But otherwise I
wasn't sure how to interrupt it from removeTask() without a separate map
of TaskThread to Thread.  There's all a null pointer race in this patch
if you try to remove a task before it's run() method executes.

Chris

Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Well it looks like the Hudson test that failed had nothing to do with 
TaskThread.

Keep you posted after I run some more tests.

Cheers,

Peter.

This is the test that failed:


[java] com/sun/jini/test/impl/discoverymanager/RemoveGroupsLocsDiscard.td

     [java] Test Failed: Setup Failed: com.sun.jini.qa.harness.TestException: Problem creating service for net.jini.core.lookup.ServiceRegistrar; nested exception is: 
     [java] 	RemoteException occurred in server thread; nested exception is: 
     [java] 	java.rmi.RemoteException: Create failed; nested exception is: 
     [java] 	java.lang.reflect.InvocationTargetException

     [java] TIME: 4:08:02 PM
     [java] 
     [java] Resolver.setToken FINEST: setting token <config> to none
     [java] QAConfig.loadTestConfiguration FINER: Test Configuration options:
     [java] QAConfig.loadTestConfiguration FINER:    -
     [java] QAConfig.loadTestConfiguration FINER:    net.jini.discovery.LookupDiscovery.multicastAnnouncementInterval = 20000
     [java] QAConfig.loadTestConfiguration FINER:    multicast.ttl = 0
     [java] Running com/sun/jini/test/impl/discoverymanager/RemoveGroupsLocsDiscard.td
     [java] Time is Tue Jun 08 16:08:02 UTC 2010
     [java] Starting test in separate process with command:
     [java] /home/hudson/tools/java/jdk1.6.0_20-32/jre/bin/java -Djava.security.policy=file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/harness/policy/defaulttest.policy -Djava.rmi.server.codebase=http://minerva.apache.org:8082/qa1-share-dl.jar -cp /home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jiniharness.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jinitests.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/jsk-platform.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/jsk-lib.jar -client -Djava.ext.dirs=/home/hudson/tools/java/jdk1.6.0_20-32/jre/lib/ext:/usr/java/packages/lib/ext:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib-ext:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib-ext -Dcom.sun.jini.jsk.port=8080 -Dcom.sun.jini.qa.port=8081 -Dcom.sun.jini.jsk.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk -Dcom.sun.jini.qa.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa -Dcom.sun.jini.qa.harness.harnessJar=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jiniharness.jar -Dcom.sun.jini.qa.harness.testJar=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jinitests.jar -Dcom.sun.jini.qa.harness.runjiniserver=true -Dcom.sun.jini.qa.harness.runkitserver=true -Djava.security.properties=file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/harness/trust/dynamic-policy.properties -Djava.util.logging.config.file=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/src/com/sun/jini/test/resources/qa1.logging -Dcom.sun.jini.test.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa -Dcom.sun.jini.test.port=8082 -Dcom.sun.jini.qa.harness.policies=file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/src/com/sun/jini/test/resources/jinitest.policy -Djava.ext.dirs=/home/hudson/tools/java/jdk1.6.0_20-32/jre/lib/ext:/usr/java/packages/lib/ext:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib-ext:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib-ext com.sun.jini.qa.harness.MasterTest com/sun/jini/test/impl/discoverymanager/RemoveGroupsLocsDiscard.td 
     [java] Jun 8, 2010 4:08:02 PM com.sun.jini.qa.harness.MasterTest main
     [java] FINE: Starting MasterTest
     [java] 
     [java] TIME: 4:08:03 PM
     [java] 
     [java] QAConfig.loadTestConfiguration FINER: Test Configuration options:
     [java] QAConfig.loadTestConfiguration FINER:    -
     [java] QAConfig.loadTestConfiguration FINER:    net.jini.discovery.LookupDiscovery.multicastAnnouncementInterval = 20000
     [java] QAConfig.loadTestConfiguration FINER:    multicast.ttl = 0
     [java] MasterTest.doTest INFO: 
     [java] ============================== CALLING SETUP() ==============================
     [java] 
     [java] AdminManager.startService FINE: starting qaClassServer
     [java] AdminManager.getAdmin FINEST: getAdmin called with prefix qaClassServer
     [java] FINE: 
     [java] FINE: Parameters for qaClassServer(.0):
     [java] FINE:      type              : classServer
     [java] FINE:      impl              : com.sun.jini.tool.ClassServer
     [java] FINE:      directory         : /home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib
     [java] FINE:      options           : 
     [java] Jun 8, 2010 4:08:03 PM com.sun.jini.tool.ClassServer run
     [java] INFO: ClassServer started [[/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/], port 8081]
     [java] AdminManager.startService FINE: starting jiniClassServer
     [java] AdminManager.getAdmin FINEST: getAdmin called with prefix jiniClassServer
     [java] FINE: 
     [java] FINE: Parameters for jiniClassServer(.0):
     [java] FINE:      type              : classServer
     [java] FINE:      impl              : com.sun.jini.tool.ClassServer
     [java] FINE:      directory         : /home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib-dl
     [java] FINE:      options           : 
     [java] AdminManager.startService FINE: starting testClassServer
     [java] Jun 8, 2010 4:08:03 PM com.sun.jini.tool.ClassServer run
     [java] INFO: ClassServer started [[/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib-dl/], port 8080]
     [java] AdminManager.getAdmin FINEST: getAdmin called with prefix testClassServer
     [java] FINE: 
     [java] FINE: Parameters for testClassServer(.0):
     [java] FINE:      type              : classServer
     [java] FINE:      impl              : com.sun.jini.tool.ClassServer
     [java] FINE:      directory         : /home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib
     [java] FINE:      options           : 
     [java] BaseQATest.setup FINE:  setup()
     [java] Jun 8, 2010 4:08:03 PM com.sun.jini.tool.ClassServer run
     [java] INFO: ClassServer started [[/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/], port 8082]
     [java] BaseQATest.getSetupInfo FINE:  ----- Harness Info ----- 
     [java] BaseQATest.getSetupInfo FINE:  harness codebase         -- http://minerva.apache.org:8082/qa1-share-dl.jar
     [java] BaseQATest.getSetupInfo FINE:  harness classpath        -- /home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jiniharness.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jinitests.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/jsk-platform.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/jsk-lib.jar
     [java] BaseQATest.getSetupInfo FINE:  net.jini.discovery.debug        -- false
     [java] BaseQATest.getSetupInfo FINE:  com.sun.jini.reggie.proxy.debug -- false
     [java] BaseQATest.getSetupInfo FINE:  com.sun.jini.join.debug         -- false
     [java] BaseQATest.getSetupInfo FINE:  com.sun.jini.sdm.debug          -- false
     [java] BaseQATest.getSetupInfo FINE:  max secs event wait             -- 180
     [java] BaseQATest.getSetupInfo FINE:  ----- Lookup Service Info ----- 
     [java] BaseQATest.getSetupInfo FINE:  # of lookup services to start            -- 1
     [java] BaseQATest.getSetupInfo FINE:  # of additional lookup services to start -- 0
     [java] BaseQATest.getSetupInfo FINE:  seconds to wait for discovery            -- 10
     [java] BaseQATest.getSetupInfo FINE:  discard if no announcements in (nSecs =) -- 20
     [java] QAConfig.getServiceHost FINE: Selecting service host
     [java] QAConfig.getServiceHost FINE: Not distributed - selecting this host
     [java] BaseQATest.getTestLocator FINER: getServiceHost returned null
     [java] BaseQATest.startLookup FINE:  starting lookup service 0
     [java] AdminManager.startService FINE: starting net.jini.core.lookup.ServiceRegistrar
     [java] AdminManager.getAdmin FINEST: getAdmin called with prefix net.jini.core.lookup.ServiceRegistrar
     [java] QAConfig.getServiceHost FINE: Selecting service host
     [java] QAConfig.getServiceHost FINE: Not distributed - selecting this host
     [java] AbstractServiceAdmin.addServiceExporter FINER: no exporter definition provided
     [java] NonActivatableServiceStarterAdmin.getGroup FINER: Creating shared group
     [java] AdminManager.startService FINE: starting nonActivatableGroup
     [java] AdminManager.getAdmin FINEST: getAdmin called with prefix nonActivatableGroup
     [java] NonActivatableGroupAdmin.start FINER: NonActivatableGroup exec command line: '/home/hudson/tools/java/jdk1.6.0_20-32/jre/bin/java -Djava.rmi.server.codebase=http://minerva.apache.org:8081/nonactivatablegroup-dl.jar -Djava.security.policy=file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/harness/policy/defaultnonactvm.policy -server -Djava.ext.dirs=/home/hudson/tools/java/jdk1.6.0_20-32/jre/lib/ext:/usr/java/packages/lib/ext:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib-ext:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib-ext -Dcom.sun.jini.jsk.port=8080 -Dcom.sun.jini.qa.port=8081 -Dcom.sun.jini.jsk.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk -Dcom.sun.jini.qa.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa -Dcom.sun.jini.qa.harness.harnessJar=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jiniharness.jar -Dcom.sun.jini.qa.harness.testJar=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jinitests.jar -Dcom.sun.jini.qa.harness.runjiniserver=true -Dcom.sun.jini.qa.harness.runkitserver=true -Djava.security.properties=file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/harness/trust/dynamic-policy.properties -Djava.util.logging.config.file=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/src/com/sun/jini/test/resources/qa1.logging -Dcom.sun.jini.test.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa -Dcom.sun.jini.test.port=8082 -Dcom.sun.jini.qa.harness.policies=file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/src/com/sun/jini/test/resources/jinitest.policy -cp /home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/nonactivatablegroup.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/start.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/jsk-platform.jar com.sun.jini.qa.harness.NonActivatableGroupImpl'
     [java] FINE: 
     [java] FINE: Parameters for nonActivatableGroup(.0):
     [java] FINE:      type              : nonactivatablegroup
     [java] FINE:      codebase          : http://minerva.apache.org:8081/nonactivatablegroup-dl.jar
     [java] FINE:      impl              : com.sun.jini.qa.harness.NonActivatableGroupImpl
     [java] FINE:      policy file       : file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/harness/policy/defaultnonactvm.policy
     [java] FINE:      classpath         : /home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/nonactivatablegroup.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/start.jar:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/jsk-platform.jar
     [java] FINE:      options           : -server
     [java] FINE:      properties        : -Djava.ext.dirs=/home/hudson/tools/java/jdk1.6.0_20-32/jre/lib/ext:/usr/java/packages/lib/ext:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib-ext:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib-ext
     [java] FINE:                        : -Dcom.sun.jini.jsk.port=8080
     [java] FINE:                        : -Dcom.sun.jini.qa.port=8081
     [java] FINE:                        : -Dcom.sun.jini.jsk.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk
     [java] FINE:                        : -Dcom.sun.jini.qa.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa
     [java] FINE:                        : -Dcom.sun.jini.qa.harness.harnessJar=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jiniharness.jar
     [java] FINE:                        : -Dcom.sun.jini.qa.harness.testJar=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/lib/jinitests.jar
     [java] FINE:                        : -Dcom.sun.jini.qa.harness.runjiniserver=true
     [java] FINE:                        : -Dcom.sun.jini.qa.harness.runkitserver=true
     [java] FINE:                        : -Djava.security.properties=file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/harness/trust/dynamic-policy.properties
     [java] FINE:                        : -Djava.util.logging.config.file=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/src/com/sun/jini/test/resources/qa1.logging
     [java] FINE:                        : -Dcom.sun.jini.test.home=/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa
     [java] FINE:                        : -Dcom.sun.jini.test.port=8082
     [java] FINE:                        : -Dcom.sun.jini.qa.harness.policies=file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/src/com/sun/jini/test/resources/jinitest.policy
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPEndpoint <clinit>
     [java] NonActGrp-out: FINE: main: localHostKnown = true, localHost = 67.195.138.8
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport <init>
     [java] NonActGrp-out: FINE: main: Version = 2, ep = [67.195.138.8:0]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
     [java] NonActGrp-out: FINE: main: created local endpoint for socket factory null on port 0
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport listen
     [java] NonActGrp-out: FINE: main: (port 0) create server socket
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPEndpoint newServerSocket
     [java] NonActGrp-out: FINER: main: creating server socket on [67.195.138.8:0]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPEndpoint setDefaultPort
     [java] NonActGrp-out: FINE: main: default port for server socket factory null and client socket factory null set to 48290
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
     [java] NonActGrp-out: FINE: RMI TCP Accept-0: listening on port 48290
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.WeakRef pin
     [java] NonActGrp-out: FINER: main: strongRef = sun.rmi.transport.DGCImpl@fe748f
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.ObjectTable putTarget
     [java] NonActGrp-out: FINER: main: add object [0:0:0, 2]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.ObjectTable putTarget
     [java] NonActGrp-out: FINER: main: add object [-3b9058b:1291851876c:-7fff, 5108795070014869142]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.server.Util computeMethodHash
     [java] NonActGrp-out: FINER: main: string used for method hash: "stop()V"
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.server.Util computeMethodHash
     [java] NonActGrp-out: FINER: main: string used for method hash: "startService(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;Lcom/sun/jini/qa/harness/ServiceDescriptorTransformer;)Ljava/lang/Object;"
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl_Stub", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM sun.net.www.protocol.http.HttpURLConnection writeRequests
     [java] FINE: sun.net.www.MessageHeader@119298d5 pairs: {GET /nonactivatablegroup-dl.jar HTTP/1.1: null}{User-Agent: Java/1.6.0_20}{Host: minerva.apache.org:8081}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
     [java] Jun 8, 2010 4:08:03 PM com.sun.jini.tool.ClassServer$Task run
     [java] FINER: nonactivatablegroup-dl.jar requested from minerva.apache.org:49261
     [java] Jun 8, 2010 4:08:03 PM sun.net.www.protocol.http.HttpURLConnection getInputStream
     [java] FINE: sun.net.www.MessageHeader@f726173 pairs: {null: HTTP/1.0 200 OK}{Content-Length: 3429}{Content-Type: application/java}
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl_Stub" found via defaultLoader, defined by sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.rmi.server.RemoteStub", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.rmi.server.RemoteStub" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.rmi.server.RemoteObject", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.rmi.server.RemoteObject" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPEndpoint <clinit>
     [java] FINE: main: localHostKnown = true, localHost = 67.195.138.8
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef newCall
     [java] FINE: main: get connection
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport <init>
     [java] FINE: main: Version = 2, ep = [67.195.138.8:0]
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
     [java] FINE: main: created local endpoint for socket factory null on port 0
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel createConnection
     [java] FINE: main: create connection
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPEndpoint newSocket
     [java] FINER: main: opening socket to [67.195.138.8:48290]
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
     [java] FINE: main: host: 67.195.138.8, port: 48290
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport$ConnectionHandler run0
     [java] NonActGrp-out: FINE: RMI TCP Connection(1)-67.195.138.8: accepted socket from [67.195.138.8:33816]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport$ConnectionHandler run0
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: (port 48290) suggesting 67.195.138.8:33816
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel createConnection
     [java] FINER: main: server suggested 67.195.138.8:33816
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel createConnection
     [java] FINER: main: using 67.195.138.8:0
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport$ConnectionHandler run0
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: (port 48290) client using 67.195.138.8:0
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef newCall
     [java] FINER: main: create call context
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef logClientCall
     [java] FINER: main: outbound call: [endpoint:[67.195.138.8:48290](remote),objID:[0:0:0, 2]] : sun.rmi.transport.DGCImpl_Stub[0:0:0, 2]: java.rmi.dgc.Lease dirty(java.rmi.server.ObjID[], long, java.rmi.dgc.Lease)
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall <init>
     [java] FINER: main: write remote call header...
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getOutputStream
     [java] FINER: main: getting output stream
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef invoke
     [java] FINER: main: execute call
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport handleMessages
     [java] NonActGrp-out: FINE: RMI TCP Connection(1)-67.195.138.8: (port 48290) op = 80
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getInputStream
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: getting input stream
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.Transport serviceCall
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: call dispatcher
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastServerRef logCall
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: [67.195.138.8: sun.rmi.transport.DGCImpl[0:0:0, 2]: java.rmi.dgc.Lease dirty(java.rmi.server.ObjID[], long, java.rmi.dgc.Lease)]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINE: name="[Ljava.rmi.server.ObjID;", codebase="http://minerva.apache.org:8082/qa1-share-dl.jar", defaultLoader=null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@16f0472)
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.net.www.protocol.http.HttpURLConnection writeRequests
     [java] NonActGrp-out: FINE: sun.net.www.MessageHeader@b01d435 pairs: {GET /qa1-share-dl.jar HTTP/1.1: null}{User-Agent: Java/1.6.0_20}{Host: minerva.apache.org:8082}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
     [java] Jun 8, 2010 4:08:03 PM com.sun.jini.tool.ClassServer$Task run
     [java] FINER: qa1-share-dl.jar requested from minerva.apache.org:51805
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.net.www.protocol.http.HttpURLConnection getInputStream
     [java] NonActGrp-out: FINE: sun.net.www.MessageHeader@177b3cd3 pairs: {null: HTTP/1.0 200 OK}{Content-Length: 69656}{Content-Type: application/java}
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: class "[Ljava.rmi.server.ObjID;" found via codebase loader, defined by null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINE: name="java.rmi.server.ObjID", codebase="http://minerva.apache.org:8082/qa1-share-dl.jar", defaultLoader=null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@16f0472)
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getInputStream
     [java] FINER: main: getting input stream
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: class "java.rmi.server.ObjID" found via codebase loader, defined by null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINE: name="java.rmi.server.UID", codebase="http://minerva.apache.org:8082/qa1-share-dl.jar", defaultLoader=null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@16f0472)
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: class "java.rmi.server.UID" found via codebase loader, defined by null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINE: name="java.rmi.dgc.Lease", codebase="http://minerva.apache.org:8082/qa1-share-dl.jar", defaultLoader=null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.rmi.dgc.Lease", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@16f0472)
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: class "java.rmi.dgc.Lease" found via codebase loader, defined by null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINE: name="java.rmi.dgc.VMID", codebase="http://minerva.apache.org:8082/qa1-share-dl.jar", defaultLoader=null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@16f0472)
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: class "java.rmi.dgc.VMID" found via codebase loader, defined by null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINE: name="[B", codebase=null, defaultLoader=null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.DGCImpl dirty
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: vmid = dbf17c0820afd71b:12c6d08f:12918518845:-7fff
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.DGCImpl dirty
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: id = [-3b9058b:1291851876c:-7fff, 5108795070014869142], vmid = dbf17c0820afd71b:12c6d08f:12918518845:-7fff, duration = 600000
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.WeakRef pin
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: strongRef = com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl@1592174
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.Target referenced
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: add to dirty set: dbf17c0820afd71b:12c6d08f:12918518845:-7fff
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getOutputStream
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: getting output stream
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.rmi.dgc.Lease" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.rmi.dgc.VMID", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.rmi.dgc.VMID" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="[B", codebase=null, defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.rmi.server.UID", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.rmi.server.UID" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef done
     [java] FINE: main: free connection (reuse = true)
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel free
     [java] FINE: main: reuse connection
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel free
     [java] FINE: main: create reaper
     [java] FINE: 
     [java] FINE: Parameters for net.jini.core.lookup.ServiceRegistrar(.0):
     [java] FINE:      type              : transient
     [java] FINE:      codebase          : http://minerva.apache.org:8080/reggie-dl.jar http://minerva.apache.org:8080/jsk-dl.jar
     [java] FINE:      impl              : com.sun.jini.reggie.TransientRegistrarImpl
     [java] FINE:      component name    : com.sun.jini.reggie
     [java] FINE:      policy file       : file:/home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/qa/harness/policy/defaultreggie.policy
     [java] FINE:      classpath         : /home/hudson/hudson-slave/workspace/River-trunk/jtsk/trunk/lib/reggie.jar
     [java] FINE:      service conf file : -
     [java] FINE:      starter conf file : -
     [java] FINE:      proxy preparer    : test.reggiePreparer
     [java] FINE:      option args 0     : -
     [java] FINE:      option args 1     : com.sun.jini.reggie.initialMemberGroups = new String[]{"LDMGroup0_A_minerva.apache.org_1276013282827", "LDMGroup0_B_minerva.apache.org_1276013282827", "LDMGroup0_C_minerva.apache.org_1276013282827"}
     [java] FINE:      option args 2     : com.sun.jini.reggie.initialUnicastDiscoveryPort=8000
     [java] FINE:      option args 3     : net.jini.discovery.LookupDiscovery.multicastAnnouncementInterval=20000
     [java] FINE:      option args 4     : com.sun.jini.reggie.multicastAnnouncementInterval=20000
     [java] FINE:      option args 5     : multicast.ttl=0
     [java] FINE: 
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef invoke
     [java] FINER: main: method: public abstract java.lang.Object com.sun.jini.qa.harness.NonActivatableGroup.startService(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String[],java.lang.String,com.sun.jini.qa.harness.ServiceDescriptorTransformer) throws java.rmi.RemoteException
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef logClientCall
     [java] FINER: main: outbound call: [endpoint:[67.195.138.8:48290](remote),objID:[-3b9058b:1291851876c:-7fff, 5108795070014869142]] : com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl_Stub[-3b9058b:1291851876c:-7fff, 5108795070014869142]: public abstract java.lang.Object com.sun.jini.qa.harness.NonActivatableGroup.startService(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String[],java.lang.String,com.sun.jini.qa.harness.ServiceDescriptorTransformer) throws java.rmi.RemoteException
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport handleMessages
     [java] NonActGrp-out: FINE: RMI TCP Connection(1)-67.195.138.8: (port 48290) op = 82
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel newConnection
     [java] FINE: main: reuse connection
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef invoke
     [java] FINER: main: opnum = -3440598291276346209
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall <init>
     [java] FINER: main: write remote call header...
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getOutputStream
     [java] FINER: main: getting output stream
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport handleMessages
     [java] NonActGrp-out: FINE: RMI TCP Connection(1)-67.195.138.8: (port 48290) op = 80
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getInputStream
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: getting input stream
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.Transport serviceCall
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: call dispatcher
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastServerRef logCall
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: [67.195.138.8: com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl[-3b9058b:1291851876c:-7fff, 5108795070014869142]: public abstract java.lang.Object com.sun.jini.qa.harness.NonActivatableGroup.startService(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String[],java.lang.String,com.sun.jini.qa.harness.ServiceDescriptorTransformer) throws java.rmi.RemoteException]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINE: name="[Ljava.lang.String;", codebase="http://minerva.apache.org:8082/qa1-share-dl.jar", defaultLoader=null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@16f0472)
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] NonActGrp-out: FINEST: class "[Ljava.lang.String;" found via codebase loader, defined by null
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.net.www.protocol.http.HttpURLConnection writeRequests
     [java] NonActGrp-out: FINE: sun.net.www.MessageHeader@19836ed5 pairs: {GET /reggie-dl.jar HTTP/1.1: null}{User-Agent: Java/1.6.0_20}{Host: minerva.apache.org:8080}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
     [java] Jun 8, 2010 4:08:03 PM com.sun.jini.tool.ClassServer$Task run
     [java] FINER: reggie-dl.jar requested from minerva.apache.org:42569
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.net.www.protocol.http.HttpURLConnection getInputStream
     [java] NonActGrp-out: FINE: sun.net.www.MessageHeader@3e0ebb3 pairs: {null: HTTP/1.0 200 OK}{Content-Length: 50827}{Content-Type: application/java}
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.net.www.protocol.http.HttpURLConnection writeRequests
     [java] NonActGrp-out: FINE: sun.net.www.MessageHeader@39443f5 pairs: {GET /jsk-dl.jar HTTP/1.1: null}{User-Agent: Java/1.6.0_20}{Host: minerva.apache.org:8080}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
     [java] Jun 8, 2010 4:08:03 PM com.sun.jini.tool.ClassServer$Task run
     [java] FINER: jsk-dl.jar requested from minerva.apache.org:42570
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.net.www.protocol.http.HttpURLConnection getInputStream
     [java] NonActGrp-out: FINE: sun.net.www.MessageHeader@1afae453 pairs: {null: HTTP/1.0 200 OK}{Content-Length: 57298}{Content-Type: application/java}
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM com.sun.jini.reggie.RegistrarImpl <init>
     [java] NonActGrp-out: SEVERE: Reggie initialization failed
     [java] NonActGrp-out: java.net.BindException: Address already in use
     [java] NonActGrp-out: 	at java.net.PlainSocketImpl.socketBind(Native Method)
     [java] NonActGrp-out: 	at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
     [java] NonActGrp-out: 	at java.net.ServerSocket.bind(ServerSocket.java:319)
     [java] NonActGrp-out: 	at java.net.ServerSocket.<init>(ServerSocket.java:185)
     [java] NonActGrp-out: 	at java.net.ServerSocket.<init>(ServerSocket.java:97)
     [java] NonActGrp-out: 	at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:157)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl$UnicastThread.<init>(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl.init(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl.access$000(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl$1.run(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl.<init>(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.TransientRegistrarImpl.<init>(Unknown Source)
     [java] NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
     [java] NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
     [java] NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
     [java] NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
     [java] NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(Unknown Source)
     [java] NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java] NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java] NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:597)
     [java] NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
     [java] NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:159)
     [java] NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
     [java] NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
     [java] NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
     [java] NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
     [java] NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
     [java] NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java] NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java] NonActGrp-out: 	at java.lang.Thread.run(Thread.java:619)
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastServerRef logCallException
     [java] NonActGrp-out: FINE: RMI TCP Connection(1)-67.195.138.8: [67.195.138.8] exception: 
     [java] NonActGrp-out: java.rmi.RemoteException: Create failed; nested exception is: 
     [java] NonActGrp-out: 	java.lang.reflect.InvocationTargetException
     [java] NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(Unknown Source)
     [java] NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java] NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java] NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:597)
     [java] NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
     [java] NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:159)
     [java] NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
     [java] NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
     [java] NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
     [java] NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
     [java] NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
     [java] NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java] NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java] NonActGrp-out: 	at java.lang.Thread.run(Thread.java:619)
     [java] NonActGrp-out: Caused by: java.lang.reflect.InvocationTargetException
     [java] NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
     [java] NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
     [java] NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
     [java] NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
     [java] NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(Unknown Source)
     [java] NonActGrp-out: 	... 15 more
     [java] NonActGrp-out: Caused by: java.net.BindException: Address already in use
     [java] NonActGrp-out: 	at java.net.PlainSocketImpl.socketBind(Native Method)
     [java] NonActGrp-out: 	at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
     [java] NonActGrp-out: 	at java.net.ServerSocket.bind(ServerSocket.java:319)
     [java] NonActGrp-out: 	at java.net.ServerSocket.<init>(ServerSocket.java:185)
     [java] NonActGrp-out: 	at java.net.ServerSocket.<init>(ServerSocket.java:97)
     [java] NonActGrp-out: 	at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:157)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl$UnicastThread.<init>(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl.init(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl.access$000(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl$1.run(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.RegistrarImpl.<init>(Unknown Source)
     [java] NonActGrp-out: 	at com.sun.jini.reggie.TransientRegistrarImpl.<init>(Unknown Source)
     [java] NonActGrp-out: 	... 20 more
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getOutputStream
     [java] NonActGrp-out: FINER: RMI TCP Connection(1)-67.195.138.8: getting output stream
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getInputStream
     [java] FINER: main: getting input stream
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.rmi.ServerException", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.rmi.ServerException" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.rmi.RemoteException", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.rmi.RemoteException" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.io.IOException", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.io.IOException" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.lang.Exception", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.lang.Exception" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.lang.Throwable", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.lang.Throwable" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="[Ljava.lang.StackTraceElement;", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "[Ljava.lang.StackTraceElement;" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.lang.StackTraceElement", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.lang.StackTraceElement" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.lang.reflect.InvocationTargetException", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.lang.reflect.InvocationTargetException" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.net.BindException", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.net.BindException" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINE: name="java.net.SocketException", codebase="http://minerva.apache.org:8081/nonactivatablegroup-dl.jar", defaultLoader=sun.misc.Launcher$AppClassLoader@7d772e
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: (thread context class loader: sun.misc.Launcher$AppClassLoader@7d772e)
     [java] Jun 8, 2010 4:08:03 PM net.jini.loader.pref.PreferredClassProvider loadClass
     [java] FINEST: class "java.net.SocketException" found via defaultLoader, defined by null
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall exceptionReceivedFromServer
     [java] FINE: main: outbound call received exception: [67.195.138.8:48290] exception: 
     [java] java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
     [java] 	java.rmi.RemoteException: Create failed; nested exception is: 
     [java] 	java.lang.reflect.InvocationTargetException
     [java] 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:336)
     [java] 	at sun.rmi.transport.Transport$1.run(Transport.java:159)
     [java] 	at java.security.AccessController.doPrivileged(Native Method)
     [java] 	at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
     [java] 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
     [java] 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
     [java] 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java] 	at java.lang.Thread.run(Thread.java:619)
     [java] 	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
     [java] 	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
     [java] 	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
     [java] 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl_Stub.startService(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.NonActivatableServiceStarterAdmin.start(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.AdminManager.startService(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.AdminManager.startLookupService(Unknown Source)
     [java] 	at com.sun.jini.test.share.BaseQATest.startLookup(Unknown Source)
     [java] 	at com.sun.jini.test.share.BaseQATest.startInitLookups(Unknown Source)
     [java] 	at com.sun.jini.test.share.BaseQATest.setup(Unknown Source)
     [java] 	at com.sun.jini.test.spec.discoverymanager.AbstractBaseTest.setup(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.MasterTest.doTest(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.MasterTest.main(Unknown Source)
     [java] Caused by: java.rmi.RemoteException: Create failed; nested exception is: 
     [java] 	java.lang.reflect.InvocationTargetException
     [java] 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(Unknown Source)
     [java] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java] 	at java.lang.reflect.Method.invoke(Method.java:597)
     [java] 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
     [java] 	at sun.rmi.transport.Transport$1.run(Transport.java:159)
     [java] 	at java.security.AccessController.doPrivileged(Native Method)
     [java] 	at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
     [java] 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
     [java] 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
     [java] 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java] 	at java.lang.Thread.run(Thread.java:619)
     [java] Caused by: java.lang.reflect.InvocationTargetException
     [java] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
     [java] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
     [java] 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
     [java] 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
     [java] 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(Unknown Source)
     [java] 	... 15 more
     [java] Caused by: java.net.BindException: Address already in use
     [java] 	at java.net.PlainSocketImpl.socketBind(Native Method)
     [java] 	at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
     [java] 	at java.net.ServerSocket.bind(ServerSocket.java:319)
     [java] 	at java.net.ServerSocket.<init>(ServerSocket.java:185)
     [java] 	at java.net.ServerSocket.<init>(ServerSocket.java:97)
     [java] 	at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:157)
     [java] 	at com.sun.jini.reggie.RegistrarImpl$UnicastThread.<init>(Unknown Source)
     [java] 	at com.sun.jini.reggie.RegistrarImpl.init(Unknown Source)
     [java] 	at com.sun.jini.reggie.RegistrarImpl.access$000(Unknown Source)
     [java] 	at com.sun.jini.reggie.RegistrarImpl$1.run(Unknown Source)
     [java] 	at com.sun.jini.reggie.RegistrarImpl.<init>(Unknown Source)
     [java] 	at com.sun.jini.reggie.TransientRegistrarImpl.<init>(Unknown Source)
     [java] 	... 20 more
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef invoke
     [java] FINE: main: free connection (reuse = false)
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel free
     [java] FINE: main: close connection
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPConnection close
     [java] FINE: main: close connection
     [java] com.sun.jini.qa.harness.TestException: Problem creating service for net.jini.core.lookup.ServiceRegistrar; nested exception is: 
     [java] 	RemoteException occurred in server thread; nested exception is: 
     [java] 	java.rmi.RemoteException: Create failed; nested exception is: 
     [java] 	java.lang.reflect.InvocationTargetException
     [java] 	at com.sun.jini.qa.harness.NonActivatableServiceStarterAdmin.start(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.AdminManager.startService(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.AdminManager.startLookupService(Unknown Source)
     [java] 	at com.sun.jini.test.share.BaseQATest.startLookup(Unknown Source)
     [java] 	at com.sun.jini.test.share.BaseQATest.startInitLookups(Unknown Source)
     [java] 	at com.sun.jini.test.share.BaseQATest.setup(Unknown Source)
     [java] 	at com.sun.jini.test.spec.discoverymanager.AbstractBaseTest.setup(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.MasterTest.doTest(Unknown Source)
     [java] 	at com.sun.jini.qa.harness.MasterTest.main(Unknown Source)
     [java] Caused by: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
     [java] 	java.rmi.RemoteException: Create failed; nested exception is: 
     [java] 	java.lang.reflect.InvocationTargetException
     [java] 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:336)
     [java] 	at sun.rmi.transport.Transport$1.run(Transport.java:159)
     [java] 	at java.security.AccessController.doPrivileged(Native Method)
     [java] 	at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
     [java] 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
     [java] 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
     [java] 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java] 	at java.lang.Thread.run(Thread.java:619)
     [java] 	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
     [java] 	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
     [java] 	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
     [java] 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl_Stub.startService(Unknown Source)
     [java] 	... 9 more
     [java] Caused by: java.rmi.RemoteException: Create failed; nested exception is: 
     [java] 	java.lang.reflect.InvocationTargetException
     [java] 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(Unknown Source)
     [java] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java] 	at java.lang.reflect.Method.invoke(Method.java:597)
     [java] 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
     [java] 	at sun.rmi.transport.Transport$1.run(Transport.java:159)
     [java] 	at java.security.AccessController.doPrivileged(Native Method)
     [java] 	at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
     [java] 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
     [java] 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
     [java] 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java] 	at java.lang.Thread.run(Thread.java:619)
     [java] Caused by: java.lang.reflect.InvocationTargetException
     [java] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
     [java] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
     [java] 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
     [java] 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
     [java] 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(Unknown Source)
     [java] 	... 15 more
     [java] Caused by: java.net.BindException: Address already in use
     [java] 	at java.net.PlainSocketImpl.socketBind(Native Method)
     [java] 	at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
     [java] 	at java.net.ServerSocket.bind(ServerSocket.java:319)
     [java] 	at java.net.ServerSocket.<init>(ServerSocket.java:185)
     [java] 	at java.net.ServerSocket.<init>(ServerSocket.java:97)
     [java] 	at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:157)
     [java] 	at com.sun.jini.reggie.RegistrarImpl$UnicastThread.<init>(Unknown Source)
     [java] 	at com.sun.jini.reggie.RegistrarImpl.init(Unknown Source)
     [java] 	at com.sun.jini.reggie.RegistrarImpl.access$000(Unknown Source)
     [java] 	at com.sun.jini.reggie.RegistrarImpl$1.run(Unknown Source)
     [java] 	at com.sun.jini.reggie.RegistrarImpl.<init>(Unknown Source)
     [java] 	at com.sun.jini.reggie.TransientRegistrarImpl.<init>(Unknown Source)
     [java] 	... 20 more
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport handleMessages
     [java] NonActGrp-out: FINE: RMI TCP Connection(1)-67.195.138.8: (port 48290) connection closed
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPConnection close
     [java] NonActGrp-out: FINE: RMI TCP Connection(1)-67.195.138.8: close connection
     [java] MasterTest.doTest INFO: 
     [java] ============================ CALLING TEARDOWN() =============================
     [java] 
     [java] QATest.tearDown FINE: Destroying remaining managed services
     [java] AdminManager.destroyService FINE: destroying service: class com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl_Stub
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef invoke
     [java] FINER: main: method: public abstract void com.sun.jini.qa.harness.NonActivatableGroup.stop() throws java.rmi.RemoteException
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef logClientCall
     [java] FINER: main: outbound call: [endpoint:[67.195.138.8:48290](remote),objID:[-3b9058b:1291851876c:-7fff, 5108795070014869142]] : com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl_Stub[-3b9058b:1291851876c:-7fff, 5108795070014869142]: public abstract void com.sun.jini.qa.harness.NonActivatableGroup.stop() throws java.rmi.RemoteException
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel createConnection
     [java] FINE: main: create connection
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPEndpoint newSocket
     [java] FINER: main: opening socket to [67.195.138.8:48290]
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.proxy.RMIMasterSocketFactory createSocket
     [java] FINE: main: host: 67.195.138.8, port: 48290
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport$ConnectionHandler run0
     [java] NonActGrp-out: FINE: RMI TCP Connection(2)-67.195.138.8: accepted socket from [67.195.138.8:33820]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport$ConnectionHandler run0
     [java] NonActGrp-out: FINER: RMI TCP Connection(2)-67.195.138.8: (port 48290) suggesting 67.195.138.8:33820
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel createConnection
     [java] FINER: main: server suggested 67.195.138.8:33820
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel createConnection
     [java] FINER: main: using 67.195.138.8:0
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef invoke
     [java] FINER: main: opnum = -2856118408655404442
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall <init>
     [java] FINER: main: write remote call header...
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getOutputStream
     [java] FINER: main: getting output stream
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport$ConnectionHandler run0
     [java] NonActGrp-out: FINER: RMI TCP Connection(2)-67.195.138.8: (port 48290) client using 67.195.138.8:0
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPTransport handleMessages
     [java] NonActGrp-out: FINE: RMI TCP Connection(2)-67.195.138.8: (port 48290) op = 80
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getInputStream
     [java] NonActGrp-out: FINER: RMI TCP Connection(2)-67.195.138.8: getting input stream
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.Transport serviceCall
     [java] NonActGrp-out: FINER: RMI TCP Connection(2)-67.195.138.8: call dispatcher
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastServerRef logCall
     [java] NonActGrp-out: FINER: RMI TCP Connection(2)-67.195.138.8: [67.195.138.8: com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl[-3b9058b:1291851876c:-7fff, 5108795070014869142]: public abstract void com.sun.jini.qa.harness.NonActivatableGroup.stop() throws java.rmi.RemoteException]
     [java] NonActGrp-out: Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getOutputStream
     [java] NonActGrp-out: FINER: RMI TCP Connection(2)-67.195.138.8: getting output stream
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.StreamRemoteCall getInputStream
     [java] FINER: main: getting input stream
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.server.UnicastRef invoke
     [java] FINE: main: free connection (reuse = true)
     [java] Jun 8, 2010 4:08:03 PM sun.rmi.transport.tcp.TCPChannel free
     [java] FINE: main: reuse connection
     [java] NonActGrp-out: Jun 8, 2010 4:08:05 PM sun.rmi.transport.ObjectTable removeTarget
     [java] NonActGrp-out: FINER: destroy: remove object [-3b9058b:1291851876c:-7fff, 5108795070014869142]
     [java] AdminManager.destroyService FINE: destroying service: class com.sun.jini.tool.ClassServer
     [java] Jun 8, 2010 4:08:06 PM com.sun.jini.tool.ClassServer terminate
     [java] INFO: ClassServer terminated [port 8080]
     [java] AdminManager.destroyService FINE: destroying service: class com.sun.jini.tool.ClassServer
     [java] Jun 8, 2010 4:08:06 PM com.sun.jini.tool.ClassServer terminate
     [java] INFO: ClassServer terminated [port 8080]
     [java] Jun 8, 2010 4:08:06 PM com.sun.jini.tool.ClassServer terminate
     [java] INFO: ClassServer terminated [port 8082]
     [java] AdminManager.destroyService FINE: destroying service: class com.sun.jini.tool.ClassServer
     [java] Jun 8, 2010 4:08:06 PM com.sun.jini.tool.ClassServer terminate
     [java] INFO: ClassServer terminated [port 8082]
     [java] Jun 8, 2010 4:08:06 PM com.sun.jini.tool.ClassServer terminate
     [java] INFO: ClassServer terminated [port 8081]
     [java] Jun 8, 2010 4:08:06 PM com.sun.jini.tool.ClassServer terminate
     [java] INFO: ClassServer terminated [port 8081]
     [java] 
     [java] TIME: 4:08:06 PM
     [java] 
     [java] Test process was destroyed and returned code 1
     [java] com/sun/jini/test/impl/discoverymanager/RemoveGroupsLocsDiscard.td
     [java] Test Failed: Setup Failed: com.sun.jini.qa.harness.TestException: Problem creating service for net.jini.core.lookup.ServiceRegistrar; nested exception is: 
     [java] 	RemoteException occurred in server thread; nested exception is: 
     [java] 	java.rmi.RemoteException: Create failed; nested exception is: 
     [java] 	java.lang.reflect.InvocationTargetException
     [java] 
     [java] 
     [java] -----------------------------------------



Peter Firmstone wrote:
> Hey Thanks Chris,  Awesome to see some contribution.
>
> I committed something very similar last night, it could use some 
> refactoring too, the only difference from what you have here is the 
> TaskThread creates the thread in its constructor, but it doesn't start 
> the thread in the constructor.  I added an additional method 
> TaskTread.start() to start the thread.  It avoids the null reference 
> and thread could be final, can't remember if I made the Thread 
> reference final or not.
>
> Oddly enough though , Hudson failed shortly after, so I'll need to 
> investigate why, here's the Hudson web site:
>
> http://hudson.zones.apache.org/hudson/view/River/job/River-trunk/
>
> Cheers,
>
> Peter.
>
> Christopher Dolan wrote:
>> OK, sorry for the repeated messages. The list apparently doesn't allow
>> attachments...
>>
>> --- TaskManager.java    2010-03-02 11:49:30.629703600 -0600
>> +++ TaskManager2.java    2010-06-08 11:11:54.309412900 -0500
>> @@ -190,10 +190,13 @@
>>      logAdd(t);  //DAS 6/06 add
>>      boolean poke = true;
>>      while (threads.size() < maxThreads && needThread()) {
>> -        Thread th;
>> +        TaskThread th;
>>          try {
>>          th = new TaskThread();
>> -        th.start();
>> +        Thread thread = new Thread(new TaskThread());
>> +        thread.setName("task");
>> +        thread.setDaemon(true);
>> +        thread.start();
>>          } catch (Throwable tt) {
>>          try {
>>              logger.log(threads.isEmpty() ?
>> @@ -290,8 +293,8 @@
>>              for (int j = threads.size(); --j >= 0; ) {
>>              TaskThread thread = (TaskThread)threads.get(j);
>>              if (thread.task == t) {
>> -                if (thread != Thread.currentThread())
>> -                thread.interrupt();
>> +                if (thread.myThread !=
>> Thread.currentThread())
>> +                thread.myThread.interrupt();
>>                  break;
>>              }
>>              }
>> @@ -327,15 +330,11 @@
>>      return maxThreads;
>>      }
>>  
>> -    private class TaskThread extends Thread {
>> +    private class TaskThread implements Runnable {
>>  
>>      /** The task being run, if any */
>>      public Task task = null;
>> -
>> -    public TaskThread() {
>> -        super("task");
>> -        setDaemon(true);
>> -    }
>> +    public Thread myThread = null;
>>  
>>      /**
>>       * Find the next task that can be run, and mark it taken by
>> @@ -363,8 +362,9 @@
>>      }
>>  
>>      public void run() {
>> +        myThread = Thread.currentThread();
>>          while (true) {
>>          synchronized (TaskManager.this) {
>>              if (terminated)
>>              return;
>>              if (task != null) {
>> @@ -376,11 +376,11 @@
>>                  }
>>              }
>>              task = null;
>> -            interrupted(); // clear interrupt bit
>> +            myThread.interrupted(); // clear interrupt bit
>>              }
>>              if (!takeTask()) {
>>              try {
>>                  TaskManager.this.wait(timeout);
>>              } catch (InterruptedException e) {
>>              }
>>              if (terminated || !takeTask()) {
>>
>>
>> -----Original Message-----
>> From: Christopher Dolan [mailto:christopher.dolan@avid.com] Sent: 
>> Tuesday, June 08, 2010 11:13 AM
>> To: river-dev@incubator.apache.org
>> Subject: RE: com.sun.jini.thread lock contention
>>
>> Oops, let me try that patch one more time...
>> Chris
>>
>>
>> -----Original Message-----
>> From: Christopher Dolan Sent: Tuesday, June 08, 2010 11:11 AM
>> To: 'river-dev@incubator.apache.org'
>> Subject: RE: com.sun.jini.thread lock contention
>>
>> Attached is a minimalist initial patch.  I'm not happy that I had to
>> store a reference to the thread in the Runnable...  But otherwise I
>> wasn't sure how to interrupt it from removeTask() without a separate map
>> of TaskThread to Thread.  There's all a null pointer race in this patch
>> if you try to remove a task before it's run() method executes.
>>
>> Chris
>>
>>   
>
>



Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Hey Thanks Chris,  Awesome to see some contribution.

I committed something very similar last night, it could use some 
refactoring too, the only difference from what you have here is the 
TaskThread creates the thread in its constructor, but it doesn't start 
the thread in the constructor.  I added an additional method 
TaskTread.start() to start the thread.  It avoids the null reference and 
thread could be final, can't remember if I made the Thread reference 
final or not.

Oddly enough though , Hudson failed shortly after, so I'll need to 
investigate why, here's the Hudson web site:

http://hudson.zones.apache.org/hudson/view/River/job/River-trunk/

Cheers,

Peter.

Christopher Dolan wrote:
> OK, sorry for the repeated messages. The list apparently doesn't allow
> attachments...
>
> --- TaskManager.java	2010-03-02 11:49:30.629703600 -0600
> +++ TaskManager2.java	2010-06-08 11:11:54.309412900 -0500
> @@ -190,10 +190,13 @@
>  	logAdd(t);  //DAS 6/06 add
>  	boolean poke = true;
>  	while (threads.size() < maxThreads && needThread()) {
> -	    Thread th;
> +	    TaskThread th;
>  	    try {
>  		th = new TaskThread();
> -		th.start();
> +		Thread thread = new Thread(new TaskThread());
> +		thread.setName("task");
> +		thread.setDaemon(true);
> +		thread.start();
>  	    } catch (Throwable tt) {
>  		try {
>  		    logger.log(threads.isEmpty() ?
> @@ -290,8 +293,8 @@
>  		    for (int j = threads.size(); --j >= 0; ) {
>  			TaskThread thread = (TaskThread)threads.get(j);
>  			if (thread.task == t) {
> -			    if (thread != Thread.currentThread())
> -				thread.interrupt();
> +			    if (thread.myThread !=
> Thread.currentThread())
> +				thread.myThread.interrupt();
>  			    break;
>  			}
>  		    }
> @@ -327,15 +330,11 @@
>  	return maxThreads;
>      }
>  
> -    private class TaskThread extends Thread {
> +    private class TaskThread implements Runnable {
>  
>  	/** The task being run, if any */
>  	public Task task = null;
> -
> -	public TaskThread() {
> -	    super("task");
> -	    setDaemon(true);
> -	}
> +	public Thread myThread = null;
>  
>  	/**
>  	 * Find the next task that can be run, and mark it taken by
> @@ -363,8 +362,9 @@
>  	}
>  
>  	public void run() {
> +	    myThread = Thread.currentThread();
>  	    while (true) {
>  		synchronized (TaskManager.this) {
>  		    if (terminated)
>  			return;
>  		    if (task != null) {
> @@ -376,11 +376,11 @@
>  			    }
>  			}
>  			task = null;
> -			interrupted(); // clear interrupt bit
> +			myThread.interrupted(); // clear interrupt bit
>  		    }
>  		    if (!takeTask()) {
>  			try {
>  			    TaskManager.this.wait(timeout);
>  			} catch (InterruptedException e) {
>  			}
>  			if (terminated || !takeTask()) {
>
>
> -----Original Message-----
> From: Christopher Dolan [mailto:christopher.dolan@avid.com] 
> Sent: Tuesday, June 08, 2010 11:13 AM
> To: river-dev@incubator.apache.org
> Subject: RE: com.sun.jini.thread lock contention
>
> Oops, let me try that patch one more time...
> Chris
>
>
> -----Original Message-----
> From: Christopher Dolan 
> Sent: Tuesday, June 08, 2010 11:11 AM
> To: 'river-dev@incubator.apache.org'
> Subject: RE: com.sun.jini.thread lock contention
>
> Attached is a minimalist initial patch.  I'm not happy that I had to
> store a reference to the thread in the Runnable...  But otherwise I
> wasn't sure how to interrupt it from removeTask() without a separate map
> of TaskThread to Thread.  There's all a null pointer race in this patch
> if you try to remove a task before it's run() method executes.
>
> Chris
>
>   


RE: com.sun.jini.thread lock contention

Posted by Christopher Dolan <ch...@avid.com>.
OK, sorry for the repeated messages. The list apparently doesn't allow
attachments...

--- TaskManager.java	2010-03-02 11:49:30.629703600 -0600
+++ TaskManager2.java	2010-06-08 11:11:54.309412900 -0500
@@ -190,10 +190,13 @@
 	logAdd(t);  //DAS 6/06 add
 	boolean poke = true;
 	while (threads.size() < maxThreads && needThread()) {
-	    Thread th;
+	    TaskThread th;
 	    try {
 		th = new TaskThread();
-		th.start();
+		Thread thread = new Thread(new TaskThread());
+		thread.setName("task");
+		thread.setDaemon(true);
+		thread.start();
 	    } catch (Throwable tt) {
 		try {
 		    logger.log(threads.isEmpty() ?
@@ -290,8 +293,8 @@
 		    for (int j = threads.size(); --j >= 0; ) {
 			TaskThread thread = (TaskThread)threads.get(j);
 			if (thread.task == t) {
-			    if (thread != Thread.currentThread())
-				thread.interrupt();
+			    if (thread.myThread !=
Thread.currentThread())
+				thread.myThread.interrupt();
 			    break;
 			}
 		    }
@@ -327,15 +330,11 @@
 	return maxThreads;
     }
 
-    private class TaskThread extends Thread {
+    private class TaskThread implements Runnable {
 
 	/** The task being run, if any */
 	public Task task = null;
-
-	public TaskThread() {
-	    super("task");
-	    setDaemon(true);
-	}
+	public Thread myThread = null;
 
 	/**
 	 * Find the next task that can be run, and mark it taken by
@@ -363,8 +362,9 @@
 	}
 
 	public void run() {
+	    myThread = Thread.currentThread();
 	    while (true) {
 		synchronized (TaskManager.this) {
 		    if (terminated)
 			return;
 		    if (task != null) {
@@ -376,11 +376,11 @@
 			    }
 			}
 			task = null;
-			interrupted(); // clear interrupt bit
+			myThread.interrupted(); // clear interrupt bit
 		    }
 		    if (!takeTask()) {
 			try {
 			    TaskManager.this.wait(timeout);
 			} catch (InterruptedException e) {
 			}
 			if (terminated || !takeTask()) {


-----Original Message-----
From: Christopher Dolan [mailto:christopher.dolan@avid.com] 
Sent: Tuesday, June 08, 2010 11:13 AM
To: river-dev@incubator.apache.org
Subject: RE: com.sun.jini.thread lock contention

Oops, let me try that patch one more time...
Chris


-----Original Message-----
From: Christopher Dolan 
Sent: Tuesday, June 08, 2010 11:11 AM
To: 'river-dev@incubator.apache.org'
Subject: RE: com.sun.jini.thread lock contention

Attached is a minimalist initial patch.  I'm not happy that I had to
store a reference to the thread in the Runnable...  But otherwise I
wasn't sure how to interrupt it from removeTask() without a separate map
of TaskThread to Thread.  There's all a null pointer race in this patch
if you try to remove a task before it's run() method executes.

Chris

RE: com.sun.jini.thread lock contention

Posted by Christopher Dolan <ch...@avid.com>.
Oops, let me try that patch one more time...
Chris


-----Original Message-----
From: Christopher Dolan 
Sent: Tuesday, June 08, 2010 11:11 AM
To: 'river-dev@incubator.apache.org'
Subject: RE: com.sun.jini.thread lock contention

Attached is a minimalist initial patch.  I'm not happy that I had to
store a reference to the thread in the Runnable...  But otherwise I
wasn't sure how to interrupt it from removeTask() without a separate map
of TaskThread to Thread.  There's all a null pointer race in this patch
if you try to remove a task before it's run() method executes.

Chris

Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Hi Gregg,

I see it now, it's the SoftCache that is accessed just before the 
security check.

Yep this could use some refactoring.  I'll play with it on the weekend, 
have you experimented with it yet?

Cheers,

Peter.

    /** cache of subclass security audit results */
    private static final SoftCache subclassAudits = new SoftCache(10);

   /**
     * Verifies that this (possibly subclass) instance can be constructed
     * without violating security constraints: the subclass must not 
override
     * security-sensitive non-final methods, or else the
     * "enableContextClassLoaderOverride" RuntimePermission is checked.
     */
    private static boolean isCCLOverridden(Class cl) {
    if (cl == Thread.class)
        return false;
    Boolean result = null;
    synchronized (subclassAudits) {


    /*
     * Do we have the required permissions?
     */
    if (security != null) {
        if (isCCLOverridden(getClass())) {
            security.checkPermission(SUBCLASS_IMPLEMENTATION_PERMISSION);
        }
    }

<---------------Thread.java Selected source code 
below--------------------------->

 public Thread(String name) {
        init(null, null, name, 0);
    }

    public final void checkAccess() {
        SecurityManager security = System.getSecurityManager();
        if (security != null) {
            security.checkAccess(this);
        }
    }

    private void init(ThreadGroup g, Runnable target, String name,
                      long stackSize) {
    Thread parent = currentThread();
    SecurityManager security = System.getSecurityManager();
    if (g == null) {
        /* Determine if it's an applet or not */
       
        /* If there is a security manager, ask the security manager
           what to do. */
        if (security != null) {
        g = security.getThreadGroup();
        }

        /* If the security doesn't have a strong opinion of the matter
           use the parent thread group. */
        if (g == null) {
        g = parent.getThreadGroup();
        }
    }

    /* checkAccess regardless of whether or not threadgroup is
           explicitly passed in. */
    g.checkAccess();

    /*
     * Do we have the required permissions?
     */
    if (security != null) {
        if (isCCLOverridden(getClass())) {
            security.checkPermission(SUBCLASS_IMPLEMENTATION_PERMISSION);
        }
    }


        g.addUnstarted();

    this.group = g;
    this.daemon = parent.isDaemon();
    this.priority = parent.getPriority();
    this.name = name.toCharArray();
    if (security == null || isCCLOverridden(parent.getClass()))
        this.contextClassLoader = parent.getContextClassLoader();
    else
        this.contextClassLoader = parent.contextClassLoader;
    this.inheritedAccessControlContext = AccessController.getContext();
    this.target = target;
    setPriority(priority);
        if (parent.inheritableThreadLocals != null)
        this.inheritableThreadLocals =
        ThreadLocal.createInheritedMap(parent.inheritableThreadLocals);
        /* Stash the specified stack size in case the VM cares */
        this.stackSize = stackSize;

        /* Set thread ID */
        tid = nextThreadID();
    }

   /**
     * Verifies that this (possibly subclass) instance can be constructed
     * without violating security constraints: the subclass must not 
override
     * security-sensitive non-final methods, or else the
     * "enableContextClassLoaderOverride" RuntimePermission is checked.
     */
    private static boolean isCCLOverridden(Class cl) {
    if (cl == Thread.class)
        return false;
    Boolean result = null;
    synchronized (subclassAudits) {
        result = (Boolean) subclassAudits.get(cl);
        if (result == null) {
        /*
         * Note: only new Boolean instances (i.e., not Boolean.TRUE or
         * Boolean.FALSE) must be used as cache values, otherwise cache
         * entry will pin associated class.
         */
        result = new Boolean(auditSubclass(cl));
        subclassAudits.put(cl, result);
        }
    }
    return result.booleanValue();
    }


Gregg Wonderly wrote:
> I am on the road so not able to be most timely.  The issue is that 
> there is a class level lock applied when a thead instance is created 
> with a subclass of thread.  Look at the constructors got the details.
>
> Gregg Wonderly
>
> Sent from my iPhone
>
> On Jun 4, 2010, at 11:02 PM, Peter Firmstone <ji...@zeus.net.au> wrote:
>
>> Hi Gregg,
>>
>> The contention appears to be around the use of the existing single 
>> thread synchronized Policy implementations like DynamicPolicyProvider.
>>
>> There are numerous places where Thread is checked (not just 
>> subclasses) by the SecurityManager using the checkAccess(Thread) 
>> method, this is delegated to AccessController.checkPermission(perm)
>>
>> checkPermission(perm) gets the AccessControlContext of the currently 
>> executing stack, AccessControlContext.optimize() is called to remove 
>> duplicate PermissionDomain's
>>
>> then AccessControlContext.checkPermission(perm) is called to check 
>> the permission's of all the ProtectionDomain's on the current stack, 
>> the ProtectionDomain's on the stack are iterated through, the first 
>> ProtectionDomain.implies(Permission) that returns false returns 
>> causes the AccessControlContext to return false.
>>
>> This means that if your thread is executing with several CodeSource's 
>> it will have to check several ProtectionDomains.
>>
>> Each ProtectionDomain will check the policy, if the ProtectionDomain 
>> was created with the dynamic policy domain constructor:
>>
>> public ProtectionDomain(CodeSource codesource,
>>               PermissionCollection permissions,
>>               ClassLoader classloader,
>>               Principal[] principals)
>>
>> The ProtectionDomain first check's the Policy for the permission, 
>> then it checks it's own private PermissionCollection permissions 
>> passed in via the constructor, or that which has been merged with the 
>> permissions from the Policy.
>> Policy.getPermissions(ProtectionDomain)
>>
>> In my RevokeablePolicy implementation, getPermissions doesn't return 
>> any dynamic Permission grant's, if it did, they could never be 
>> revoked as they would become merged with the ProtectionDomains 
>> private PermissionCollection.
>>
>> So to sum up, you should get much better mileage if you try out my 
>> new ConcurrentDynamicPolicy implementation.
>>
>> I know you've got a privately forked copy of River, but if you simply 
>> copy concurrent_policy_utils.jar and jsk-policy.jar from River's 
>> Hudson build, to your jre/lib/ext directory, you should be able to 
>> utilise it.
>>
>> The ConcurrentDynanicPolicyProvider has passed all qa tests and jtreg 
>> tests, I'm just waiting for someone like yourself to report back 
>> concurrency improvements.
>>
>> Cheers,
>>
>> Peter.
>>
>>
>> Gregg Wonderly wrote:
>>> The TaskManager class has a subclass of Thread that it uses to run 
>>> tasks.  The problem with subclassing Thread, is that there is a lock 
>>> that is used in Thread to see if some methods are overridden so that 
>>> security can be maintained.  So, in an environment where Thread is 
>>> used with a SecurityManager, it is better to use a Runnable to 
>>> encapsulate the custom code, instead of a Thread.
>>>
>>> The TaskThread class should be a Runnable instead of a Thread, and 
>>> then there are just 3 other places where it is referenced that need 
>>> some adjustment.  The thread that is running inside of the runnable 
>>> needs to be put into a per instance variable for one place where 
>>> interrupt processing is handled.
>>>
>>> Gregg Wonderly
>>>
>>
>


Re: com.sun.jini.thread lock contention

Posted by Gregg Wonderly <gr...@wonderly.org>.
I am on the road so not able to be most timely.  The issue is that  
there is a class level lock applied when a thead instance is created  
with a subclass of thread.  Look at the constructors got the details.

Gregg Wonderly

Sent from my iPhone

On Jun 4, 2010, at 11:02 PM, Peter Firmstone <ji...@zeus.net.au> wrote:

> Hi Gregg,
>
> The contention appears to be around the use of the existing single  
> thread synchronized Policy implementations like DynamicPolicyProvider.
>
> There are numerous places where Thread is checked (not just  
> subclasses) by the SecurityManager using the checkAccess(Thread)  
> method, this is delegated to AccessController.checkPermission(perm)
>
> checkPermission(perm) gets the AccessControlContext of the currently  
> executing stack, AccessControlContext.optimize() is called to remove  
> duplicate PermissionDomain's
>
> then AccessControlContext.checkPermission(perm) is called to check  
> the permission's of all the ProtectionDomain's on the current stack,  
> the ProtectionDomain's on the stack are iterated through, the first  
> ProtectionDomain.implies(Permission) that returns false returns  
> causes the AccessControlContext to return false.
>
> This means that if your thread is executing with several  
> CodeSource's it will have to check several ProtectionDomains.
>
> Each ProtectionDomain will check the policy, if the ProtectionDomain  
> was created with the dynamic policy domain constructor:
>
> public ProtectionDomain(CodeSource codesource,
>               PermissionCollection permissions,
>               ClassLoader classloader,
>               Principal[] principals)
>
> The ProtectionDomain first check's the Policy for the permission,  
> then it checks it's own private PermissionCollection permissions  
> passed in via the constructor, or that which has been merged with  
> the permissions from the Policy.
> Policy.getPermissions(ProtectionDomain)
>
> In my RevokeablePolicy implementation, getPermissions doesn't return  
> any dynamic Permission grant's, if it did, they could never be  
> revoked as they would become merged with the ProtectionDomains  
> private PermissionCollection.
>
> So to sum up, you should get much better mileage if you try out my  
> new ConcurrentDynamicPolicy implementation.
>
> I know you've got a privately forked copy of River, but if you  
> simply copy concurrent_policy_utils.jar and jsk-policy.jar from  
> River's Hudson build, to your jre/lib/ext directory, you should be  
> able to utilise it.
>
> The ConcurrentDynanicPolicyProvider has passed all qa tests and  
> jtreg tests, I'm just waiting for someone like yourself to report  
> back concurrency improvements.
>
> Cheers,
>
> Peter.
>
>
> Gregg Wonderly wrote:
>> The TaskManager class has a subclass of Thread that it uses to run  
>> tasks.  The problem with subclassing Thread, is that there is a  
>> lock that is used in Thread to see if some methods are overridden  
>> so that security can be maintained.  So, in an environment where  
>> Thread is used with a SecurityManager, it is better to use a  
>> Runnable to encapsulate the custom code, instead of a Thread.
>>
>> The TaskThread class should be a Runnable instead of a Thread, and  
>> then there are just 3 other places where it is referenced that need  
>> some adjustment.  The thread that is running inside of the runnable  
>> needs to be put into a per instance variable for one place where  
>> interrupt processing is handled.
>>
>> Gregg Wonderly
>>
>

Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Hi Gregg,

The contention appears to be around the use of the existing single 
thread synchronized Policy implementations like DynamicPolicyProvider.

There are numerous places where Thread is checked (not just subclasses) 
by the SecurityManager using the checkAccess(Thread) method, this is 
delegated to AccessController.checkPermission(perm)

checkPermission(perm) gets the AccessControlContext of the currently 
executing stack, AccessControlContext.optimize() is called to remove 
duplicate PermissionDomain's

then AccessControlContext.checkPermission(perm) is called to check the 
permission's of all the ProtectionDomain's on the current stack, the 
ProtectionDomain's on the stack are iterated through, the first 
ProtectionDomain.implies(Permission) that returns false returns causes 
the AccessControlContext to return false.

This means that if your thread is executing with several CodeSource's it 
will have to check several ProtectionDomains.

Each ProtectionDomain will check the policy, if the ProtectionDomain was 
created with the dynamic policy domain constructor:

public ProtectionDomain(CodeSource codesource,
                PermissionCollection permissions,
                ClassLoader classloader,
                Principal[] principals)

The ProtectionDomain first check's the Policy for the permission, then 
it checks it's own private PermissionCollection permissions passed in 
via the constructor, or that which has been merged with the permissions 
from the Policy. 

Policy.getPermissions(ProtectionDomain)

In my RevokeablePolicy implementation, getPermissions doesn't return any 
dynamic Permission grant's, if it did, they could never be revoked as 
they would become merged with the ProtectionDomains private 
PermissionCollection.

So to sum up, you should get much better mileage if you try out my new 
ConcurrentDynamicPolicy implementation.

I know you've got a privately forked copy of River, but if you simply 
copy concurrent_policy_utils.jar and jsk-policy.jar from River's Hudson 
build, to your jre/lib/ext directory, you should be able to utilise it.

The ConcurrentDynanicPolicyProvider has passed all qa tests and jtreg 
tests, I'm just waiting for someone like yourself to report back 
concurrency improvements.

Cheers,

Peter.


Gregg Wonderly wrote:
> The TaskManager class has a subclass of Thread that it uses to run 
> tasks.  The problem with subclassing Thread, is that there is a lock 
> that is used in Thread to see if some methods are overridden so that 
> security can be maintained.  So, in an environment where Thread is 
> used with a SecurityManager, it is better to use a Runnable to 
> encapsulate the custom code, instead of a Thread.
>
> The TaskThread class should be a Runnable instead of a Thread, and 
> then there are just 3 other places where it is referenced that need 
> some adjustment.  The thread that is running inside of the runnable 
> needs to be put into a per instance variable for one place where 
> interrupt processing is handled.
>
> Gregg Wonderly
>


RE: com.sun.jini.thread lock contention

Posted by Christopher Dolan <ch...@avid.com>.
The big feature that TaskManager has that Java 1.5 lacks is the ability
to reorder the tasks by asking each task if it wants to be before the
others (Task.runAfter()).  That could be refactored into a custom
BlockingQueue implementation, I suppose.

Chris

-----Original Message-----
From: Patrick Wright [mailto:pdoubleya@gmail.com] 
Sent: Friday, June 04, 2010 11:56 AM
To: river-dev@incubator.apache.org
Subject: Re: com.sun.jini.thread lock contention

On Fri, Jun 4, 2010 at 6:18 PM, Dennis Reedy <de...@gmail.com>
wrote:
> Hi Gregg,
>
> Does it make sense to move to concurrency utilities and move away from
TaskManager altogether?

You mean Java 5 java.util.concurrent? I'd be all for that, personally.

Re: com.sun.jini.thread lock contention

Posted by Dennis Reedy <de...@gmail.com>.
On Jun 4, 2010, at 1256PM, Patrick Wright wrote:

> On Fri, Jun 4, 2010 at 6:18 PM, Dennis Reedy <de...@gmail.com> wrote:
>> Hi Gregg,
>> 
>> Does it make sense to move to concurrency utilities and move away from TaskManager altogether?
> 
> You mean Java 5 java.util.concurrent?

Yes, exactly 

Dennis

Re: com.sun.jini.thread lock contention

Posted by Patrick Wright <pd...@gmail.com>.
On Fri, Jun 4, 2010 at 6:18 PM, Dennis Reedy <de...@gmail.com> wrote:
> Hi Gregg,
>
> Does it make sense to move to concurrency utilities and move away from TaskManager altogether?

You mean Java 5 java.util.concurrent? I'd be all for that, personally.

Re: com.sun.jini.thread lock contention

Posted by Gregg Wonderly <gr...@wonderly.org>.
If you set maximumPoolSize to Integer.MAX_VALUE, then you have created a TPE 
instance that won't fail to invoke a needed task, so that would be okay.  I'm 
just trying to make the point that any maximumPoolSize that is not essentially 
infinite can, at some point, create problems.

Infinite is relative to the size of the application, so there are values that 
will work, I just want to try and make the point that guessing the appropriate 
value can be problematic, so having behavior where there is essentially no 
maximum is really what we need.

At some point, a large enough machine and application could even make 
Integer.MAX_VALUE problematic... :=)

Gregg Wonderly

James Grahn wrote:
> Would this be roughly equivalent to:
> Executors.newCachedThreadPool()?
> 
> The doc says:
> "Creates a thread pool that creates new threads as needed, but will 
> reuse previously constructed threads when they are available."
> 
> That factory creates a ThreadPoolExecutor with a core size of 0, maximum 
> size of Integer.MAX_VALUE, and it kills idle threads after 60 seconds.
> 
> We can construct our own ThreadPoolExecutor if we need finer control of 
> those parameters.
> 
> I may be missing some subtle detail, though.
> 
> jamesG
> 
> On 6/4/2010 4:11 PM, Gregg Wonderly wrote:
>> I'd have to say no for the existing Executor implementations in the JDK.
>>   The concurrency utilities related to Executor in the JDK are tailored
>> for, and specifically limited to applications where you have unrelated
>> tasks that need to be throttled amongst a limited set of threads.
>>
>> River's TaskManager will always create more threads, but will prune
>> those threads after they sit idle for too long. We need this behavior to
>> keep away from distributed deadlock which can occur anytime another
>> remote operation might be the only way that progress can happen in the
>> overall processing of a distributed application.
>>
>> Be very careful where you use j.u.c ThreadPoolExecutor et.al, because of
>> their design. It would be possible to use an appropriate Executor
>> implementation, but it would have to behave like TaskManager and always
>> create a new thread for new work, when no idle threads are available.
>>
>> Gregg Wonderly
>>
>> Dennis Reedy wrote:
>>> Hi Gregg,
>>>
>>> Does it make sense to move to concurrency utilities and move away from
>>> TaskManager altogether?
>>>
>>> Dennis
>>>
>>> On Jun 4, 2010, at 1100AM, Gregg Wonderly wrote:
>>>
>>>> The TaskManager class has a subclass of Thread that it uses to run
>>>> tasks. The problem with subclassing Thread, is that there is a lock
>>>> that is used in Thread to see if some methods are overridden so that
>>>> security can be maintained. So, in an environment where Thread is
>>>> used with a SecurityManager, it is better to use a Runnable to
>>>> encapsulate the custom code, instead of a Thread.
>>>>
>>>> The TaskThread class should be a Runnable instead of a Thread, and
>>>> then there are just 3 other places where it is referenced that need
>>>> some adjustment. The thread that is running inside of the runnable
>>>> needs to be put into a per instance variable for one place where
>>>> interrupt processing is handled.
>>>>
>>>> Gregg Wonderly
>>>
>>>
>>
>>
> 


Re: com.sun.jini.thread lock contention

Posted by James Grahn <jg...@simulexinc.com>.
Would this be roughly equivalent to:
Executors.newCachedThreadPool()?

The doc says:
"Creates a thread pool that creates new threads as needed, but will 
reuse previously constructed threads when they are available."

That factory creates a ThreadPoolExecutor with a core size of 0, maximum 
size of Integer.MAX_VALUE, and it kills idle threads after 60 seconds.

We can construct our own ThreadPoolExecutor if we need finer control of 
those parameters.

I may be missing some subtle detail, though.

jamesG

On 6/4/2010 4:11 PM, Gregg Wonderly wrote:
> I'd have to say no for the existing Executor implementations in the JDK.
>   The concurrency utilities related to Executor in the JDK are tailored
> for, and specifically limited to applications where you have unrelated
> tasks that need to be throttled amongst a limited set of threads.
>
> River's TaskManager will always create more threads, but will prune
> those threads after they sit idle for too long. We need this behavior to
> keep away from distributed deadlock which can occur anytime another
> remote operation might be the only way that progress can happen in the
> overall processing of a distributed application.
>
> Be very careful where you use j.u.c ThreadPoolExecutor et.al, because of
> their design. It would be possible to use an appropriate Executor
> implementation, but it would have to behave like TaskManager and always
> create a new thread for new work, when no idle threads are available.
>
> Gregg Wonderly
>
> Dennis Reedy wrote:
>> Hi Gregg,
>>
>> Does it make sense to move to concurrency utilities and move away from
>> TaskManager altogether?
>>
>> Dennis
>>
>> On Jun 4, 2010, at 1100AM, Gregg Wonderly wrote:
>>
>>> The TaskManager class has a subclass of Thread that it uses to run
>>> tasks. The problem with subclassing Thread, is that there is a lock
>>> that is used in Thread to see if some methods are overridden so that
>>> security can be maintained. So, in an environment where Thread is
>>> used with a SecurityManager, it is better to use a Runnable to
>>> encapsulate the custom code, instead of a Thread.
>>>
>>> The TaskThread class should be a Runnable instead of a Thread, and
>>> then there are just 3 other places where it is referenced that need
>>> some adjustment. The thread that is running inside of the runnable
>>> needs to be put into a per instance variable for one place where
>>> interrupt processing is handled.
>>>
>>> Gregg Wonderly
>>
>>
>
>

Re: com.sun.jini.thread lock contention

Posted by Patrick Wright <pd...@gmail.com>.
On Thu, Jun 10, 2010 at 8:06 AM, Gregg Wonderly <gr...@wonderly.org> wrote:
> As was kind of discussed earlier, TPE uses max threads in a different way
> then most people would think.

I think we will end up needing a very precise description of the
behavior we expect from our thread manager, esp. regarding queueing.
If we can't know beforehand which incoming calls are critical, but
have a (natural) upper limit on native threads we can instantiate,
then perhaps the manager should be configured to reject the incoming
call entirely instead of queueing it.

Below the natural maximum amount of native threads, we can force a new
thread to be created per call (assuming none are idle). From the TPE
documentation, it seems this is covered in the class JavaDocs, under
Queueing, "Direct handoffs. A good default choice for a work queue is
a SynchronousQueue that hands off tasks to threads without otherwise
holding them. Here, an attempt to queue a task will fail if no threads
are immediately available to run it, so a new thread will be
constructed. This policy avoids lockups when handling sets of requests
that might have internal dependencies. Direct handoffs generally
require unbounded maximumPoolSizes to avoid rejection of new submitted
tasks. This in turn admits the possibility of unbounded thread growth
when commands continue to arrive on average faster than they can be
processed."[1]


Patrick

[1] http://java.sun.com/javase/6/docs/api/java/util/concurrent/ThreadPoolExecutor.html

Re: com.sun.jini.thread lock contention

Posted by Sim IJskes - QCG <si...@qcg.nl>.
On 06/19/2010 02:22 AM, Peter Firmstone wrote:
> Agreed, however we still need to be able to provide a way to ensure some
> tasks are run first there are dependencies between tasks, order sequence
> etc, Chris put it most eloquently.

Ok. Thanks!

Gr. Sim

Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Hi Sim,

Agreed, however we still need to be able to provide a way to ensure some 
tasks are run first there are dependencies between tasks, order sequence 
etc, Chris put it most eloquently.

> The big feature that TaskManager has that Java 1.5 lacks is the ability
> to reorder the tasks by asking each task if it wants to be before the
> others (Task.runAfter()).  That could be refactored into a custom
> BlockingQueue implementation, I suppose.
>
> Chris
>   
TaskManager (from recollection) returns any tasks that aren't ready to 
run to the beginning of the queue.  This wouldn't be acceptable with a 
BlockingQueue when max threads has been reached.

Instead, I've created a class, DependantTask extends FutureTask, and an 
interface called PriorityHandler, which the DependantTask expects in its 
constructor.

When a DependantTask is run(), it first asks the PriorityHandler if 
anything else should be run first, passing itself in as the argument.  
The Runnable contained within the DependantTask can be revealed to a 
PriorityHandler implementation, all the tasks on a BlockingQueue can be 
observed, so the PriorityHandler implementation can use several 
mechanisms in its determination.

The RunnableFuture's returned to DependantTask, by the PriorityHander 
are run first, these may also be running in concurrent threads, however 
this is acceptable as it will reduce the number of tasks run by the 
DependantTask's thread.

Have a look on SVN, under org.apache.river.imp.util.*  I'm not 100% 
happy with it, perhaps you can improve it.

Cheers,

Peter.

Sim IJskes - QCG wrote:
> On 06/17/2010 03:26 PM, Sim IJskes - QCG wrote:
>> So this would be an implementation that will mimmick the old behaviour?
>>
>>
>> ThreadFactory tf = new DaemonThreadFactory();
>>
>> execSvc = new ThreadPoolExecutor(0, maxThreads, 15L, TimeUnit.MINUTES,
>> new LinkedBlockingQueue<Runnable>(1), tf, new AbortPolicy() );
>>
>> execSvc.allowCoreThreadTimeOut(true);
>>
>> maxThreads is the limit on the number of threads.
>
> And a possible improvement would be:
>
> tpe = new ThreadPoolExecutor(maxThreads, maxThreads, 15, 
> TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), tf, new 
> AbortPolicy() );
> tpe.allowCoreThreadTimeOut(true);
>
> Gr. Sim
>


Re: com.sun.jini.thread lock contention

Posted by Sim IJskes - QCG <si...@qcg.nl>.
On 06/17/2010 03:26 PM, Sim IJskes - QCG wrote:
> So this would be an implementation that will mimmick the old behaviour?
>
>
> ThreadFactory tf = new DaemonThreadFactory();
>
> execSvc = new ThreadPoolExecutor(0, maxThreads, 15L, TimeUnit.MINUTES,
> new LinkedBlockingQueue<Runnable>(1), tf, new AbortPolicy() );
>
> execSvc.allowCoreThreadTimeOut(true);
>
> maxThreads is the limit on the number of threads.

And a possible improvement would be:

tpe = new ThreadPoolExecutor(maxThreads, maxThreads, 15, 
TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), tf, new 
AbortPolicy() );
tpe.allowCoreThreadTimeOut(true);

Gr. Sim

Re: com.sun.jini.thread lock contention

Posted by Sim IJskes - QCG <si...@qcg.nl>.
On 10-06-10 08:06, Gregg Wonderly wrote:
> As was kind of discussed earlier, TPE uses max threads in a different
> way then most people would think.
>
> It creates up to min threads, as long as the "queue is not full"
> (offer() succeeding is "not full"). When the queue is "full", it creates
> up to maxPoolSize threads for (maxPoolSize-corePoolSize) more adds.
>
> So for an open ended queue, maxPoolSize has no meaning, and only
> corePoolSize threads will ever be created it looks to me.
>

So this would be an implementation that will mimmick the old behaviour?


ThreadFactory tf = new DaemonThreadFactory();

execSvc = new ThreadPoolExecutor(0, maxThreads, 15L, TimeUnit.MINUTES, 
new LinkedBlockingQueue<Runnable>(1), tf, new AbortPolicy() );

execSvc.allowCoreThreadTimeOut(true);

maxThreads is the limit on the number of threads.

Gr. Sim

-- 
QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl
Quality Consultancy Group b.v., Leiderdorp, Kvk Den Haag: 28088397

Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Come come now, lets not speak poorly of TPE, that is only one form of 
expected behaviour, it happens to be very flexible, it's also only one 
implementation of an ExecutorService.

For example if you use TPE with a BlockingQueue and a ThreadFactory (to 
create daemon threads), where the blocking queue is of a fixed size, it 
can be tuned to the environment it runs on. 

TPE runs with the core pool size while the blocking queue isn't full, 
however once it is, TPE adds more threads until it isn't, so the 
blocking queue doesn't remain full, instead TPE finds a balance, where 
you have a fixed size queue and sufficient threads that reach some point 
of equilibrium between the core pool size and max threads.

You might set max threads based on the time you expect some of your 
threads to be stalled and the number of available cpu threads.  This you 
can determine at runtime.

I've just committed a RunnableFuture that can be submitted to an 
Executor, called DependantTask, that asks a PriorityHandler what tasks 
must be executed first.  The DependantTask and all the tasks it depends 
upon can be executed asymmetrically by the queue, but when the 
DependantTask is run, it submits itself to the PriorityHandler (an 
interface) asking for dependent RunnableFuture's and runs them first, 
many of them, may have already run from concurrent threads, however it 
ensures that all dependent tasks complete first, by calling the Runnable 
future get() method that waits for completion before runnning itself.  
This is independent of the Executor implementation, as the Executor need 
not concern itself with the dependencies, neither should the 
DependantTask, that's the job of the PriorityHandler, which can work 
across multiple Executors too.

Keep an open mind lads. 

My +1 goes to Patrick, pickup the benefits of performance as platforms 
improve.  They've got more time & money than we'll ever have.

Cheers,

Peter.

Christopher Dolan wrote:
> Wow, I did not know that about TPE.  I need to go fix some code...
> I withdraw my advocacy for switching TaskManager to use TPE.
>
> Chris
>
> -----Original Message-----
> From: Gregg Wonderly [mailto:gregg@wonderly.org] 
> Sent: Thursday, June 10, 2010 1:07 AM
> To: river-dev@incubator.apache.org
> Subject: Re: com.sun.jini.thread lock contention
>
> As was kind of discussed earlier, TPE uses max threads in a different
> way then 
> most people would think.
>
> It creates up to min threads, as long as the "queue is not full"
> (offer() 
> succeeding is "not full").  When the queue is "full", it creates up to 
> maxPoolSize threads for (maxPoolSize-corePoolSize) more adds.
>
> So for an open ended queue, maxPoolSize has no meaning, and only
> corePoolSize 
> threads will ever be created it looks to me.
>
> I think there are still bugs to be fixed and behavior to be better
> defined in 
> TPE in particular.  I'm all for using the interfaces in j.u.c that make
> sense to 
> use.  But I am a bit worried about swapping out TaskManager for TPE
> without a 
> lot more studying around the exact behaviors and failure modes, which
> might help 
> identify some issues we need to address overall regarding how threads
> are managed.
>
> For example, the default life of 15mins on threads created by
> TaskManager in its 
> current version, seems extreme.  15 seconds would be much better for
> resource 
> management I think.
>
> Gregg Wonderly
>
> Patrick Wright wrote:
>   
>> One point I'd like to raise about using java.util.concurrent and TPE:
>> I think that over the long term, it makes sense to (re)use existing
>> utilities which are being maintained by domain experts rather than
>> custom utilities you've written yourself. The concurrent libraries
>> available since Java 5 were written and maintained by people widely
>> recognized to be very, very good at a very hard problem. That doesn't
>> mean they, or the library, is perfect, just that there is value in
>> building on their work and letting them take care of the bugs and
>> optimizations over time. The downside would be that if a River user
>> was stuck with, say, Java 5, they couldn't take advantage of bugfixes
>> or improvements in Java 6. On the other hand, that's true of the
>> entire JDK.
>>
>> The max threads issue seems to me a non-issue. A JVM can allocate only
>> so many native threads before it runs out of OS resources; that's a
>> hard limit. You can set a max of Integer.MAX_VALUE but your VM would
>> die long, long before it reached that.
>>
>> For me this is more of design policy decision. Re-use, intelligently
>> and selectively, where possible, to reduce your project's workload.
>>
>>
>> Patrick
>>
>>     
>
>
>   


RE: com.sun.jini.thread lock contention

Posted by Christopher Dolan <ch...@avid.com>.
Wow, I did not know that about TPE.  I need to go fix some code...
I withdraw my advocacy for switching TaskManager to use TPE.

Chris

-----Original Message-----
From: Gregg Wonderly [mailto:gregg@wonderly.org] 
Sent: Thursday, June 10, 2010 1:07 AM
To: river-dev@incubator.apache.org
Subject: Re: com.sun.jini.thread lock contention

As was kind of discussed earlier, TPE uses max threads in a different
way then 
most people would think.

It creates up to min threads, as long as the "queue is not full"
(offer() 
succeeding is "not full").  When the queue is "full", it creates up to 
maxPoolSize threads for (maxPoolSize-corePoolSize) more adds.

So for an open ended queue, maxPoolSize has no meaning, and only
corePoolSize 
threads will ever be created it looks to me.

I think there are still bugs to be fixed and behavior to be better
defined in 
TPE in particular.  I'm all for using the interfaces in j.u.c that make
sense to 
use.  But I am a bit worried about swapping out TaskManager for TPE
without a 
lot more studying around the exact behaviors and failure modes, which
might help 
identify some issues we need to address overall regarding how threads
are managed.

For example, the default life of 15mins on threads created by
TaskManager in its 
current version, seems extreme.  15 seconds would be much better for
resource 
management I think.

Gregg Wonderly

Patrick Wright wrote:
> One point I'd like to raise about using java.util.concurrent and TPE:
> I think that over the long term, it makes sense to (re)use existing
> utilities which are being maintained by domain experts rather than
> custom utilities you've written yourself. The concurrent libraries
> available since Java 5 were written and maintained by people widely
> recognized to be very, very good at a very hard problem. That doesn't
> mean they, or the library, is perfect, just that there is value in
> building on their work and letting them take care of the bugs and
> optimizations over time. The downside would be that if a River user
> was stuck with, say, Java 5, they couldn't take advantage of bugfixes
> or improvements in Java 6. On the other hand, that's true of the
> entire JDK.
> 
> The max threads issue seems to me a non-issue. A JVM can allocate only
> so many native threads before it runs out of OS resources; that's a
> hard limit. You can set a max of Integer.MAX_VALUE but your VM would
> die long, long before it reached that.
> 
> For me this is more of design policy decision. Re-use, intelligently
> and selectively, where possible, to reduce your project's workload.
> 
> 
> Patrick
> 


Re: com.sun.jini.thread lock contention

Posted by Gregg Wonderly <gr...@wonderly.org>.
As was kind of discussed earlier, TPE uses max threads in a different way then 
most people would think.

It creates up to min threads, as long as the "queue is not full" (offer() 
succeeding is "not full").  When the queue is "full", it creates up to 
maxPoolSize threads for (maxPoolSize-corePoolSize) more adds.

So for an open ended queue, maxPoolSize has no meaning, and only corePoolSize 
threads will ever be created it looks to me.

I think there are still bugs to be fixed and behavior to be better defined in 
TPE in particular.  I'm all for using the interfaces in j.u.c that make sense to 
use.  But I am a bit worried about swapping out TaskManager for TPE without a 
lot more studying around the exact behaviors and failure modes, which might help 
identify some issues we need to address overall regarding how threads are managed.

For example, the default life of 15mins on threads created by TaskManager in its 
current version, seems extreme.  15 seconds would be much better for resource 
management I think.

Gregg Wonderly

Patrick Wright wrote:
> One point I'd like to raise about using java.util.concurrent and TPE:
> I think that over the long term, it makes sense to (re)use existing
> utilities which are being maintained by domain experts rather than
> custom utilities you've written yourself. The concurrent libraries
> available since Java 5 were written and maintained by people widely
> recognized to be very, very good at a very hard problem. That doesn't
> mean they, or the library, is perfect, just that there is value in
> building on their work and letting them take care of the bugs and
> optimizations over time. The downside would be that if a River user
> was stuck with, say, Java 5, they couldn't take advantage of bugfixes
> or improvements in Java 6. On the other hand, that's true of the
> entire JDK.
> 
> The max threads issue seems to me a non-issue. A JVM can allocate only
> so many native threads before it runs out of OS resources; that's a
> hard limit. You can set a max of Integer.MAX_VALUE but your VM would
> die long, long before it reached that.
> 
> For me this is more of design policy decision. Re-use, intelligently
> and selectively, where possible, to reduce your project's workload.
> 
> 
> Patrick
> 


Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
In summary, it is possible to create a new better TaskManger that takes 
advantage of new concurrency libraries, but it cannot be directly 
replaced by ThreadPoolExceutor, which is what Gregg is trying to say.

Fundamentally, we must learn everything about what we have and 
understand it, in order to improve or replace it.  I think we all are in 
the process of understanding, but we must be careful not to be too 
critical of each other, and be mindful not to hold our ground too 
steadfast either, to remain humble, so the list doesn't descend into 
argument.

I also think Chris has gotten to the heart of the matter:
> The big feature that TaskManager has that Java 1.5 lacks is the ability
> to reorder the tasks by asking each task if it wants to be before the
> others (Task.runAfter()).  That could be refactored into a custom
> BlockingQueue implementation, I suppose.
>
> Chris
It is interesting TPE doesn't create extra threads until the queue is 
full. 

Reading the source code from TaskManager it is apparent, maxThreads does 
in fact limit the total number of threads in TaskManager and I was 
previously mistaken.

This is taken directly from ThreadManager:

    /** Add a new task. */
    public synchronized void add(Task t) {
    tasks.add(t);
    boolean poke = true;
    while (threads.size() < maxThreads && needThread()) {
        Thread th;
        try {
        th = new TaskThread();
        th.start();

<comment> A new TaskThread (which is a Thread) is created every time the 
available threads is less than maxThreads AND a new thread is needed!  
So even if a new Thread is needed, we cannot have more than 
maxThreads</comment>

    /** Return true if a new thread should be created (ignoring 
maxThreads). */
    protected boolean needThread() {
    int bound = (int)(loadFactor * threads.size());
    int max = tasks.size();
    if (max < bound)
        return false;

<comment> Above: This first piece of code shows that the load factor 
influences the creation of new threads, in this case, the upper bound is 
the load factor * the number of threads currently in use.  So you must 
have more tasks than threads * load factor, to get past this check.  
This is similar to having the full Queue in TPE.</comment>

    max--;
    if (runAfter((Task)tasks.get(max), max))
        return false;

<comment> Above: If the last task must be run after some other task 
preceding it in the task list, return false.</comment>

    int ready = firstPending + 1;
    if (ready > bound)
        return true;

<comment> Above: First pending is incremented every time a task is found 
ready to run using "private boolean takeTask() " and it is decremented 
every time a task is "public void run()"  This may influence the 
creation of a new thread, if there are more ready tasks than total 
threads * loadFactor. </comment>

    for (int i = firstPending; i < max; i++) {
        if (!runAfter((Task)tasks.get(i), i)) {
        ready++;
        if (ready > bound)
            return true;
        }

<comment> Above this, piece of code loops over the tasks between 
firstPending and the maximum bound of the tasks List.  ready is 
incremented every time a ready task is found.  Then if there are more 
tasks ready than the total number of threads * loadFactor, a new thread 
will be created.

    }
    return false;
    }

Gregg Wonderly wrote:
> I am not against the thoughts here.  TaskManager has a very important 
> roll that has specific needs.  It is used to dispatch inbound calls 
> amongst other things.  It is essential that it never fail to create a 
> thread to handle an inbound call.  Without this behavior there is a 
> chance for distributed deadlock.
>
> The key clause in TPE is that it says the extra threads are not 
> created until the queue is full.  That is another form of "maximum".
>
> If it has a documented behavior that has no maximum, then I am okay 
> with using it.  I have just never seen it behave correctly based on 
> what I have experienced and so I am concerned about the maximum thread 
> issue.
>
> There are of course many types of applications that can use a ceiling 
> on maximum threads to throttle things.  But if you look at some of the 
> bug reports and discussions on the concurrency-interest list, there 
> are cases of this behavior popping up in the fork-join pool stuff too, 
> where the complexity of hoe work is divided and distributed creates 
> problems.
>
> Gregg Wonderly
>
> Sent from my iPhone
>
> On Jun 5, 2010, at 8:25 AM, Dennis Reedy <de...@gmail.com> wrote:
>
>>
>> On Jun 5, 2010, at 609AM, Patrick Wright wrote:
>>
>>> One point I'd like to raise about using java.util.concurrent and TPE:
>>> I think that over the long term, it makes sense to (re)use existing
>>> utilities which are being maintained by domain experts rather than
>>> custom utilities you've written yourself. The concurrent libraries
>>> available since Java 5 were written and maintained by people widely
>>> recognized to be very, very good at a very hard problem. That doesn't
>>> mean they, or the library, is perfect, just that there is value in
>>> building on their work and letting them take care of the bugs and
>>> optimizations over time. The downside would be that if a River user
>>> was stuck with, say, Java 5, they couldn't take advantage of bugfixes
>>> or improvements in Java 6. On the other hand, that's true of the
>>> entire JDK.
>>>
>>> The max threads issue seems to me a non-issue. A JVM can allocate only
>>> so many native threads before it runs out of OS resources; that's a
>>> hard limit. You can set a max of Integer.MAX_VALUE but your VM would
>>> die long, long before it reached that.
>>>
>>> For me this is more of design policy decision. Re-use, intelligently
>>> and selectively, where possible, to reduce your project's workload.
>>>
>>>
>>> Patrick
>>
>> +1
>


Re: com.sun.jini.thread lock contention

Posted by Gregg Wonderly <gr...@wonderly.org>.
I am not against the thoughts here.  TaskManager has a very important  
roll that has specific needs.  It is used to dispatch inbound calls  
amongst other things.  It is essential that it never fail to create a  
thread to handle an inbound call.  Without this behavior there is a  
chance for distributed deadlock.

The key clause in TPE is that it says the extra threads are not  
created until the queue is full.  That is another form of "maximum".

If it has a documented behavior that has no maximum, then I am okay  
with using it.  I have just never seen it behave correctly based on  
what I have experienced and so I am concerned about the maximum thread  
issue.

There are of course many types of applications that can use a ceiling  
on maximum threads to throttle things.  But if you look at some of the  
bug reports and discussions on the concurrency-interest list, there  
are cases of this behavior popping up in the fork-join pool stuff too,  
where the complexity of hoe work is divided and distributed creates  
problems.

Gregg Wonderly

Sent from my iPhone

On Jun 5, 2010, at 8:25 AM, Dennis Reedy <de...@gmail.com> wrote:

>
> On Jun 5, 2010, at 609AM, Patrick Wright wrote:
>
>> One point I'd like to raise about using java.util.concurrent and TPE:
>> I think that over the long term, it makes sense to (re)use existing
>> utilities which are being maintained by domain experts rather than
>> custom utilities you've written yourself. The concurrent libraries
>> available since Java 5 were written and maintained by people widely
>> recognized to be very, very good at a very hard problem. That doesn't
>> mean they, or the library, is perfect, just that there is value in
>> building on their work and letting them take care of the bugs and
>> optimizations over time. The downside would be that if a River user
>> was stuck with, say, Java 5, they couldn't take advantage of bugfixes
>> or improvements in Java 6. On the other hand, that's true of the
>> entire JDK.
>>
>> The max threads issue seems to me a non-issue. A JVM can allocate  
>> only
>> so many native threads before it runs out of OS resources; that's a
>> hard limit. You can set a max of Integer.MAX_VALUE but your VM would
>> die long, long before it reached that.
>>
>> For me this is more of design policy decision. Re-use, intelligently
>> and selectively, where possible, to reduce your project's workload.
>>
>>
>> Patrick
>
> +1

Re: com.sun.jini.thread lock contention

Posted by Dennis Reedy <de...@gmail.com>.
On Jun 5, 2010, at 609AM, Patrick Wright wrote:

> One point I'd like to raise about using java.util.concurrent and TPE:
> I think that over the long term, it makes sense to (re)use existing
> utilities which are being maintained by domain experts rather than
> custom utilities you've written yourself. The concurrent libraries
> available since Java 5 were written and maintained by people widely
> recognized to be very, very good at a very hard problem. That doesn't
> mean they, or the library, is perfect, just that there is value in
> building on their work and letting them take care of the bugs and
> optimizations over time. The downside would be that if a River user
> was stuck with, say, Java 5, they couldn't take advantage of bugfixes
> or improvements in Java 6. On the other hand, that's true of the
> entire JDK.
> 
> The max threads issue seems to me a non-issue. A JVM can allocate only
> so many native threads before it runs out of OS resources; that's a
> hard limit. You can set a max of Integer.MAX_VALUE but your VM would
> die long, long before it reached that.
> 
> For me this is more of design policy decision. Re-use, intelligently
> and selectively, where possible, to reduce your project's workload.
> 
> 
> Patrick

+1

Re: com.sun.jini.thread lock contention

Posted by Patrick Wright <pd...@gmail.com>.
One point I'd like to raise about using java.util.concurrent and TPE:
I think that over the long term, it makes sense to (re)use existing
utilities which are being maintained by domain experts rather than
custom utilities you've written yourself. The concurrent libraries
available since Java 5 were written and maintained by people widely
recognized to be very, very good at a very hard problem. That doesn't
mean they, or the library, is perfect, just that there is value in
building on their work and letting them take care of the bugs and
optimizations over time. The downside would be that if a River user
was stuck with, say, Java 5, they couldn't take advantage of bugfixes
or improvements in Java 6. On the other hand, that's true of the
entire JDK.

The max threads issue seems to me a non-issue. A JVM can allocate only
so many native threads before it runs out of OS resources; that's a
hard limit. You can set a max of Integer.MAX_VALUE but your VM would
die long, long before it reached that.

For me this is more of design policy decision. Re-use, intelligently
and selectively, where possible, to reduce your project's workload.


Patrick

Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Gregg is mostly right about the max number of threads and is probably 
commenting from experience. MaxThreads is not a hard limit.  Will have 
to look into it further though to determine if it can be used for DOS, 
it doesn't look like it indefinitely increases the number of threads, 
but it is designed to prevent thread starvation deadlock.

 /** Return true if a new thread should be created (ignoring maxThreads). */
    protected boolean needThread()

TaskManager is quite clever.  I don't think TaskThread should be a 
Runnable rather than a Thread, Task is the Runnable.  TaskThread is used 
to run the Runnable Task.

Try out ConcurrentDynamicPolicyProvider first, then we'll look at how to 
improve it, if needed.

When I look at how Policy, PermissionCollection and Permission was 
designed, it would have been so much cleaner if PermissionCollection 
were immutable.

Cheers,

Peter.

Peter Firmstone wrote:
> Dennis Reedy wrote:
>>> Anytime that you need more than corePoolSize threads, you risk 
>>> seeing a task not being executed immediately.  The fact that there 
>>> is a maximumPoolSize is the bigger issue though.  Only that many 
>>> total threads can ever be running.  If you know enough about all 
>>> parts of all services and clients using your River application, you 
>>> might be able to accurately guess what maximumPoolSize would need to 
>>> be.  But ultimately, in a distributed application where circular 
>>> invocation scenarios are possible, you need a lot of information and 
>>> there has to be a lot of control in the applications to make sure 
>>> that there are never more than maximumPoolSize simultaneous method 
>>> invocations in an instance.
>>>
>>> I don't think that is possible, nor really a good idea to have a 
>>> maximumPoolSize.  A thread pool design, like TaskManager, where only 
>>> a minimum exists, is the type of design we need in River.
>>>     
>>
>> Not sure I agree with this. You should _always_ have an upper bound, 
>> you cannot assume that you have limitless resources. Just creating 
>> threads because an application "says so", is generally not advisable, 
>> and can lead to OOME as well as DOS issues.
>>   
> Ditto.
>
>    /**
>     * Create a task manager with maxThreads = 10, timeout = 15 seconds,
>     * and loadFactor = 3.0.
>     */
>    public TaskManager() {
>    this(10, 1000 * 15, 3.0f);
>    }
>
> public TaskManager(int maxThreads, long timeout, float loadFactor)
>
> Cheers,
>
> Peter.
>


Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Dennis Reedy wrote:
>> Anytime that you need more than corePoolSize threads, you risk seeing a task not being executed immediately.  
>> The fact that there is a maximumPoolSize is the bigger issue though.  Only that many total threads can ever be running.  If you know enough about all parts of all services and clients using your River application, you might be able to accurately guess what maximumPoolSize would need to be.  But ultimately, in a distributed application where circular invocation scenarios are possible, you need a lot of information and there has to be a lot of control in the applications to make sure that there are never more than maximumPoolSize simultaneous method invocations in an instance.
>>
>> I don't think that is possible, nor really a good idea to have a maximumPoolSize.  A thread pool design, like TaskManager, where only a minimum exists, is the type of design we need in River.
>>     
>
> Not sure I agree with this. You should _always_ have an upper bound, you cannot assume that you have limitless resources. Just creating threads because an application "says so", is generally not advisable, and can lead to OOME as well as DOS issues.
>   
Ditto.

    /**
     * Create a task manager with maxThreads = 10, timeout = 15 seconds,
     * and loadFactor = 3.0.
     */
    public TaskManager() {
    this(10, 1000 * 15, 3.0f);
    }

public TaskManager(int maxThreads, long timeout, float loadFactor)

Cheers,

Peter.

Re: com.sun.jini.thread lock contention

Posted by Gregg Wonderly <gr...@wonderly.org>.
I am going to reply to some of these again/more in depth now that I am back at 
my computer keyboard.

Comments below.

Dennis Reedy wrote:
>> Anytime that you need more than corePoolSize threads, you risk seeing 
 >> a task not being executed immediately.
>> The fact that there is a maximumPoolSize is the bigger issue though. 
>> Only that many total threads can ever be running.  If you know enough
>> about all parts of all services and clients using your River application, 
>> you might be able to accurately guess what maximumPoolSize would need to
>> be.  But ultimately, in a distributed application where circular 
>> invocation scenarios are possible, you need a lot of information and 
 >> there has to be a lot of control in the applications to make sure that
>> there are never more than maximumPoolSize simultaneous method invocations
>> in an instance.
>>
>> I don't think that is possible, nor really a good idea to have a
>> maximumPoolSize.  A thread pool design, like TaskManager, where only
>> a minimum exists, is the type of design we need in River.
> 
> Not sure I agree with this. You should _always_ have an upper bound,
> you cannot assume that you have limitless resources. Just creating
> threads because an application "says so", is generally not advisable,
> and can lead to OOME as well as DOS issues.

I won't deny this is an issue.  The bigger problem, is the fact that TaskManager 
is used to create threads for inbound calls.  If you have all of River running 
inside of a single Java process, each river service started under 
ServiceStarter, each one will have its own instance of TaskManager with it's own 
parameterization.

The problem, is how can you possibly know which inbound call is the right one to 
allow in and which ones should not be allowed in?  Which are going to commit the 
transaction, or add the object to the javaspace or cancel the lease etc.

It's not a simple issue, and I contend it's not a job for the individual River 
services nor for the mechanics of remote calls.  It's an application design 
issue.  Distributed systems need something on the outside that manages load and 
handles requests at a higher level as they come into the system so that the 
mechanics of all the services are unimpeded in performing the tasks they are 
designed/need to perform to work correctly.

Gregg Wonderly

Re: com.sun.jini.thread lock contention

Posted by Dennis Reedy <de...@gmail.com>.
> 
> Anytime that you need more than corePoolSize threads, you risk seeing a task not being executed immediately.  
> The fact that there is a maximumPoolSize is the bigger issue though.  Only that many total threads can ever be running.  If you know enough about all parts of all services and clients using your River application, you might be able to accurately guess what maximumPoolSize would need to be.  But ultimately, in a distributed application where circular invocation scenarios are possible, you need a lot of information and there has to be a lot of control in the applications to make sure that there are never more than maximumPoolSize simultaneous method invocations in an instance.
> 
> I don't think that is possible, nor really a good idea to have a maximumPoolSize.  A thread pool design, like TaskManager, where only a minimum exists, is the type of design we need in River.

Not sure I agree with this. You should _always_ have an upper bound, you cannot assume that you have limitless resources. Just creating threads because an application "says so", is generally not advisable, and can lead to OOME as well as DOS issues.




Re: com.sun.jini.thread lock contention

Posted by Gregg Wonderly <gr...@wonderly.org>.
Patrick Wright wrote:
> On Fri, Jun 4, 2010 at 10:11 PM, Gregg Wonderly <gr...@wonderly.org> wrote:
>> I'd have to say no for the existing Executor implementations in the JDK.
>>  The concurrency utilities related to Executor in the JDK are tailored for,
>> and specifically limited to applications where you have unrelated tasks that
>> need to be throttled amongst a limited set of threads.
>>
>> River's TaskManager will always create more threads, but will prune those
>> threads after they sit idle for too long.  We need this behavior to keep
>> away from distributed deadlock which can occur anytime another remote
>> operation might be the only way that progress can happen in the overall
>> processing of a distributed application.
>>
>> Be very careful where you use j.u.c ThreadPoolExecutor et.al, because of
>> their design.  It would be possible to use an appropriate Executor
>> implementation, but it would have to behave like TaskManager and always
>> create a new thread for new work, when no idle threads are available.
> 
> Hi Gregg
> 
> This is not how I understood ThreadPoolExecutor, even from the JavaDoc
> http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ThreadPoolExecutor.html
> 
> "A ThreadPoolExecutor will automatically adjust the pool size (see
> getPoolSize()) according to the bounds set by corePoolSize (see
> getCorePoolSize()) and maximumPoolSize (see getMaximumPoolSize()).
> When a new task is submitted in method execute(java.lang.Runnable),
> and fewer than corePoolSize threads are running, a new thread is
> created to handle the request, even if other worker threads are idle.
> If there are more than corePoolSize but less than maximumPoolSize
> threads running, a new thread will be created only if the queue is
> full."
> 
> and
> 
> "f the pool currently has more than corePoolSize threads, excess
> threads will be terminated if they have been idle for more than the
> keepAliveTime (see getKeepAliveTime(java.util.concurrent.TimeUnit)).
> This provides a means of reducing resource consumption when the pool
> is not being actively used. If the pool becomes more active later, new
> threads will be constructed."
> 
> Under what configuration of TPE would we see a problem?

Anytime that you need more than corePoolSize threads, you risk seeing a task not 
being executed immediately.  The fact that there is a maximumPoolSize is the 
bigger issue though.  Only that many total threads can ever be running.  If you 
know enough about all parts of all services and clients using your River 
application, you might be able to accurately guess what maximumPoolSize would 
need to be.  But ultimately, in a distributed application where circular 
invocation scenarios are possible, you need a lot of information and there has 
to be a lot of control in the applications to make sure that there are never 
more than maximumPoolSize simultaneous method invocations in an instance.

I don't think that is possible, nor really a good idea to have a 
maximumPoolSize.  A thread pool design, like TaskManager, where only a minimum 
exists, is the type of design we need in River.

Even today, I am working on debugging an old application where 
ThreadPoolExecutor was used and the maximumPoolSize is creating extended delays 
in processing until some timeouts occur and fallback to a single threaded 
approach to try and recover.  So, this is an important issue with how TPE was 
designed to work.

Gregg Wonderly

Re: com.sun.jini.thread lock contention

Posted by Patrick Wright <pd...@gmail.com>.
On Fri, Jun 4, 2010 at 10:11 PM, Gregg Wonderly <gr...@wonderly.org> wrote:
> I'd have to say no for the existing Executor implementations in the JDK.
>  The concurrency utilities related to Executor in the JDK are tailored for,
> and specifically limited to applications where you have unrelated tasks that
> need to be throttled amongst a limited set of threads.
>
> River's TaskManager will always create more threads, but will prune those
> threads after they sit idle for too long.  We need this behavior to keep
> away from distributed deadlock which can occur anytime another remote
> operation might be the only way that progress can happen in the overall
> processing of a distributed application.
>
> Be very careful where you use j.u.c ThreadPoolExecutor et.al, because of
> their design.  It would be possible to use an appropriate Executor
> implementation, but it would have to behave like TaskManager and always
> create a new thread for new work, when no idle threads are available.

Hi Gregg

This is not how I understood ThreadPoolExecutor, even from the JavaDoc
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ThreadPoolExecutor.html

"A ThreadPoolExecutor will automatically adjust the pool size (see
getPoolSize()) according to the bounds set by corePoolSize (see
getCorePoolSize()) and maximumPoolSize (see getMaximumPoolSize()).
When a new task is submitted in method execute(java.lang.Runnable),
and fewer than corePoolSize threads are running, a new thread is
created to handle the request, even if other worker threads are idle.
If there are more than corePoolSize but less than maximumPoolSize
threads running, a new thread will be created only if the queue is
full."

and

"f the pool currently has more than corePoolSize threads, excess
threads will be terminated if they have been idle for more than the
keepAliveTime (see getKeepAliveTime(java.util.concurrent.TimeUnit)).
This provides a means of reducing resource consumption when the pool
is not being actively used. If the pool becomes more active later, new
threads will be constructed."

Under what configuration of TPE would we see a problem?

Thanks
Patrick

Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Gregg,

Thanks for sharing these nuggets of wisdom.

Peter.


Shay Hassidim wrote:
> Amen.
>
> This is exactly what we have learned in GigaSpaces when had to address these Jini lock contention issues long time ago. 
>
> Shay
>
> ----- Original Message -----
> From: Gregg Wonderly <gr...@wonderly.org>
> To: river-dev@incubator.apache.org <ri...@incubator.apache.org>
> Sent: Fri Jun 04 13:11:15 2010
> Subject: Re: com.sun.jini.thread lock contention
>
> I'd have to say no for the existing Executor implementations in the JDK.  The 
> concurrency utilities related to Executor in the JDK are tailored for, and 
> specifically limited to applications where you have unrelated tasks that need to 
> be throttled amongst a limited set of threads.
>
> River's TaskManager will always create more threads, but will prune those 
> threads after they sit idle for too long.  We need this behavior to keep away 
> from distributed deadlock which can occur anytime another remote operation might 
> be the only way that progress can happen in the overall processing of a 
> distributed application.
>
> Be very careful where you use j.u.c ThreadPoolExecutor et.al, because of their 
> design.  It would be possible to use an appropriate Executor implementation, but 
> it would have to behave like TaskManager and always create a new thread for new 
> work, when no idle threads are available.
>
> Gregg Wonderly
>
> Dennis Reedy wrote:
>   
>> Hi Gregg,
>>
>> Does it make sense to move to concurrency utilities and move away from TaskManager altogether?
>>
>> Dennis
>>
>> On Jun 4, 2010, at 1100AM, Gregg Wonderly wrote:
>>
>>     
>>> The TaskManager class has a subclass of Thread that it uses to run tasks.  The problem with subclassing Thread, is that there is a lock that is used in Thread to see if some methods are overridden so that security can be maintained.  So, in an environment where Thread is used with a SecurityManager, it is better to use a Runnable to encapsulate the custom code, instead of a Thread.
>>>
>>> The TaskThread class should be a Runnable instead of a Thread, and then there are just 3 other places where it is referenced that need some adjustment.  The thread that is running inside of the runnable needs to be put into a per instance variable for one place where interrupt processing is handled.
>>>
>>> Gregg Wonderly
>>>       
>>     
>
>
>
>
>   


Re: com.sun.jini.thread lock contention

Posted by Shay Hassidim <sh...@gigaspaces.com>.
Amen.

This is exactly what we have learned in GigaSpaces when had to address these Jini lock contention issues long time ago. 

Shay

----- Original Message -----
From: Gregg Wonderly <gr...@wonderly.org>
To: river-dev@incubator.apache.org <ri...@incubator.apache.org>
Sent: Fri Jun 04 13:11:15 2010
Subject: Re: com.sun.jini.thread lock contention

I'd have to say no for the existing Executor implementations in the JDK.  The 
concurrency utilities related to Executor in the JDK are tailored for, and 
specifically limited to applications where you have unrelated tasks that need to 
be throttled amongst a limited set of threads.

River's TaskManager will always create more threads, but will prune those 
threads after they sit idle for too long.  We need this behavior to keep away 
from distributed deadlock which can occur anytime another remote operation might 
be the only way that progress can happen in the overall processing of a 
distributed application.

Be very careful where you use j.u.c ThreadPoolExecutor et.al, because of their 
design.  It would be possible to use an appropriate Executor implementation, but 
it would have to behave like TaskManager and always create a new thread for new 
work, when no idle threads are available.

Gregg Wonderly

Dennis Reedy wrote:
> Hi Gregg,
> 
> Does it make sense to move to concurrency utilities and move away from TaskManager altogether?
> 
> Dennis
> 
> On Jun 4, 2010, at 1100AM, Gregg Wonderly wrote:
> 
>> The TaskManager class has a subclass of Thread that it uses to run tasks.  The problem with subclassing Thread, is that there is a lock that is used in Thread to see if some methods are overridden so that security can be maintained.  So, in an environment where Thread is used with a SecurityManager, it is better to use a Runnable to encapsulate the custom code, instead of a Thread.
>>
>> The TaskThread class should be a Runnable instead of a Thread, and then there are just 3 other places where it is referenced that need some adjustment.  The thread that is running inside of the runnable needs to be put into a per instance variable for one place where interrupt processing is handled.
>>
>> Gregg Wonderly
> 
> 




Re: com.sun.jini.thread lock contention

Posted by Gregg Wonderly <gr...@wonderly.org>.
I'd have to say no for the existing Executor implementations in the JDK.  The 
concurrency utilities related to Executor in the JDK are tailored for, and 
specifically limited to applications where you have unrelated tasks that need to 
be throttled amongst a limited set of threads.

River's TaskManager will always create more threads, but will prune those 
threads after they sit idle for too long.  We need this behavior to keep away 
from distributed deadlock which can occur anytime another remote operation might 
be the only way that progress can happen in the overall processing of a 
distributed application.

Be very careful where you use j.u.c ThreadPoolExecutor et.al, because of their 
design.  It would be possible to use an appropriate Executor implementation, but 
it would have to behave like TaskManager and always create a new thread for new 
work, when no idle threads are available.

Gregg Wonderly

Dennis Reedy wrote:
> Hi Gregg,
> 
> Does it make sense to move to concurrency utilities and move away from TaskManager altogether?
> 
> Dennis
> 
> On Jun 4, 2010, at 1100AM, Gregg Wonderly wrote:
> 
>> The TaskManager class has a subclass of Thread that it uses to run tasks.  The problem with subclassing Thread, is that there is a lock that is used in Thread to see if some methods are overridden so that security can be maintained.  So, in an environment where Thread is used with a SecurityManager, it is better to use a Runnable to encapsulate the custom code, instead of a Thread.
>>
>> The TaskThread class should be a Runnable instead of a Thread, and then there are just 3 other places where it is referenced that need some adjustment.  The thread that is running inside of the runnable needs to be put into a per instance variable for one place where interrupt processing is handled.
>>
>> Gregg Wonderly
> 
> 


Re: com.sun.jini.thread lock contention

Posted by Dennis Reedy <de...@gmail.com>.
Hi Gregg,

Does it make sense to move to concurrency utilities and move away from TaskManager altogether?

Dennis

On Jun 4, 2010, at 1100AM, Gregg Wonderly wrote:

> The TaskManager class has a subclass of Thread that it uses to run tasks.  The problem with subclassing Thread, is that there is a lock that is used in Thread to see if some methods are overridden so that security can be maintained.  So, in an environment where Thread is used with a SecurityManager, it is better to use a Runnable to encapsulate the custom code, instead of a Thread.
> 
> The TaskThread class should be a Runnable instead of a Thread, and then there are just 3 other places where it is referenced that need some adjustment.  The thread that is running inside of the runnable needs to be put into a per instance variable for one place where interrupt processing is handled.
> 
> Gregg Wonderly


Re: com.sun.jini.thread lock contention

Posted by Peter Firmstone <ji...@zeus.net.au>.
Gregg Wonderly wrote:
> The TaskManager class has a subclass of Thread that it uses to run 
> tasks.  The problem with subclassing Thread, is that there is a lock 
> that is used in Thread to see if some methods are overridden so that 
> security can be maintained.  So, in an environment where Thread is 
> used with a SecurityManager, it is better to use a Runnable to 
> encapsulate the custom code, instead of a Thread.

Makes sense.

>
> The TaskThread class should be a Runnable instead of a Thread, and 
> then there are just 3 other places where it is referenced that need 
> some adjustment.  The thread that is running inside of the runnable 
> needs to be put into a per instance variable for one place where 
> interrupt processing is handled.
>
> Gregg Wonderly
>
Got a patch?

Cheers,

Peter.