You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Peter Firmstone <ji...@zeus.net.au> on 2012/11/24 04:02:01 UTC

Deadlock in MultiCastMonitorStopReplace test

Anyone have any thoughts on the following?

The attached patch fixes some visibility issues, I'm now seeing a 
deadlock in MulticastMonitorStopReplace.

I've also attached the output from the jenkins arm build, without the 
patch applied, where only 2 out of 3 events are recieved, the missing 
one was interrupted before its member group could be changed, other 
failing tests have the same issue, execept all are interrupted.

    BaseQATest.replaceMemberGroups FINE:  lookup service 0 - replacing member groups with -- 
    BaseQATest.replaceMemberGroups FINE:  failed to change member groups for lookup service 0
    java.rmi.ConnectIOException: I/O exception connecting to BasicObjectEndpoint[37abfa80-a6e1-4340-9cdb-37caccf52d5a,TcpEndpoint[64.20.246.146:46371]]; nested exception is: 
    	java.io.IOException: interrupt waiting for connection header
    	at net.jini.jeri.BasicInvocationHandler.wrapSafeIOException(BasicInvocationHandler.java:893)
    	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:711)
    	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:659)
    	at net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:528)
    	at com.sun.jini.test.services.lookupsimulator.$Proxy0.getMemberGroups(Unknown Source)
    	at com.sun.jini.test.services.lookupsimulator.LookupSimulatorProxy.getMemberGroups(LookupSimulatorProxy.java:144)
    	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2578)
    	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2628)
    	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2660)
    	at com.sun.jini.test.spec.lookupdiscovery.MulticastMonitorStopReplace.run(MulticastMonitorStopReplace.java:103)
    	at com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:254)
    	at com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
    Caused by: java.io.IOException: interrupt waiting for connection header
    	at com.sun.jini.jeri.internal.mux.Mux.start(Mux.java:238)
    	at com.sun.jini.jeri.internal.mux.MuxClient.start(MuxClient.java:35)
    	at net.jini.jeri.connection.ConnectionManager$OutboundMux.newRequest(ConnectionManager.java:375)
    	at net.jini.jeri.connection.ConnectionManager$ReqIterator.next(ConnectionManager.java:665)
    	at net.jini.jeri.BasicObjectEndpoint$1.next(BasicObjectEndpoint.java:370)
    	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:708)
    	... 10 more
    Caused by: java.lang.InterruptedException
    	at java.lang.Object.wait(Native Method)
    	at com.sun.jini.jeri.internal.mux.Mux.start(Mux.java:230)
    	... 15 more
    BaseQATest.replaceMemberGroups FINE:  lookup service 1 - replacing member groups with -- 
    BaseQATest.replaceMemberGroups FINE:    newGroups[0] = LDGroup2_A_je-cal-17_1353590184346_new
    BaseQATest.replaceMemberGroups FINE:    newGroups[1] = LDGroup2_B_je-cal-17_1353590184346_new
    BaseQATest.replaceMemberGroups FINE:    newGroups[2] = LDGroup2_C_je-cal-17_1353590184346_new
    BaseQATest.replaceMemberGroups FINE:  lookup service 2 - replacing member groups with -- 
    BaseQATest.replaceMemberGroups FINE:    newGroups[0] = LDGroup1_A_je-cal-17_1353590184346_new
    BaseQATest.replaceMemberGroups FINE:    newGroups[1] = LDGroup1_B_je-cal-17_1353590184346_new
    BaseQATest.replaceMemberGroups FINE:    newGroups[2] = LDGroup1_C_je-cal-17_1353590184346_new
    BaseQATest.replaceMemberGroups FINE:    newGroups[3] = LDGroup1_D_je-cal-17_1353590184346_new
    BaseQATest.waitForDiscard FINE:  for DISCARD events -- waiting at least 30 seconds, but no more than 180 seconds ...
      

The following deadlock is experienced on a 4 way sparc:

    Running
    com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td

    $ jstack 4568
    2012-11-24 12:24:23
    Full thread dump Java HotSpot(TM) Server VM (20.5-b03 mixed mode):

    "Attach Listener" daemon prio=3 tid=0x00889800 nid=0x35 waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "(JSK) Idle" daemon prio=3 tid=0x008b8c00 nid=0x1e in Object.wait()
    [0xb5c7f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb887910> (a java.lang.Object)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
            - locked <0xbb887910> (a java.lang.Object)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) Idle" daemon prio=3 tid=0x00289400 nid=0x1d in Object.wait()
    [0xb5d7f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb88b9d8> (a java.lang.Object)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
            - locked <0xbb88b9d8> (a java.lang.Object)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) Idle" daemon prio=3 tid=0x00381800 nid=0x1c in Object.wait()
    [0xb5e7f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb887910> (a java.lang.Object)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
            - locked <0xbb887910> (a java.lang.Object)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) mux reader" daemon prio=3 tid=0x00517c00 nid=0x1b runnable
    [0xb667f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:129)
            at
    com.sun.jini.jeri.internal.mux.StreamConnectionIO$1.read(StreamConnectionIO.java:355)
            at
    com.sun.jini.jeri.internal.mux.StreamConnectionIO$Reader.run(StreamConnectionIO.java:262)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:140)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) Idle" daemon prio=3 tid=0x006a1400 nid=0x1a in Object.wait()
    [0xb5f7f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb88b9d8> (a java.lang.Object)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
            - locked <0xbb88b9d8> (a java.lang.Object)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) mux writer" daemon prio=3 tid=0x0085c800 nid=0x19 in
    Object.wait() [0xb607f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb9b5aa0> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:485)
            at
    com.sun.jini.jeri.internal.mux.StreamConnectionIO$Writer.run(StreamConnectionIO.java:168)
            - locked <0xbb9b5aa0> (a java.lang.Object)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:140)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) Idle" daemon prio=3 tid=0x006cb000 nid=0x18 in Object.wait()
    [0xb617f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb887910> (a java.lang.Object)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
            - locked <0xbb887910> (a java.lang.Object)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) KeepAlive" prio=3 tid=0x003c2800 nid=0x17 waiting on
    condition [0xb627f000]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
            at java.lang.Thread.sleep(Native Method)
            at
    com.sun.jini.jeri.internal.runtime.JvmLifeSupport$2.run(JvmLifeSupport.java:130)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) Reaper" daemon prio=3 tid=0x003c1400 nid=0x16 in
    Object.wait() [0xb637f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb887a48> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xbb887a48> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
            at
    com.sun.jini.jeri.internal.runtime.ImplRefManager$Reaper.run(ImplRefManager.java:426)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK)
    TcpServerEndpoint.LH[ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=33781]]
    accept loop" daemon prio=3 tid=0x003be800 nid=0x15 runnable [0xb647f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xbb893860> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at
    net.jini.jeri.tcp.TcpServerEndpoint$LH.executeAcceptLoop(TcpServerEndpoint.java:798)
            at
    net.jini.jeri.tcp.TcpServerEndpoint$LH.access$400(TcpServerEndpoint.java:736)
            at
    net.jini.jeri.tcp.TcpServerEndpoint$LH$1.run(TcpServerEndpoint.java:768)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:140)
            at java.lang.Thread.run(Thread.java:662)

    "RMI Scheduler(0)" daemon prio=3 tid=0x0043b000 nid=0x14 waiting on
    condition [0xb657f000]
       java.lang.Thread.State: TIMED_WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0xbb886590> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
            at
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
            at
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:662)

    "DestroyJavaVM" prio=3 tid=0x00031800 nid=0x2 waiting on condition
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Thread-1" prio=3 tid=0x0043fc00 nid=0x12 waiting on condition
    [0xb677f000]
       java.lang.Thread.State: TIMED_WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0xbb8954b8> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
            at
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
            at
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:662)

    "GC Daemon" daemon prio=3 tid=0x00512800 nid=0x11 in Object.wait()
    [0xb687f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb804060> (a sun.misc.GC$LatencyLock)
            at sun.misc.GC$Daemon.run(GC.java:100)
            - locked <0xbb804060> (a sun.misc.GC$LatencyLock)

    "RMI Reaper" prio=3 tid=0x00555000 nid=0x10 in Object.wait()
    [0xb697f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
            at
    sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
            at java.lang.Thread.run(Thread.java:662)

    "RMI TCP Accept-0" daemon prio=3 tid=0x00554000 nid=0xf runnable
    [0xb6a7f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xbb8001a8> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at
    sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
            at
    sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
            at java.lang.Thread.run(Thread.java:662)

    "Low Memory Detector" daemon prio=3 tid=0x00123c00 nid=0xd runnable
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread1" daemon prio=3 tid=0x00121000 nid=0xc waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread0" daemon prio=3 tid=0x0011e800 nid=0xb waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Signal Dispatcher" daemon prio=3 tid=0x0011d000 nid=0xa runnable
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Finalizer" daemon prio=3 tid=0x00109800 nid=0x9 in Object.wait()
    [0xb707f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb800518> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xbb800518> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
            at
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

    "Reference Handler" daemon prio=3 tid=0x00107c00 nid=0x8 in
    Object.wait() [0xb717f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb804050> (a java.lang.ref.Reference$Lock)
            at java.lang.Object.wait(Object.java:485)
            at
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
            - locked <0xbb804050> (a java.lang.ref.Reference$Lock)

    "VM Thread" prio=3 tid=0x00104000 nid=0x7 runnable

    "GC task thread#0 (ParallelGC)" prio=3 tid=0x00039400 nid=0x3 runnable

    "GC task thread#1 (ParallelGC)" prio=3 tid=0x0003a800 nid=0x4 runnable

    "GC task thread#2 (ParallelGC)" prio=3 tid=0x0003bc00 nid=0x5 runnable

    "GC task thread#3 (ParallelGC)" prio=3 tid=0x0003d000 nid=0x6 runnable

    "VM Periodic Task Thread" prio=3 tid=0x00136000 nid=0xe waiting on
    condition

    JNI global references: 1576



Re: Deadlock in MultiCastMonitorStopReplace test

Posted by Peter Firmstone <ji...@zeus.net.au>.
Peter Firmstone wrote:
> Ok,
>
> The unicast socket didn't have a timeout set, the harness was stuck on 
> .join() waiting for the socket to close.
>
> The tests still failing but it's no longer hanging.
>
> Cheers,
>
> Peter.

I set a timeout on the socket, the test fails and exits, but now I've 
got a stale processes.  I've fixed some concurrency / visibility issues 
in the patches attached.

I'm not going to be able to play with this again until next weekend, 
does anyone else have time to look into it?  It's almost as if the 
client and server are attempting to communicate on different ports or 
something?

Thanks, Peter.

This looks like a timing or Socket issue, there no data (test output):

     [java] java.rmi.UnmarshalException: exception unmarshalling 
response; nested exception is:
     [java]     java.io.IOException: I/O error reading from mux 
connection: java.io.EOFException
     [java]     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:847)
     [java]     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:659)
     [java]     at 
net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:528)
     [java]     at 
com.sun.jini.test.services.lookupsimulator.$Proxy0.getLocator(Unknown 
Source)
     [java]     at 
com.sun.jini.test.services.lookupsimulator.LookupSimulatorProxy.getLocator(LookupSimulatorProxy.java:134)
     [java]     at 
com.sun.jini.test.share.BaseQATest.startLookup(BaseQATest.java:1722)
     [java]     at 
com.sun.jini.test.share.BaseQATest.startInitLookups(BaseQATest.java:1538)
     [java]     at 
com.sun.jini.test.share.BaseQATest.setup(BaseQATest.java:1063)
     [java]     at 
com.sun.jini.test.spec.lookupdiscovery.AbstractBaseTest.setup(AbstractBaseTest.java:77)
     [java]     at 
com.sun.jini.test.spec.lookupdiscovery.Discovered.setup(Discovered.java:69)
     [java]     at 
com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:219)
     [java]     at 
com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
     [java] Caused by: java.io.IOException: I/O error reading from mux 
connection: java.io.EOFException
     [java]     at 
com.sun.jini.jeri.internal.mux.Session$MuxInputStream.read(Session.java:874)
     [java]     at 
net.jini.jeri.connection.ConnectionManager$Outbound$Input.read(ConnectionManager.java:550)
     [java]     at 
net.jini.jeri.BasicObjectEndpoint.executeCall(BasicObjectEndpoint.java:410)
     [java]     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:806)
     [java]     ... 11 more
     [java] Caused by: java.io.EOFException
     [java]     at 
com.sun.jini.jeri.internal.mux.StreamConnectionIO$Reader.run(StreamConnectionIO.java:264)
     [java]     at 
com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:140)
     [java]     at 
com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:174)
     [java]     at java.lang.Thread.run(Thread.java:662)
     [java]
     [java] TIME: 9:39:51 PM
     [java]
     [java] Test process was destroyed and returned code 1
     [java] 
com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td
     [java] Test Failed: Unexpected Exception in setup: 
java.rmi.UnmarshalException: exception unmarshalling response; nested 
exception is:
     [java]     java.io.IOException: I/O error reading from mux 
connection: java.io.EOFException


The remaining phantom process:

$ jstack 5001
2012-11-25 21:41:19
Full thread dump Java HotSpot(TM) Server VM (20.5-b03 mixed mode):

"Attach Listener" daemon prio=3 tid=0x00196400 nid=0x1e waiting on 
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"(JSK) Idle" daemon prio=3 tid=0x00394800 nid=0x1d waiting on condition 
[0xb5c7f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xe7939b30> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
        at com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:191)
        at java.lang.Thread.run(Thread.java:662)

"(JSK) Idle" daemon prio=3 tid=0x006e9800 nid=0x1c waiting on condition 
[0xb5d7f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xe7900070> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
        at com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:191)
        at java.lang.Thread.run(Thread.java:662)

"(JSK) Idle" daemon prio=3 tid=0x00675000 nid=0x1b waiting on condition 
[0xb5e7f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xe7900070> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
        at com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:191)
        at java.lang.Thread.run(Thread.java:662)

"(JSK) Idle" daemon prio=3 tid=0x00195800 nid=0x1a waiting on condition 
[0xb5f7f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xe7900070> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
        at com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:191)
        at java.lang.Thread.run(Thread.java:662)

"(JSK) Idle" daemon prio=3 tid=0x006fd800 nid=0x19 waiting on condition 
[0xb607f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xe7939b30> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
        at com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:191)
        at java.lang.Thread.run(Thread.java:662)

"(JSK) Idle" daemon prio=3 tid=0x003ed400 nid=0x18 waiting on condition 
[0xb617f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xe7939b30> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
        at com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:191)
        at java.lang.Thread.run(Thread.java:662)

"(JSK) KeepAlive" prio=3 tid=0x00417000 nid=0x17 waiting on condition 
[0xb627f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at 
com.sun.jini.jeri.internal.runtime.JvmLifeSupport$2.run(JvmLifeSupport.java:130)
        at java.lang.Thread.run(Thread.java:662)

"(JSK) Reaper" daemon prio=3 tid=0x00593c00 nid=0x16 in Object.wait() 
[0xb637f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe7904308> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xe7904308> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at 
com.sun.jini.jeri.internal.runtime.ImplRefManager$Reaper.run(ImplRefManager.java:426)
        at java.lang.Thread.run(Thread.java:662)

"(JSK) 
TcpServerEndpoint.LH[ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=34300]] 
accept loop" daemon prio=3 tid=0x00593400 nid=0x15 runnable [0xb647f000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
        - locked <0xe7905620> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:462)
        at java.net.ServerSocket.accept(ServerSocket.java:430)
        at 
net.jini.jeri.tcp.TcpServerEndpoint$LH.executeAcceptLoop(TcpServerEndpoint.java:798)
        at 
net.jini.jeri.tcp.TcpServerEndpoint$LH.access$400(TcpServerEndpoint.java:736)
        at 
net.jini.jeri.tcp.TcpServerEndpoint$LH$1.run(TcpServerEndpoint.java:768)
        at com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:140)
        at com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:174)
        at java.lang.Thread.run(Thread.java:662)

"RMI Scheduler(0)" daemon prio=3 tid=0x00410000 nid=0x14 waiting on 
condition [0xb657f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xbb8824b8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

"DestroyJavaVM" prio=3 tid=0x00031800 nid=0x2 waiting on condition 
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"Thread-1" prio=3 tid=0x00589000 nid=0x12 waiting on condition [0xb677f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xe7920a90> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

"GC Daemon" daemon prio=3 tid=0x00606400 nid=0x11 in Object.wait() 
[0xb687f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xbb808038> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:100)
        - locked <0xbb808038> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=3 tid=0x002e0c00 nid=0x10 in Object.wait() [0xb697f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
        at java.lang.Thread.run(Thread.java:662)

"RMI TCP Accept-0" daemon prio=3 tid=0x003c0c00 nid=0xf runnable 
[0xb6a7f000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
        - locked <0xbb8001a8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:462)
        at java.net.ServerSocket.accept(ServerSocket.java:430)
        at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=3 tid=0x00123c00 nid=0xd runnable 
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=3 tid=0x00121000 nid=0xc waiting on 
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=3 tid=0x0011e800 nid=0xb waiting on 
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=3 tid=0x0011d000 nid=0xa runnable 
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=3 tid=0x0010cc00 nid=0x9 in Object.wait() 
[0xb707f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xbb804230> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xbb804230> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=3 tid=0x00107c00 nid=0x8 in 
Object.wait() [0xb717f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xbb800358> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0xbb800358> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=3 tid=0x00104000 nid=0x7 runnable

"GC task thread#0 (ParallelGC)" prio=3 tid=0x00039400 nid=0x3 runnable

"GC task thread#1 (ParallelGC)" prio=3 tid=0x0003a800 nid=0x4 runnable

"GC task thread#2 (ParallelGC)" prio=3 tid=0x0003bc00 nid=0x5 runnable

"GC task thread#3 (ParallelGC)" prio=3 tid=0x0003d000 nid=0x6 runnable

"VM Periodic Task Thread" prio=3 tid=0x0012e000 nid=0xe waiting on condition

JNI global references: 1449

$


Re: Deadlock in MultiCastMonitorStopReplace test

Posted by Peter Firmstone <ji...@zeus.net.au>.
Ok,

The unicast socket didn't have a timeout set, the harness was stuck on 
.join() waiting for the socket to close.

The tests still failing but it's no longer hanging.

Cheers,

Peter.


Peter Firmstone wrote:
> Ok this time I've got 3 separate jvm stacks and a stack trace, and the 
> test is deadlocked (cpu idle):
>
>     [java] Running 
> com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td
>     [java] Time is Sat Nov 24 16:47:24 EST 2012
>     [java] Starting test in separate process with command:
>     [java] /usr/jdk/jdk1.6.0_30/jre/bin/java 
> -Djava.security.manager=org.apache.river.api.security.CombinerSecurityManager 
> -Djava.security.policy=file:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/harness/policy/defaulttest.policy 
> -cp 
> /opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib/jiniharness.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib/jinitests.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib/jsk-platform.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib/jsk-lib.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib/high-scale-lib.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib/reference-collections-1.0.1.jar 
> -client 
> -Djava.ext.dirs=/usr/jdk/jdk1.6.0_30/jre/lib/ext:/usr/jdk/packages/lib/ext:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib-ext:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib-ext 
> -Dcom.sun.jini.jsk.port=9080 -Dcom.sun.jini.qa.port=9081 
> -Dcom.sun.jini.jsk.home=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy 
> -Dcom.sun.jini.qa.home=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa 
> -Dcom.sun.jini.qa.harness.harnessJar=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib/jiniharness.jar 
> -Dcom.sun.jini.qa.harness.testJar=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib/jinitests.jar 
> -Dcom.sun.jini.qa.harness.runjiniserver=true 
> -Dcom.sun.jini.qa.harness.runkitserver=true 
> -Djava.security.properties=file:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/harness/trust/dynamic-policy.properties 
> -Dcom.sun.jini.qa.harness.testhosts= 
> -Djava.util.logging.config.file=/home/peter/logging.properties 
> -Dcom.sun.jini.test.home=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa 
> -Dcom.sun.jini.test.port=9082 
> -Dcom.sun.jini.qa.harness.policies=file:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/src/com/sun/jini/test/resources/jinitest.policy 
> -Djava.ext.dirs=/usr/jdk/jdk1.6.0_30/jre/lib/ext:/usr/jdk/packages/lib/ext:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib-ext:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib-ext 
> com.sun.jini.qa.harness.MasterTest 
> com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td
>
>     [java] java.rmi.UnmarshalException: exception unmarshalling 
> response; nested exception is:
>     [java]     java.io.IOException: I/O error reading from mux 
> connection: java.io.EOFException
>     [java]     at 
> net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:847) 
>
>     [java]     at 
> net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:659) 
>
>     [java]     at 
> net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:528) 
>
>     [java]     at 
> com.sun.jini.test.services.lookupsimulator.$Proxy0.getLocator(Unknown 
> Source)
>     [java]     at 
> com.sun.jini.test.services.lookupsimulator.LookupSimulatorProxy.getLocator(LookupSimulatorProxy.java:134) 
>
>     [java]     at 
> com.sun.jini.test.share.BaseQATest.startLookup(BaseQATest.java:1722)
>     [java]     at 
> com.sun.jini.test.share.BaseQATest.startInitLookups(BaseQATest.java:1538)
>     [java]     at 
> com.sun.jini.test.share.BaseQATest.setup(BaseQATest.java:1063)
>     [java]     at 
> com.sun.jini.test.spec.lookupdiscovery.AbstractBaseTest.setup(AbstractBaseTest.java:76) 
>
>     [java]     at 
> com.sun.jini.test.spec.lookupdiscovery.Discovered.setup(Discovered.java:69) 
>
>     [java]     at 
> com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:219)
>     [java]     at 
> com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
>     [java] Caused by: java.io.IOException: I/O error reading from mux 
> connection: java.io.EOFException
>     [java]     at 
> com.sun.jini.jeri.internal.mux.Session$MuxInputStream.read(Session.java:874) 
>
>     [java]     at 
> net.jini.jeri.connection.ConnectionManager$Outbound$Input.read(ConnectionManager.java:550) 
>
>     [java]     at 
> net.jini.jeri.BasicObjectEndpoint.executeCall(BasicObjectEndpoint.java:410) 
>
>     [java]     at 
> net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:806) 
>
>     [java]     ... 11 more
>     [java] Caused by: java.io.EOFException
>     [java]     at 
> com.sun.jini.jeri.internal.mux.StreamConnectionIO$Reader.run(StreamConnectionIO.java:264) 
>
>     [java]     at 
> com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:140)
>     [java]     at 
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:174)
>     [java]     at java.lang.Thread.run(Thread.java:662)
>
>    $ jstack 19013
>    2012-11-24 17:09:27
>    Full thread dump Java HotSpot(TM) Client VM (20.5-b03 mixed mode,
>    sharing):
>
>    "Attach Listener" daemon prio=3 tid=0x001cf400 nid=0x23 waiting on
>    condition [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "unicast request" daemon prio=3 tid=0x0020d400 nid=0x1f runnable
>    [0xfa97f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.net.PlainSocketImpl.socketAccept(Native Method)
>            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>            - locked <0xc44f0090> (a java.net.SocksSocketImpl)
>            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>            at java.net.ServerSocket.accept(ServerSocket.java:430)
>            at
>    
> com.sun.jini.test.share.DiscoveryProtocolSimulator$UnicastThread.run(DiscoveryProtocolSimulator.java:616) 
>
>
>    "RMI Scheduler(0)" daemon prio=3 tid=0x001e6800 nid=0x14 waiting on
>    condition [0xfb07f000]
>       java.lang.Thread.State: WAITING (parking)
>            at sun.misc.Unsafe.park(Native Method)
>            - parking to wait for  <0xc44f0220> (a
>    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>            at
>    java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>            at
>    
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) 
>
>            at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) 
>
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "GC Daemon" daemon prio=3 tid=0x001e1c00 nid=0x13 in Object.wait()
>    [0xfb17f000]
>       java.lang.Thread.State: TIMED_WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xc95ef040> (a sun.misc.GC$LatencyLock)
>            at sun.misc.GC$Daemon.run(GC.java:100)
>            - locked <0xc95ef040> (a sun.misc.GC$LatencyLock)
>
>    "RMI RenewClean-[10.1.1.2:34487]" daemon prio=3 tid=0x00196800
>    nid=0x12 in Object.wait() [0xfb27f000]
>       java.lang.Thread.State: TIMED_WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xc95eec50> (a 
> java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>            - locked <0xc95eec50> (a java.lang.ref.ReferenceQueue$Lock)
>            at
>    
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "NonActivatableGroup_system-out" daemon prio=3 tid=0x0023c400
>    nid=0x10 runnable [0xfb37f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.io.FileInputStream.readBytes(Native Method)
>            at java.io.FileInputStream.read(FileInputStream.java:220)
>            at
>    
> java.lang.UNIXProcess$DeferredCloseInputStream.read(UNIXProcess.java:227)
>            at
>    java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>            at
>    java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>            at
>    java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>            - locked <0xc95f1440> (a java.io.BufferedInputStream)
>            at java.io.FilterInputStream.read(FilterInputStream.java:90)
>            at com.sun.jini.qa.harness.Pipe.run(Pipe.java:119)
>            at java.lang.Thread.run(Thread.java:662)
>
>    "process reaper" daemon prio=3 tid=0x00239800 nid=0xf runnable
>    [0xfb47f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.lang.UNIXProcess.waitForProcessExit(Native Method)
>            at java.lang.UNIXProcess.access$900(UNIXProcess.java:17)
>            at java.lang.UNIXProcess$2$1.run(UNIXProcess.java:86)
>
>    "Thread-5" prio=3 tid=0x00232400 nid=0xe runnable [0xfb57f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.net.PlainSocketImpl.socketAccept(Native Method)
>            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>            - locked <0xc95f16d0> (a java.net.SocksSocketImpl)
>            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>            at java.net.ServerSocket.accept(ServerSocket.java:430)
>            at com.sun.jini.tool.ClassServer.run(ClassServer.java:397)
>
>    "Thread-4" prio=3 tid=0x00230800 nid=0xd runnable [0xfb67f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.net.PlainSocketImpl.socketAccept(Native Method)
>            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>            - locked <0xc95f1900> (a java.net.SocksSocketImpl)
>            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>            at java.net.ServerSocket.accept(ServerSocket.java:430)
>            at com.sun.jini.tool.ClassServer.run(ClassServer.java:397)
>
>    "Thread-3" prio=3 tid=0x00226800 nid=0xc runnable [0xfb77f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.net.PlainSocketImpl.socketAccept(Native Method)
>            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>            - locked <0xc95f1b30> (a java.net.SocksSocketImpl)
>            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>            at java.net.ServerSocket.accept(ServerSocket.java:430)
>            at com.sun.jini.tool.ClassServer.run(ClassServer.java:397)
>
>    "no heart without soul" daemon prio=3 tid=0x00221800 nid=0xb waiting
>    on condition [0xfb87f000]
>       java.lang.Thread.State: TIMED_WAITING (sleeping)
>            at java.lang.Thread.sleep(Native Method)
>            at
>    
> com.sun.jini.qa.harness.HeartOfTheMachine.ticktack(HeartOfTheMachine.java:63) 
>
>            at
>    
> com.sun.jini.qa.harness.HeartOfTheMachine.access$000(HeartOfTheMachine.java:29) 
>
>            at
>    
> com.sun.jini.qa.harness.HeartOfTheMachine$1.run(HeartOfTheMachine.java:45) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "Low Memory Detector" daemon prio=3 tid=0x00169000 nid=0x9 runnable
>    [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "C1 CompilerThread0" daemon prio=3 tid=0x00166800 nid=0x8 waiting on
>    condition [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "Signal Dispatcher" daemon prio=3 tid=0x0015c000 nid=0x7 runnable
>    [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "Thread-1" prio=3 tid=0x00150000 nid=0x6 waiting on condition
>    [0xfbd7f000]
>       java.lang.Thread.State: TIMED_WAITING (parking)
>            at sun.misc.Unsafe.park(Native Method)
>            - parking to wait for  <0xc9565ed0> (a
>    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>            at
>    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>            at
>    
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) 
>
>            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) 
>
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "Finalizer" daemon prio=3 tid=0x00095000 nid=0x5 in Object.wait()
>    [0xfbe7f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xc9566028> (a 
> java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>            - locked <0xc9566028> (a java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>            at
>    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
>    "Reference Handler" daemon prio=3 tid=0x00093c00 nid=0x4 in
>    Object.wait() [0xfbf7f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xc95660b8> (a java.lang.ref.Reference$Lock)
>            at java.lang.Object.wait(Object.java:485)
>            at
>    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>            - locked <0xc95660b8> (a java.lang.ref.Reference$Lock)
>
>    "main" prio=3 tid=0x00031c00 nid=0x2 in Object.wait() [0xfe6ff000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xc44f0010> (a
>    com.sun.jini.test.share.DiscoveryProtocolSimulator$UnicastThread)
>            at java.lang.Thread.join(Thread.java:1186)
>            - locked <0xc44f0010> (a
>    com.sun.jini.test.share.DiscoveryProtocolSimulator$UnicastThread)
>            at java.lang.Thread.join(Thread.java:1239)
>            at
>    
> com.sun.jini.test.share.DiscoveryProtocolSimulator.stopAnnouncements(DiscoveryProtocolSimulator.java:311) 
>
>            at
>    
> com.sun.jini.test.share.DiscoveryProtocolSimulator.stopAnnouncements(DiscoveryProtocolSimulator.java:298) 
>
>            at
>    com.sun.jini.test.share.BaseQATest.tearDown(BaseQATest.java:1090)
>            - locked <0xc95f6b88> (a 
> java.util.Collections$SynchronizedMap)
>            at
>    
> com.sun.jini.test.spec.lookupdiscovery.AbstractBaseTest.tearDown(AbstractBaseTest.java:111) 
>
>            at
>    com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:280)
>            at 
> com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
>
>    "VM Thread" prio=3 tid=0x0004dc00 nid=0x3 runnable
>
>    "VM Periodic Task Thread" prio=3 tid=0x0016cc00 nid=0xa waiting on
>    condition
>
>    JNI global references: 929
>
>    $ jstack 19008
>    2012-11-24 17:09:58
>    Full thread dump Java HotSpot(TM) Server VM (20.5-b03 mixed mode):
>
>    "Attach Listener" daemon prio=3 tid=0x0049a400 nid=0x15 waiting on
>    condition [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "test-err" daemon prio=3 tid=0x00315000 nid=0x14 runnable [0xb64ff000]
>       java.lang.Thread.State: RUNNABLE
>            at java.io.FileInputStream.readBytes(Native Method)
>            at java.io.FileInputStream.read(FileInputStream.java:198)
>            at
>    
> java.lang.UNIXProcess$DeferredCloseInputStream.read(UNIXProcess.java:218)
>            at com.sun.jini.qa.harness.Pipe.run(Pipe.java:119)
>            at java.lang.Thread.run(Thread.java:662)
>
>    "test-out" daemon prio=3 tid=0x001f6800 nid=0x13 runnable [0xb65ff000]
>       java.lang.Thread.State: RUNNABLE
>            at java.io.FileInputStream.readBytes(Native Method)
>            at java.io.FileInputStream.read(FileInputStream.java:220)
>            at
>    
> java.lang.UNIXProcess$DeferredCloseInputStream.read(UNIXProcess.java:227)
>            at
>    java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>            at
>    java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>            at
>    java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>            - locked <0xe69b29a8> (a java.io.BufferedInputStream)
>            at java.io.FilterInputStream.read(FilterInputStream.java:90)
>            at com.sun.jini.qa.harness.Pipe.run(Pipe.java:119)
>            at java.lang.Thread.run(Thread.java:662)
>
>    "process reaper" daemon prio=3 tid=0x001f6400 nid=0x12 runnable
>    [0xb66ff000]
>       java.lang.Thread.State: RUNNABLE
>            at java.lang.UNIXProcess.waitForProcessExit(Native Method)
>            at java.lang.UNIXProcess.access$900(UNIXProcess.java:17)
>            at java.lang.UNIXProcess$2$1.run(UNIXProcess.java:86)
>
>    "Thread-1" prio=3 tid=0x0025e400 nid=0x11 waiting on condition
>    [0xb67ff000]
>       java.lang.Thread.State: TIMED_WAITING (parking)
>            at sun.misc.Unsafe.park(Native Method)
>            - parking to wait for  <0xe68ff988> (a
>    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>            at
>    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>            at
>    
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) 
>
>            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) 
>
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "Keepalive" daemon prio=3 tid=0x00260000 nid=0x10 runnable 
> [0xb697f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.net.PlainSocketImpl.socketAccept(Native Method)
>            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>            - locked <0xe660c318> (a java.net.SocksSocketImpl)
>            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>            at java.net.ServerSocket.accept(ServerSocket.java:430)
>            at
>    
> com.sun.jini.qa.harness.MasterHarness$KeepAlivePort.run(MasterHarness.java:261) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "no heart without soul" daemon prio=3 tid=0x001f0c00 nid=0xf waiting
>    on condition [0xb6a7f000]
>       java.lang.Thread.State: TIMED_WAITING (sleeping)
>            at java.lang.Thread.sleep(Native Method)
>            at
>    
> com.sun.jini.qa.harness.HeartOfTheMachine.ticktack(HeartOfTheMachine.java:63) 
>
>            at
>    
> com.sun.jini.qa.harness.HeartOfTheMachine.access$000(HeartOfTheMachine.java:29) 
>
>            at
>    
> com.sun.jini.qa.harness.HeartOfTheMachine$1.run(HeartOfTheMachine.java:45) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "Low Memory Detector" daemon prio=3 tid=0x00123c00 nid=0xd runnable
>    [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "C2 CompilerThread1" daemon prio=3 tid=0x00121800 nid=0xc waiting on
>    condition [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "C2 CompilerThread0" daemon prio=3 tid=0x0011f000 nid=0xb waiting on
>    condition [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "Signal Dispatcher" daemon prio=3 tid=0x0011d800 nid=0xa runnable
>    [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "Finalizer" daemon prio=3 tid=0x0010d400 nid=0x9 in Object.wait()
>    [0xb707f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xe6401150> (a 
> java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>            - locked <0xe6401150> (a java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>            at
>    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
>    "Reference Handler" daemon prio=3 tid=0x00108800 nid=0x8 in
>    Object.wait() [0xb717f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xe6401050> (a java.lang.ref.Reference$Lock)
>            at java.lang.Object.wait(Object.java:485)
>            at
>    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>            - locked <0xe6401050> (a java.lang.ref.Reference$Lock)
>
>    "main" prio=3 tid=0x00032000 nid=0x2 in Object.wait() [0xfe37f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xe69b0788> (a java.lang.UNIXProcess)
>            at java.lang.Object.wait(Object.java:485)
>            at java.lang.UNIXProcess.waitFor(UNIXProcess.java:115)
>            - locked <0xe69b0788> (a java.lang.UNIXProcess)
>            at
>    
> com.sun.jini.qa.harness.MasterHarness.runTestOtherVM(MasterHarness.java:880) 
>
>            at
>    
> com.sun.jini.qa.harness.MasterHarness.access$200(MasterHarness.java:124)
>            at
>    
> com.sun.jini.qa.harness.MasterHarness$TestRunner.run(MasterHarness.java:610) 
>
>            at
>    com.sun.jini.qa.harness.MasterHarness.runTests(MasterHarness.java:437)
>            at com.sun.jini.qa.harness.QARunner.main(QARunner.java:67)
>
>    "VM Thread" prio=3 tid=0x00104800 nid=0x7 runnable
>
>    "GC task thread#0 (ParallelGC)" prio=3 tid=0x00039c00 nid=0x3 runnable
>
>    "GC task thread#1 (ParallelGC)" prio=3 tid=0x0003b000 nid=0x4 runnable
>
>    "GC task thread#2 (ParallelGC)" prio=3 tid=0x0003c400 nid=0x5 runnable
>
>    "GC task thread#3 (ParallelGC)" prio=3 tid=0x0003dc00 nid=0x6 runnable
>
>    "VM Periodic Task Thread" prio=3 tid=0x0012e400 nid=0xe waiting on
>    condition
>
>    JNI global references: 1329
>
>    $ jstack 19019
>    2012-11-24 17:10:36
>    Full thread dump Java HotSpot(TM) Server VM (20.5-b03 mixed mode):
>
>    "Attach Listener" daemon prio=3 tid=0x00231000 nid=0x22 waiting on
>    condition [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "(JSK) KeepAlive" prio=3 tid=0x00523000 nid=0x17 waiting on
>    condition [0xb627f000]
>       java.lang.Thread.State: TIMED_WAITING (sleeping)
>            at java.lang.Thread.sleep(Native Method)
>            at
>    
> com.sun.jini.jeri.internal.runtime.JvmLifeSupport$2.run(JvmLifeSupport.java:130) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "(JSK) Reaper" daemon prio=3 tid=0x00758800 nid=0x16 in
>    Object.wait() [0xb637f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xe7908a18> (a 
> java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>            - locked <0xe7908a18> (a java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>            at
>    
> com.sun.jini.jeri.internal.runtime.ImplRefManager$Reaper.run(ImplRefManager.java:426) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "(JSK)
>    
> TcpServerEndpoint.LH[ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=34492]] 
>
>    accept loop" daemon prio=3 tid=0x0034ac00 nid=0x15 runnable 
> [0xb647f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.net.PlainSocketImpl.socketAccept(Native Method)
>            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>            - locked <0xe7908ad0> (a java.net.SocksSocketImpl)
>            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>            at java.net.ServerSocket.accept(ServerSocket.java:430)
>            at
>    
> net.jini.jeri.tcp.TcpServerEndpoint$LH.executeAcceptLoop(TcpServerEndpoint.java:798) 
>
>            at
>    
> net.jini.jeri.tcp.TcpServerEndpoint$LH.access$400(TcpServerEndpoint.java:736) 
>
>            at
>    
> net.jini.jeri.tcp.TcpServerEndpoint$LH$1.run(TcpServerEndpoint.java:768)
>            at 
> com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:140)
>            at
>    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:174)
>            at java.lang.Thread.run(Thread.java:662)
>
>    "RMI Scheduler(0)" daemon prio=3 tid=0x0063e400 nid=0x14 waiting on
>    condition [0xb657f000]
>       java.lang.Thread.State: TIMED_WAITING (parking)
>            at sun.misc.Unsafe.park(Native Method)
>            - parking to wait for  <0xbb886338> (a
>    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>            at
>    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>            at
>    
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) 
>
>            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) 
>
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "DestroyJavaVM" prio=3 tid=0x00031800 nid=0x2 waiting on condition
>    [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "Thread-1" prio=3 tid=0x00545800 nid=0x12 waiting on condition
>    [0xb677f000]
>       java.lang.Thread.State: TIMED_WAITING (parking)
>            at sun.misc.Unsafe.park(Native Method)
>            - parking to wait for  <0xe7913318> (a
>    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>            at
>    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>            at
>    
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) 
>
>            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) 
>
>            at
>    
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) 
>
>            at
>    
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) 
>
>            at java.lang.Thread.run(Thread.java:662)
>
>    "GC Daemon" daemon prio=3 tid=0x00569400 nid=0x11 in Object.wait()
>    [0xb687f000]
>       java.lang.Thread.State: TIMED_WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xbb803ff8> (a sun.misc.GC$LatencyLock)
>            at sun.misc.GC$Daemon.run(GC.java:100)
>            - locked <0xbb803ff8> (a sun.misc.GC$LatencyLock)
>
>    "RMI Reaper" prio=3 tid=0x002ca000 nid=0x10 in Object.wait()
>    [0xb697f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xbb800100> (a 
> java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>            - locked <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>            at
>    sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
>            at java.lang.Thread.run(Thread.java:662)
>
>    "RMI TCP Accept-0" daemon prio=3 tid=0x003f6000 nid=0xf runnable
>    [0xb6a7f000]
>       java.lang.Thread.State: RUNNABLE
>            at java.net.PlainSocketImpl.socketAccept(Native Method)
>            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>            - locked <0xbb8001a8> (a java.net.SocksSocketImpl)
>            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>            at java.net.ServerSocket.accept(ServerSocket.java:430)
>            at
>    
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) 
>
>            at
>    
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>            at java.lang.Thread.run(Thread.java:662)
>
>    "Low Memory Detector" daemon prio=3 tid=0x00123c00 nid=0xd runnable
>    [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "C2 CompilerThread1" daemon prio=3 tid=0x00121000 nid=0xc waiting on
>    condition [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "C2 CompilerThread0" daemon prio=3 tid=0x0011e800 nid=0xb waiting on
>    condition [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "Signal Dispatcher" daemon prio=3 tid=0x0011d000 nid=0xa runnable
>    [0x00000000]
>       java.lang.Thread.State: RUNNABLE
>
>    "Finalizer" daemon prio=3 tid=0x0010cc00 nid=0x9 in Object.wait()
>    [0xb707f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xbb808200> (a 
> java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>            - locked <0xbb808200> (a java.lang.ref.ReferenceQueue$Lock)
>            at 
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>            at
>    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
>    "Reference Handler" daemon prio=3 tid=0x00107c00 nid=0x8 in
>    Object.wait() [0xb717f000]
>       java.lang.Thread.State: WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            - waiting on <0xbb800358> (a java.lang.ref.Reference$Lock)
>            at java.lang.Object.wait(Object.java:485)
>            at
>    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>            - locked <0xbb800358> (a java.lang.ref.Reference$Lock)
>
>    "VM Thread" prio=3 tid=0x00104000 nid=0x7 runnable
>
>    "GC task thread#0 (ParallelGC)" prio=3 tid=0x00039400 nid=0x3 runnable
>
>    "GC task thread#1 (ParallelGC)" prio=3 tid=0x0003a800 nid=0x4 runnable
>
>    "GC task thread#2 (ParallelGC)" prio=3 tid=0x0003bc00 nid=0x5 runnable
>
>    "GC task thread#3 (ParallelGC)" prio=3 tid=0x0003d000 nid=0x6 runnable
>
>    "VM Periodic Task Thread" prio=3 tid=0x00136000 nid=0xe waiting on
>    condition
>
>    JNI global references: 1485
>
>    $
>
>
>


Re: Deadlock in MultiCastMonitorStopReplace test

Posted by Peter Firmstone <ji...@zeus.net.au>.
Ok this time I've got 3 separate jvm stacks and a stack trace, and the 
test is deadlocked (cpu idle):

     [java] Running 
com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td
     [java] Time is Sat Nov 24 16:47:24 EST 2012
     [java] Starting test in separate process with command:
     [java] /usr/jdk/jdk1.6.0_30/jre/bin/java 
-Djava.security.manager=org.apache.river.api.security.CombinerSecurityManager 
-Djava.security.policy=file:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/harness/policy/defaulttest.policy 
-cp 
/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib/jiniharness.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib/jinitests.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib/jsk-platform.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib/jsk-lib.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib/high-scale-lib.jar:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib/reference-collections-1.0.1.jar 
-client 
-Djava.ext.dirs=/usr/jdk/jdk1.6.0_30/jre/lib/ext:/usr/jdk/packages/lib/ext:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib-ext:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib-ext 
-Dcom.sun.jini.jsk.port=9080 -Dcom.sun.jini.qa.port=9081 
-Dcom.sun.jini.jsk.home=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy 
-Dcom.sun.jini.qa.home=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa 
-Dcom.sun.jini.qa.harness.harnessJar=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib/jiniharness.jar 
-Dcom.sun.jini.qa.harness.testJar=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib/jinitests.jar 
-Dcom.sun.jini.qa.harness.runjiniserver=true 
-Dcom.sun.jini.qa.harness.runkitserver=true 
-Djava.security.properties=file:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/harness/trust/dynamic-policy.properties 
-Dcom.sun.jini.qa.harness.testhosts= 
-Djava.util.logging.config.file=/home/peter/logging.properties 
-Dcom.sun.jini.test.home=/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa 
-Dcom.sun.jini.test.port=9082 
-Dcom.sun.jini.qa.harness.policies=file:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/src/com/sun/jini/test/resources/jinitest.policy 
-Djava.ext.dirs=/usr/jdk/jdk1.6.0_30/jre/lib/ext:/usr/jdk/packages/lib/ext:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa/lib-ext:/opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/lib-ext 
com.sun.jini.qa.harness.MasterTest 
com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td

     [java] java.rmi.UnmarshalException: exception unmarshalling 
response; nested exception is:
     [java]     java.io.IOException: I/O error reading from mux 
connection: java.io.EOFException
     [java]     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:847)
     [java]     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:659)
     [java]     at 
net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:528)
     [java]     at 
com.sun.jini.test.services.lookupsimulator.$Proxy0.getLocator(Unknown 
Source)
     [java]     at 
com.sun.jini.test.services.lookupsimulator.LookupSimulatorProxy.getLocator(LookupSimulatorProxy.java:134)
     [java]     at 
com.sun.jini.test.share.BaseQATest.startLookup(BaseQATest.java:1722)
     [java]     at 
com.sun.jini.test.share.BaseQATest.startInitLookups(BaseQATest.java:1538)
     [java]     at 
com.sun.jini.test.share.BaseQATest.setup(BaseQATest.java:1063)
     [java]     at 
com.sun.jini.test.spec.lookupdiscovery.AbstractBaseTest.setup(AbstractBaseTest.java:76)
     [java]     at 
com.sun.jini.test.spec.lookupdiscovery.Discovered.setup(Discovered.java:69)
     [java]     at 
com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:219)
     [java]     at 
com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
     [java] Caused by: java.io.IOException: I/O error reading from mux 
connection: java.io.EOFException
     [java]     at 
com.sun.jini.jeri.internal.mux.Session$MuxInputStream.read(Session.java:874)
     [java]     at 
net.jini.jeri.connection.ConnectionManager$Outbound$Input.read(ConnectionManager.java:550)
     [java]     at 
net.jini.jeri.BasicObjectEndpoint.executeCall(BasicObjectEndpoint.java:410)
     [java]     at 
net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:806)
     [java]     ... 11 more
     [java] Caused by: java.io.EOFException
     [java]     at 
com.sun.jini.jeri.internal.mux.StreamConnectionIO$Reader.run(StreamConnectionIO.java:264)
     [java]     at 
com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:140)
     [java]     at 
com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:174)
     [java]     at java.lang.Thread.run(Thread.java:662)

    $ jstack 19013
    2012-11-24 17:09:27
    Full thread dump Java HotSpot(TM) Client VM (20.5-b03 mixed mode,
    sharing):

    "Attach Listener" daemon prio=3 tid=0x001cf400 nid=0x23 waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "unicast request" daemon prio=3 tid=0x0020d400 nid=0x1f runnable
    [0xfa97f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xc44f0090> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at
    com.sun.jini.test.share.DiscoveryProtocolSimulator$UnicastThread.run(DiscoveryProtocolSimulator.java:616)

    "RMI Scheduler(0)" daemon prio=3 tid=0x001e6800 nid=0x14 waiting on
    condition [0xfb07f000]
       java.lang.Thread.State: WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0xc44f0220> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
            at
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
            at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
            at
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:662)

    "GC Daemon" daemon prio=3 tid=0x001e1c00 nid=0x13 in Object.wait()
    [0xfb17f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xc95ef040> (a sun.misc.GC$LatencyLock)
            at sun.misc.GC$Daemon.run(GC.java:100)
            - locked <0xc95ef040> (a sun.misc.GC$LatencyLock)

    "RMI RenewClean-[10.1.1.2:34487]" daemon prio=3 tid=0x00196800
    nid=0x12 in Object.wait() [0xfb27f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xc95eec50> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xc95eec50> (a java.lang.ref.ReferenceQueue$Lock)
            at
    sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
            at java.lang.Thread.run(Thread.java:662)

    "NonActivatableGroup_system-out" daemon prio=3 tid=0x0023c400
    nid=0x10 runnable [0xfb37f000]
       java.lang.Thread.State: RUNNABLE
            at java.io.FileInputStream.readBytes(Native Method)
            at java.io.FileInputStream.read(FileInputStream.java:220)
            at
    java.lang.UNIXProcess$DeferredCloseInputStream.read(UNIXProcess.java:227)
            at
    java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
            at
    java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
            at
    java.io.BufferedInputStream.read(BufferedInputStream.java:317)
            - locked <0xc95f1440> (a java.io.BufferedInputStream)
            at java.io.FilterInputStream.read(FilterInputStream.java:90)
            at com.sun.jini.qa.harness.Pipe.run(Pipe.java:119)
            at java.lang.Thread.run(Thread.java:662)

    "process reaper" daemon prio=3 tid=0x00239800 nid=0xf runnable
    [0xfb47f000]
       java.lang.Thread.State: RUNNABLE
            at java.lang.UNIXProcess.waitForProcessExit(Native Method)
            at java.lang.UNIXProcess.access$900(UNIXProcess.java:17)
            at java.lang.UNIXProcess$2$1.run(UNIXProcess.java:86)

    "Thread-5" prio=3 tid=0x00232400 nid=0xe runnable [0xfb57f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xc95f16d0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at com.sun.jini.tool.ClassServer.run(ClassServer.java:397)

    "Thread-4" prio=3 tid=0x00230800 nid=0xd runnable [0xfb67f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xc95f1900> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at com.sun.jini.tool.ClassServer.run(ClassServer.java:397)

    "Thread-3" prio=3 tid=0x00226800 nid=0xc runnable [0xfb77f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xc95f1b30> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at com.sun.jini.tool.ClassServer.run(ClassServer.java:397)

    "no heart without soul" daemon prio=3 tid=0x00221800 nid=0xb waiting
    on condition [0xfb87f000]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
            at java.lang.Thread.sleep(Native Method)
            at
    com.sun.jini.qa.harness.HeartOfTheMachine.ticktack(HeartOfTheMachine.java:63)
            at
    com.sun.jini.qa.harness.HeartOfTheMachine.access$000(HeartOfTheMachine.java:29)
            at
    com.sun.jini.qa.harness.HeartOfTheMachine$1.run(HeartOfTheMachine.java:45)
            at java.lang.Thread.run(Thread.java:662)

    "Low Memory Detector" daemon prio=3 tid=0x00169000 nid=0x9 runnable
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "C1 CompilerThread0" daemon prio=3 tid=0x00166800 nid=0x8 waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Signal Dispatcher" daemon prio=3 tid=0x0015c000 nid=0x7 runnable
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Thread-1" prio=3 tid=0x00150000 nid=0x6 waiting on condition
    [0xfbd7f000]
       java.lang.Thread.State: TIMED_WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0xc9565ed0> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
            at
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
            at
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:662)

    "Finalizer" daemon prio=3 tid=0x00095000 nid=0x5 in Object.wait()
    [0xfbe7f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xc9566028> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xc9566028> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
            at
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

    "Reference Handler" daemon prio=3 tid=0x00093c00 nid=0x4 in
    Object.wait() [0xfbf7f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xc95660b8> (a java.lang.ref.Reference$Lock)
            at java.lang.Object.wait(Object.java:485)
            at
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
            - locked <0xc95660b8> (a java.lang.ref.Reference$Lock)

    "main" prio=3 tid=0x00031c00 nid=0x2 in Object.wait() [0xfe6ff000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xc44f0010> (a
    com.sun.jini.test.share.DiscoveryProtocolSimulator$UnicastThread)
            at java.lang.Thread.join(Thread.java:1186)
            - locked <0xc44f0010> (a
    com.sun.jini.test.share.DiscoveryProtocolSimulator$UnicastThread)
            at java.lang.Thread.join(Thread.java:1239)
            at
    com.sun.jini.test.share.DiscoveryProtocolSimulator.stopAnnouncements(DiscoveryProtocolSimulator.java:311)
            at
    com.sun.jini.test.share.DiscoveryProtocolSimulator.stopAnnouncements(DiscoveryProtocolSimulator.java:298)
            at
    com.sun.jini.test.share.BaseQATest.tearDown(BaseQATest.java:1090)
            - locked <0xc95f6b88> (a java.util.Collections$SynchronizedMap)
            at
    com.sun.jini.test.spec.lookupdiscovery.AbstractBaseTest.tearDown(AbstractBaseTest.java:111)
            at
    com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:280)
            at com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)

    "VM Thread" prio=3 tid=0x0004dc00 nid=0x3 runnable

    "VM Periodic Task Thread" prio=3 tid=0x0016cc00 nid=0xa waiting on
    condition

    JNI global references: 929

    $ jstack 19008
    2012-11-24 17:09:58
    Full thread dump Java HotSpot(TM) Server VM (20.5-b03 mixed mode):

    "Attach Listener" daemon prio=3 tid=0x0049a400 nid=0x15 waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "test-err" daemon prio=3 tid=0x00315000 nid=0x14 runnable [0xb64ff000]
       java.lang.Thread.State: RUNNABLE
            at java.io.FileInputStream.readBytes(Native Method)
            at java.io.FileInputStream.read(FileInputStream.java:198)
            at
    java.lang.UNIXProcess$DeferredCloseInputStream.read(UNIXProcess.java:218)
            at com.sun.jini.qa.harness.Pipe.run(Pipe.java:119)
            at java.lang.Thread.run(Thread.java:662)

    "test-out" daemon prio=3 tid=0x001f6800 nid=0x13 runnable [0xb65ff000]
       java.lang.Thread.State: RUNNABLE
            at java.io.FileInputStream.readBytes(Native Method)
            at java.io.FileInputStream.read(FileInputStream.java:220)
            at
    java.lang.UNIXProcess$DeferredCloseInputStream.read(UNIXProcess.java:227)
            at
    java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
            at
    java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
            at
    java.io.BufferedInputStream.read(BufferedInputStream.java:317)
            - locked <0xe69b29a8> (a java.io.BufferedInputStream)
            at java.io.FilterInputStream.read(FilterInputStream.java:90)
            at com.sun.jini.qa.harness.Pipe.run(Pipe.java:119)
            at java.lang.Thread.run(Thread.java:662)

    "process reaper" daemon prio=3 tid=0x001f6400 nid=0x12 runnable
    [0xb66ff000]
       java.lang.Thread.State: RUNNABLE
            at java.lang.UNIXProcess.waitForProcessExit(Native Method)
            at java.lang.UNIXProcess.access$900(UNIXProcess.java:17)
            at java.lang.UNIXProcess$2$1.run(UNIXProcess.java:86)

    "Thread-1" prio=3 tid=0x0025e400 nid=0x11 waiting on condition
    [0xb67ff000]
       java.lang.Thread.State: TIMED_WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0xe68ff988> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
            at
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
            at
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:662)

    "Keepalive" daemon prio=3 tid=0x00260000 nid=0x10 runnable [0xb697f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xe660c318> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at
    com.sun.jini.qa.harness.MasterHarness$KeepAlivePort.run(MasterHarness.java:261)
            at java.lang.Thread.run(Thread.java:662)

    "no heart without soul" daemon prio=3 tid=0x001f0c00 nid=0xf waiting
    on condition [0xb6a7f000]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
            at java.lang.Thread.sleep(Native Method)
            at
    com.sun.jini.qa.harness.HeartOfTheMachine.ticktack(HeartOfTheMachine.java:63)
            at
    com.sun.jini.qa.harness.HeartOfTheMachine.access$000(HeartOfTheMachine.java:29)
            at
    com.sun.jini.qa.harness.HeartOfTheMachine$1.run(HeartOfTheMachine.java:45)
            at java.lang.Thread.run(Thread.java:662)

    "Low Memory Detector" daemon prio=3 tid=0x00123c00 nid=0xd runnable
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread1" daemon prio=3 tid=0x00121800 nid=0xc waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread0" daemon prio=3 tid=0x0011f000 nid=0xb waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Signal Dispatcher" daemon prio=3 tid=0x0011d800 nid=0xa runnable
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Finalizer" daemon prio=3 tid=0x0010d400 nid=0x9 in Object.wait()
    [0xb707f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xe6401150> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xe6401150> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
            at
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

    "Reference Handler" daemon prio=3 tid=0x00108800 nid=0x8 in
    Object.wait() [0xb717f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xe6401050> (a java.lang.ref.Reference$Lock)
            at java.lang.Object.wait(Object.java:485)
            at
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
            - locked <0xe6401050> (a java.lang.ref.Reference$Lock)

    "main" prio=3 tid=0x00032000 nid=0x2 in Object.wait() [0xfe37f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xe69b0788> (a java.lang.UNIXProcess)
            at java.lang.Object.wait(Object.java:485)
            at java.lang.UNIXProcess.waitFor(UNIXProcess.java:115)
            - locked <0xe69b0788> (a java.lang.UNIXProcess)
            at
    com.sun.jini.qa.harness.MasterHarness.runTestOtherVM(MasterHarness.java:880)
            at
    com.sun.jini.qa.harness.MasterHarness.access$200(MasterHarness.java:124)
            at
    com.sun.jini.qa.harness.MasterHarness$TestRunner.run(MasterHarness.java:610)
            at
    com.sun.jini.qa.harness.MasterHarness.runTests(MasterHarness.java:437)
            at com.sun.jini.qa.harness.QARunner.main(QARunner.java:67)

    "VM Thread" prio=3 tid=0x00104800 nid=0x7 runnable

    "GC task thread#0 (ParallelGC)" prio=3 tid=0x00039c00 nid=0x3 runnable

    "GC task thread#1 (ParallelGC)" prio=3 tid=0x0003b000 nid=0x4 runnable

    "GC task thread#2 (ParallelGC)" prio=3 tid=0x0003c400 nid=0x5 runnable

    "GC task thread#3 (ParallelGC)" prio=3 tid=0x0003dc00 nid=0x6 runnable

    "VM Periodic Task Thread" prio=3 tid=0x0012e400 nid=0xe waiting on
    condition

    JNI global references: 1329

    $ jstack 19019
    2012-11-24 17:10:36
    Full thread dump Java HotSpot(TM) Server VM (20.5-b03 mixed mode):

    "Attach Listener" daemon prio=3 tid=0x00231000 nid=0x22 waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "(JSK) KeepAlive" prio=3 tid=0x00523000 nid=0x17 waiting on
    condition [0xb627f000]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
            at java.lang.Thread.sleep(Native Method)
            at
    com.sun.jini.jeri.internal.runtime.JvmLifeSupport$2.run(JvmLifeSupport.java:130)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK) Reaper" daemon prio=3 tid=0x00758800 nid=0x16 in
    Object.wait() [0xb637f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xe7908a18> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xe7908a18> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
            at
    com.sun.jini.jeri.internal.runtime.ImplRefManager$Reaper.run(ImplRefManager.java:426)
            at java.lang.Thread.run(Thread.java:662)

    "(JSK)
    TcpServerEndpoint.LH[ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=34492]]
    accept loop" daemon prio=3 tid=0x0034ac00 nid=0x15 runnable [0xb647f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xe7908ad0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at
    net.jini.jeri.tcp.TcpServerEndpoint$LH.executeAcceptLoop(TcpServerEndpoint.java:798)
            at
    net.jini.jeri.tcp.TcpServerEndpoint$LH.access$400(TcpServerEndpoint.java:736)
            at
    net.jini.jeri.tcp.TcpServerEndpoint$LH$1.run(TcpServerEndpoint.java:768)
            at com.sun.jini.thread.ThreadPool$Task.run(ThreadPool.java:140)
            at
    com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:174)
            at java.lang.Thread.run(Thread.java:662)

    "RMI Scheduler(0)" daemon prio=3 tid=0x0063e400 nid=0x14 waiting on
    condition [0xb657f000]
       java.lang.Thread.State: TIMED_WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0xbb886338> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
            at
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
            at
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:662)

    "DestroyJavaVM" prio=3 tid=0x00031800 nid=0x2 waiting on condition
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Thread-1" prio=3 tid=0x00545800 nid=0x12 waiting on condition
    [0xb677f000]
       java.lang.Thread.State: TIMED_WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0xe7913318> (a
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
            at
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
            at
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:662)

    "GC Daemon" daemon prio=3 tid=0x00569400 nid=0x11 in Object.wait()
    [0xb687f000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb803ff8> (a sun.misc.GC$LatencyLock)
            at sun.misc.GC$Daemon.run(GC.java:100)
            - locked <0xbb803ff8> (a sun.misc.GC$LatencyLock)

    "RMI Reaper" prio=3 tid=0x002ca000 nid=0x10 in Object.wait()
    [0xb697f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
            at
    sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
            at java.lang.Thread.run(Thread.java:662)

    "RMI TCP Accept-0" daemon prio=3 tid=0x003f6000 nid=0xf runnable
    [0xb6a7f000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
            - locked <0xbb8001a8> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:462)
            at java.net.ServerSocket.accept(ServerSocket.java:430)
            at
    sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
            at
    sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
            at java.lang.Thread.run(Thread.java:662)

    "Low Memory Detector" daemon prio=3 tid=0x00123c00 nid=0xd runnable
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread1" daemon prio=3 tid=0x00121000 nid=0xc waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread0" daemon prio=3 tid=0x0011e800 nid=0xb waiting on
    condition [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Signal Dispatcher" daemon prio=3 tid=0x0011d000 nid=0xa runnable
    [0x00000000]
       java.lang.Thread.State: RUNNABLE

    "Finalizer" daemon prio=3 tid=0x0010cc00 nid=0x9 in Object.wait()
    [0xb707f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb808200> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
            - locked <0xbb808200> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
            at
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

    "Reference Handler" daemon prio=3 tid=0x00107c00 nid=0x8 in
    Object.wait() [0xb717f000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            - waiting on <0xbb800358> (a java.lang.ref.Reference$Lock)
            at java.lang.Object.wait(Object.java:485)
            at
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
            - locked <0xbb800358> (a java.lang.ref.Reference$Lock)

    "VM Thread" prio=3 tid=0x00104000 nid=0x7 runnable

    "GC task thread#0 (ParallelGC)" prio=3 tid=0x00039400 nid=0x3 runnable

    "GC task thread#1 (ParallelGC)" prio=3 tid=0x0003a800 nid=0x4 runnable

    "GC task thread#2 (ParallelGC)" prio=3 tid=0x0003bc00 nid=0x5 runnable

    "GC task thread#3 (ParallelGC)" prio=3 tid=0x0003d000 nid=0x6 runnable

    "VM Periodic Task Thread" prio=3 tid=0x00136000 nid=0xe waiting on
    condition

    JNI global references: 1485

    $



Re: Deadlock in MultiCastMonitorStopReplace test

Posted by Peter Firmstone <ji...@zeus.net.au>.
Sorry, there's so much detail it's difficult to see what's going on, and 
it looks like I only pasted the Server stack, oops.  The test isn't 
timing out as it should, I'll run it again and grab the client stack, I 
killed the test after 2 hours, so the test was definitely deadlocked.  
There are usually 3 or more jvm's running.

The BindException is ok in this case, the default unicast discover port, 
4160 is tried first (when no port is specified), then an ephemeral port 
is grabbed if 4160 is in use- it's ok in other words, I'll have to 
change the output to something more meaningful because BindException is 
usually bad.

Like: "port 4160 in use, choosing ephemeral."

Will run again and grab state of other jvm's.

Peter.



Re: Deadlock in MultiCastMonitorStopReplace test

Posted by Gregg Wonderly <ge...@cox.net>.
On Nov 23, 2012, at 9:02 PM, Peter Firmstone <ji...@zeus.net.au> wrote:

> Anyone have any thoughts on the following?
> 
> The attached patch fixes some visibility issues, I'm now seeing a deadlock in MulticastMonitorStopReplace.

Since I don't see a "waiting to lock" in these traces, I would classify this as a lack of progress on some "event".  Is there something I missed in looking over the traces?

I see an "address in use" bind exception below.  Is that the problem?

There is another comment in line below.

Gregg


> 
> I've also attached the output from the jenkins arm build, without the patch applied, where only 2 out of 3 events are recieved, the missing one was interrupted before its member group could be changed, other failing tests have the same issue, execept all are interrupted.
> 
> BaseQATest.replaceMemberGroups FINE:  lookup service 0 - replacing member groups with --    BaseQATest.replaceMemberGroups FINE:  failed to change member groups for lookup service 0
> java.rmi.ConnectIOException: I/O exception connecting to BasicObjectEndpoint[37abfa80-a6e1-4340-9cdb-37caccf52d5a,TcpEndpoint[64.20.246.146:46371]]; nested exception is:    	java.io.IOException: interrupt waiting for connection header
> 	at net.jini.jeri.BasicInvocationHandler.wrapSafeIOException(BasicInvocationHandler.java:893)
> 	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:711)
> 	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:659)
> 	at net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:528)
> 	at com.sun.jini.test.services.lookupsimulator.$Proxy0.getMemberGroups(Unknown Source)
> 	at com.sun.jini.test.services.lookupsimulator.LookupSimulatorProxy.getMemberGroups(LookupSimulatorProxy.java:144)
> 	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2578)
> 	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2628)
> 	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2660)
> 	at com.sun.jini.test.spec.lookupdiscovery.MulticastMonitorStopReplace.run(MulticastMonitorStopReplace.java:103)
> 	at com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:254)
> 	at com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
> Caused by: java.io.IOException: interrupt waiting for connection header
> 	at com.sun.jini.jeri.internal.mux.Mux.start(Mux.java:238)
> 	at com.sun.jini.jeri.internal.mux.MuxClient.start(MuxClient.java:35)
> 	at net.jini.jeri.connection.ConnectionManager$OutboundMux.newRequest(ConnectionManager.java:375)
> 	at net.jini.jeri.connection.ConnectionManager$ReqIterator.next(ConnectionManager.java:665)
> 	at net.jini.jeri.BasicObjectEndpoint$1.next(BasicObjectEndpoint.java:370)
> 	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:708)
> 	... 10 more
> Caused by: java.lang.InterruptedException
> 	at java.lang.Object.wait(Native Method)
> 	at com.sun.jini.jeri.internal.mux.Mux.start(Mux.java:230)
> 	... 15 more
> BaseQATest.replaceMemberGroups FINE:  lookup service 1 - replacing member groups with --    BaseQATest.replaceMemberGroups FINE:    newGroups[0] = LDGroup2_A_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[1] = LDGroup2_B_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[2] = LDGroup2_C_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:  lookup service 2 - replacing member groups with --    BaseQATest.replaceMemberGroups FINE:    newGroups[0] = LDGroup1_A_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[1] = LDGroup1_B_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[2] = LDGroup1_C_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[3] = LDGroup1_D_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:  for DISCARD events -- waiting at least 30 seconds, but no more than 180 seconds ...
> 
> The following deadlock is experienced on a 4 way sparc:
> 
> Running
> com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td
> 
> $ jstack 4568
> 2012-11-24 12:24:23
> Full thread dump Java HotSpot(TM) Server VM (20.5-b03 mixed mode):
> 
> "Attach Listener" daemon prio=3 tid=0x00889800 nid=0x35 waiting on
> condition [0x00000000]
>   java.lang.Thread.State: RUNNABLE
> 
> "(JSK) Idle" daemon prio=3 tid=0x008b8c00 nid=0x1e in Object.wait()
> [0xb5c7f000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb887910> (a java.lang.Object)
>        at
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
>        - locked <0xbb887910> (a java.lang.Object)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK) Idle" daemon prio=3 tid=0x00289400 nid=0x1d in Object.wait()
> [0xb5d7f000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb88b9d8> (a java.lang.Object)
>        at
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
>        - locked <0xbb88b9d8> (a java.lang.Object)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK) Idle" daemon prio=3 tid=0x00381800 nid=0x1c in Object.wait()
> [0xb5e7f000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb887910> (a java.lang.Object)
>        at
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
>        - locked <0xbb887910> (a java.lang.Object)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK) mux reader" daemon prio=3 tid=0x00517c00 nid=0x1b runnable
> [0xb667f000]
>   java.lang.Thread.State: RUNNABLE
>        at java.net.SocketInputStream.socketRead0(Native Method)
>        at java.net.SocketInputStream.read(SocketInputStream.java:129)
>        at
> com.sun.jini.jeri.internal.mux.StreamConnectionIO$1.read(StreamConnectionIO.java:355)
>        at
> com.sun.jini.jeri.internal.mux.StreamConnectionIO$Reader.run(StreamConnectionIO.java:262)
>        at
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:140)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK) Idle" daemon prio=3 tid=0x006a1400 nid=0x1a in Object.wait()
> [0xb5f7f000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb88b9d8> (a java.lang.Object)
>        at
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
>        - locked <0xbb88b9d8> (a java.lang.Object)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK) mux writer" daemon prio=3 tid=0x0085c800 nid=0x19 in
> Object.wait() [0xb607f000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb9b5aa0> (a java.lang.Object)
>        at java.lang.Object.wait(Object.java:485)
>        at
> com.sun.jini.jeri.internal.mux.StreamConnectionIO$Writer.run(StreamConnectionIO.java:168)
>        - locked <0xbb9b5aa0> (a java.lang.Object)
>        at
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:140)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK) Idle" daemon prio=3 tid=0x006cb000 nid=0x18 in Object.wait()
> [0xb617f000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb887910> (a java.lang.Object)
>        at
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:155)
>        - locked <0xbb887910> (a java.lang.Object)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK) KeepAlive" prio=3 tid=0x003c2800 nid=0x17 waiting on
> condition [0xb627f000]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>        at java.lang.Thread.sleep(Native Method)
>        at
> com.sun.jini.jeri.internal.runtime.JvmLifeSupport$2.run(JvmLifeSupport.java:130)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK) Reaper" daemon prio=3 tid=0x003c1400 nid=0x16 in
> Object.wait() [0xb637f000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb887a48> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>        - locked <0xbb887a48> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>        at
> com.sun.jini.jeri.internal.runtime.ImplRefManager$Reaper.run(ImplRefManager.java:426)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "(JSK)
> TcpServerEndpoint.LH[ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=33781]]
> accept loop" daemon prio=3 tid=0x003be800 nid=0x15 runnable [0xb647f000]
>   java.lang.Thread.State: RUNNABLE
>        at java.net.PlainSocketImpl.socketAccept(Native Method)
>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>        - locked <0xbb893860> (a java.net.SocksSocketImpl)
>        at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>        at java.net.ServerSocket.accept(ServerSocket.java:430)
>        at
> net.jini.jeri.tcp.TcpServerEndpoint$LH.executeAcceptLoop(TcpServerEndpoint.java:798)
>        at
> net.jini.jeri.tcp.TcpServerEndpoint$LH.access$400(TcpServerEndpoint.java:736)
>        at
> net.jini.jeri.tcp.TcpServerEndpoint$LH$1.run(TcpServerEndpoint.java:768)
>        at
> com.sun.jini.thread.ThreadPool$Worker.run(ThreadPool.java:140)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "RMI Scheduler(0)" daemon prio=3 tid=0x0043b000 nid=0x14 waiting on
> condition [0xb657f000]
>   java.lang.Thread.State: TIMED_WAITING (parking)
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for  <0xbb886590> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>        at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>        at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "DestroyJavaVM" prio=3 tid=0x00031800 nid=0x2 waiting on condition
> [0x00000000]
>   java.lang.Thread.State: RUNNABLE
> 
> "Thread-1" prio=3 tid=0x0043fc00 nid=0x12 waiting on condition
> [0xb677f000]
>   java.lang.Thread.State: TIMED_WAITING (parking)
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for  <0xbb8954b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>        at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>        at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "GC Daemon" daemon prio=3 tid=0x00512800 nid=0x11 in Object.wait()
> [0xb687f000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb804060> (a sun.misc.GC$LatencyLock)
>        at sun.misc.GC$Daemon.run(GC.java:100)
>        - locked <0xbb804060> (a sun.misc.GC$LatencyLock)
> 
> "RMI Reaper" prio=3 tid=0x00555000 nid=0x10 in Object.wait()
> [0xb697f000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>        - locked <0xbb800100> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>        at
> sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "RMI TCP Accept-0" daemon prio=3 tid=0x00554000 nid=0xf runnable
> [0xb6a7f000]
>   java.lang.Thread.State: RUNNABLE
>        at java.net.PlainSocketImpl.socketAccept(Native Method)
>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>        - locked <0xbb8001a8> (a java.net.SocksSocketImpl)
>        at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>        at java.net.ServerSocket.accept(ServerSocket.java:430)
>        at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>        at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>        at java.lang.Thread.run(Thread.java:662)
> 
> "Low Memory Detector" daemon prio=3 tid=0x00123c00 nid=0xd runnable
> [0x00000000]
>   java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread1" daemon prio=3 tid=0x00121000 nid=0xc waiting on
> condition [0x00000000]
>   java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread0" daemon prio=3 tid=0x0011e800 nid=0xb waiting on
> condition [0x00000000]
>   java.lang.Thread.State: RUNNABLE
> 
> "Signal Dispatcher" daemon prio=3 tid=0x0011d000 nid=0xa runnable
> [0x00000000]
>   java.lang.Thread.State: RUNNABLE
> 
> "Finalizer" daemon prio=3 tid=0x00109800 nid=0x9 in Object.wait()
> [0xb707f000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb800518> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>        - locked <0xbb800518> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>        at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 
> "Reference Handler" daemon prio=3 tid=0x00107c00 nid=0x8 in
> Object.wait() [0xb717f000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0xbb804050> (a java.lang.ref.Reference$Lock)
>        at java.lang.Object.wait(Object.java:485)
>        at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>        - locked <0xbb804050> (a java.lang.ref.Reference$Lock)
> 
> "VM Thread" prio=3 tid=0x00104000 nid=0x7 runnable
> 
> "GC task thread#0 (ParallelGC)" prio=3 tid=0x00039400 nid=0x3 runnable
> 
> "GC task thread#1 (ParallelGC)" prio=3 tid=0x0003a800 nid=0x4 runnable
> 
> "GC task thread#2 (ParallelGC)" prio=3 tid=0x0003bc00 nid=0x5 runnable
> 
> "GC task thread#3 (ParallelGC)" prio=3 tid=0x0003d000 nid=0x6 runnable
> 
> "VM Periodic Task Thread" prio=3 tid=0x00136000 nid=0xe waiting on
> condition
> 
> JNI global references: 1576
> 
> 
> # This patch file was generated by NetBeans IDE
> # Following Index: paths are relative to: /opt/src/River_Fixed_2nd_Try/peterConcurrentPolicy/qa
> # This patch can be applied using context Tools: Patch action on respective folder.
> # It uses platform neutral UTF-8 encoding and \n newlines.
> # Above lines and this line are ignored by the patching process.
> Index: src/com/sun/jini/qa/harness/QAConfig.java
> --- src/com/sun/jini/qa/harness/QAConfig.java Base (BASE)
> +++ src/com/sun/jini/qa/harness/QAConfig.java Locally Modified (Based On LOCAL)
> @@ -48,9 +48,11 @@
> import java.security.AccessController;
> import java.security.PrivilegedAction;
> import java.util.ArrayList;
> +import java.util.Collections;
> import java.util.HashMap;
> import java.util.HashSet;
> import java.util.Iterator;
> +import java.util.List;
> import java.util.logging.Logger;
> import java.util.logging.Level;
> import java.util.MissingResourceException;
> @@ -198,102 +200,94 @@
>   * defined which are referenced by the configuration.
>   */
>  private static MethodConstraints testLocatorConstraints;
> +    private static final long serialVersionUID = 1L;
> 
>  /** A unique string which is used to generate unique group names. */
> -    private String uniqueString;
> +    private volatile String uniqueString;
> 
> -    /** the name of the configurator class */
> -    private String configuratorClassName = null;
> -
>  /** the configuration override providers */
> -    private ArrayList overrideProviders = new ArrayList();
> +    private volatile List<OverrideProvider> overrideProviders = Collections.synchronizedList(new ArrayList<OverrideProvider>());
> 
>  /** the failure analyzers */
> -    private ArrayList failureAnalyzers = new ArrayList();
> +    private final List<FailureAnalyzer> failureAnalyzers = Collections.synchronizedList(new ArrayList<FailureAnalyzer>());
> 
> -    /** The number of configurations to cycle through */
> -    private int configurationCount = -1;
> -
>  /** Properties obtained from the user configuration file. */
> -    private Properties configProps;
> +    private volatile Properties configProps;
> 
>  /** properties obtained from the configuration group file */
> -    private Properties configSetProps = new Properties();
> +    private volatile Properties configSetProps = new Properties();
> 
>  /** Properties obtained from qaDefaults.properties. */
> -    private Properties defaultProps = new Properties();
> +    private volatile Properties defaultProps = new Properties();
> 
>  /** The test <code>Configuration</code> object, not serializable*/
> -    private transient Configuration configuration;
> +    private volatile transient Configuration configuration;
> 
>  /** The TestDescription */
> -    private TestDescription td;
> +    private volatile TestDescription td;
> 
>  /** Dynamic properties set on-the-fly by tests. */
> -    private Properties dynamicProps = new Properties();
> +    private volatile Properties dynamicProps = new Properties();
> 
>  /** Override properties, for cmd lines with non-default install props */
> -    private Properties propertyOverrides = null;
> +    private volatile Properties propertyOverrides = null;
> 
>  /** The original command line arguments. */
> -    private String[] args;
> +    private final String[] args;
> 
>  /** The configuration group tags for the test run */
> -    private String[] configTags;
> +    private volatile String[] configTags;
> 
>  /** The currently active configuration group tag */
> -    private String currentTag;
> +    private volatile String currentTag;
> 
>  /** The key name to track */
> -    private String trackKey;
> +    private volatile String trackKey;
> 
>  /** The set of hosts participating in this test run */
> -    private ArrayList hostList = new ArrayList();
> +    private final List<String> hostList = Collections.synchronizedList(new ArrayList<String>());
> 
>  /** The host index memory for the uniformrandom selection policy */
> -    private ArrayList selectedIndexes;
> +    private volatile List<Integer> selectedIndexes ;
> 
> -    /** Flag controlling use of IP address or host names during resolution */
> -    private boolean useAddress = false;
> -
>  /** The resolver */
> -    private Resolver resolver;
> +    private volatile Resolver resolver;
> 
>  /** the jar file containing the tests */
> -    private String testJar = null;
> +    private volatile String testJar = null;
> 
>  /** the harness jar file name */
> -    private String harnessJar = null;
> +    private volatile String harnessJar = null;
> 
>  /** the test class loader */
> -    private transient ClassLoader testLoader;
> +    private volatile transient ClassLoader testLoader;
> 
>  /** the search list, usually empty */
> -    private String[] searchList = new String[0];
> +    private volatile String[] searchList = new String[0];
> 
>  /** lock object for suspend/resume run */
>  private transient Object runLock = new Object();
> 
>  /** suspended state (for UI) */
> -    private boolean testSuspended = false;
> +    private volatile boolean testSuspended = false;
> 
>  /** 
>   * The host selection index for the selection policy.
>   * It's intiialized to 1 in case the roundrobin policy
>   * is being used, which begins with the first slave.
>   */
> -    private int hostIndex = 1;
> +    private volatile int hostIndex = 1;
> 
>  /** The rerun pass counter */
> -    private int passCount;
> +    private volatile int passCount;
> 
> -    private boolean callAutot = false;
> +    private volatile boolean callAutot = false;
> 
> -    private int testTotal = 0;
> +    private volatile int testTotal = 0;
> 
> -    private int testIndex = 0;
> +    private volatile int testIndex = 0;
> 
> -    private String resumeMessage;
> +    private volatile String resumeMessage;
> 
>  /**
>   * Static accessor for the config instance.
> @@ -386,7 +380,7 @@
>   */
>  public QAConfig(String[] args) throws TestException {
> 	this.args = args;
> -	resolver = new Resolver(this);
> +	resolver = new Resolver(this); // Unsafe publication
> 	configProps = loadProperties(args[0]);
> 	if (configProps.size() == 0) {
> 	    throw new TestException("Config file" 
> @@ -456,12 +450,14 @@
> 	if (hostNames != null) {
> 	    StringTokenizer tok = new StringTokenizer(hostNames, "|");
> 	    if (tok.countTokens() > 1) {
> +                synchronized (hostList){
> 		while (tok.hasMoreTokens()) {
> 		    hostList.add(tok.nextToken());
> 		}
> 	    }
> 	}
>  }
> +    }
> 
>  /**
>   * Set the token &lt;gethost&gt; to have the value of the local host
> @@ -521,7 +517,6 @@
>  {
> 	stream.defaultReadObject();
> 	harnessConfig = this;
> -	runLock = new Object();
>  }
> 
>  /**
> @@ -2113,6 +2108,7 @@
>   */
>  boolean isMaster() {
> 	boolean isMaster = true;
> +        synchronized (hostList){
> 	if (hostList.size() > 0) {
> 	    try {
> 		InetAddress thisAddr = InetAddress.getLocalHost(); //XXX multinic??
> @@ -2123,6 +2119,7 @@
> 		logger.log(Level.SEVERE, "Unexpected exception", e);
> 	    }
> 	}
> +        }
> 	return isMaster;
>  }
> 
> @@ -2269,7 +2266,7 @@
>   * <li>the overrides
>   * </ul>
>   * The last source which returns a non-null value is used, so
> -     * the search is done from lowest to highest precedence. Configration
> +     * the search is done from lowest to highest precedence. Configuration
>   * definitions may be self-referential, causing a substitution
>   * to be performed from lower precedence definitions. For example
>   * assume the default property file had the following definition:
> @@ -2284,7 +2281,7 @@
>   * would be <code>foo bar</code>.
>   * <p>
>   * Before the parameter value is returned, any '$' substitutions
> -     * are performed (recursively), and any occurances of the
> +     * are performed (recursively), and any occurrences of the
>   * token <code><gethost></code> are replaced with the name of
>   * the host on which this code is running. If a parsing error
>   * is encountered during symbol resolution, an error message
> @@ -2484,7 +2481,7 @@
>   *         or an exception is thrown while instantiating the provider
>   */
>  private void registerOverrideProviders() throws TestException {
> -	overrideProviders = new ArrayList();
> +	overrideProviders = Collections.synchronizedList(new ArrayList<OverrideProvider>());
> 	String[] providers = 
> 	    parseString(getStringConfigVal("testOverrideProviders", null),
> 			", \t");
> @@ -2661,7 +2658,7 @@
> 	initUniqueString(); // so each test uses a new unique value
> 	this.td = td;
>      dynamicProps = new Properties();
> -	selectedIndexes = new ArrayList();
> +	selectedIndexes = Collections.synchronizedList(new ArrayList<Integer>());
> 	trackKey = getParameterString("track");
>      propertyOverrides = null; // paranoid, probably unnecessary
> 	resolver.setToken("config", currentTag);
> @@ -2730,7 +2727,7 @@
>   *
>   * @return the host list
>   */
> -    public ArrayList getHostList() {
> +    public List getHostList() {
> 	return hostList;
>  }
> 
> @@ -2893,9 +2890,12 @@
>  private int nextIndex(boolean includeMaster) {
> 	ArrayList available = new ArrayList();
> 	int i = (includeMaster ? 0 : 1);
> +        Integer selected;
> +        List selectedInd = selectedIndexes; // copy reference to ensure sync on correct object.

You are not doing a deep copy here, so your "alias" for selectedIndexes is no different than selectedIndexes right?

> +        synchronized (selectedInd){
> 	while (i < hostList.size()) {
> 	    Integer iInt = new Integer(i++);
> -	    if (! selectedIndexes.contains(iInt)) {
> +                if (! selectedInd.contains(iInt)) {
> 		available.add(iInt);
> 	    }
> 	}
> @@ -2903,12 +2903,13 @@
> 	    throw new IllegalStateException("Can't find an available index");
> 	}
> 	int randIndex = new Random().nextInt(available.size());
> -	Integer selected = (Integer) available.get(randIndex);
> -	selectedIndexes.add(selected);
> +            selected = (Integer) available.get(randIndex);
> +            selectedInd.add(selected);
> 	int maxSize = (includeMaster ? hostList.size() : (hostList.size() - 1));
> -	if (selectedIndexes.size() == maxSize) {
> -	    selectedIndexes.clear();
> +            if (selectedInd.size() >= maxSize) {
> +                selectedInd.clear();
> 	}
> +        }
> 	return selected.intValue();
>  }
> 
> @@ -2973,6 +2974,7 @@
> 		try {
> 		    runLock.wait();
> 		} catch (InterruptedException ignore) {
> +                    Thread.currentThread().interrupt(); //Ignore and restore.
> 		}
> 	    }
> 	}
> Index: src/com/sun/jini/qa/harness/QAConfiguration.java
> --- src/com/sun/jini/qa/harness/QAConfiguration.java Base (BASE)
> +++ src/com/sun/jini/qa/harness/QAConfiguration.java Locally Modified (Based On LOCAL)
> @@ -34,8 +34,8 @@
> */
> public class QAConfiguration implements Configuration {
> 
> -    private Configuration testConfiguration;
> -    private Configuration defaultConfiguration;
> +    private final Configuration testConfiguration;
> +    private final Configuration defaultConfiguration;
> 
>  public QAConfiguration(String[] options, QAConfig config) 
> 	throws ConfigurationException
> Index: src/com/sun/jini/qa/harness/Resolver.java
> --- src/com/sun/jini/qa/harness/Resolver.java Base (BASE)
> +++ src/com/sun/jini/qa/harness/Resolver.java Locally Modified (Based On LOCAL)
> @@ -55,8 +55,8 @@
> class Resolver implements Serializable {
> 
>  private static Logger logger = Logger.getLogger("com.sun.jini.qa.harness");
> -    private QAConfig config;
> -    private HashMap tokenMap = new HashMap();
> +    private final QAConfig config;
> +    private final HashMap tokenMap = new HashMap();
> 
>  /** 
>   * Construct an instance of a <code>Resolver</code>.
> @@ -72,8 +72,10 @@
> 	    token = "<" + token + ">";
> 	}
> 	logger.log(Level.FINEST, "setting token " + token + " to " + value);
> +        synchronized (tokenMap){
> 	tokenMap.put(token, value);
>  }
> +    }
> 
>  /** 
>   * Resolves a value in the input string by splitting it into words and
> @@ -419,6 +421,7 @@
> 	if (str == null) {
> 	    return null;
> 	}
> +        synchronized (tokenMap){
> 	Iterator it = tokenMap.keySet().iterator();
> 	while (it.hasNext()) {
> 	    String token = (String) it.next();
> @@ -430,6 +433,7 @@
> 		str = newString;
> 	    }
> 	}
> +        }
> 	while (true) {
> 	    String newString = handleFileToken(str);
> 	    if (newString == str) {
> Index: src/com/sun/jini/qa/harness/SlaveHarness.java
> --- src/com/sun/jini/qa/harness/SlaveHarness.java Base (BASE)
> +++ src/com/sun/jini/qa/harness/SlaveHarness.java Locally Modified (Based On LOCAL)
> @@ -32,6 +32,7 @@
> import java.net.Socket;
> import java.net.UnknownHostException;
> import java.util.ArrayList;
> +import java.util.List;
> import java.util.logging.Handler;
> import java.util.logging.Level;
> import java.util.logging.Logger;
> @@ -136,7 +137,7 @@
>   * @throws TestException on connection timeout
>   */
>  static void connect() throws TestException {
> -	ArrayList hostList = QAConfig.getConfig().getHostList();
> +	List hostList = QAConfig.getConfig().getHostList();
> 	if (hostList.size() < 2) {
> 	    return;
> 	}
> Index: src/com/sun/jini/qa/harness/SlaveTest.java
> --- src/com/sun/jini/qa/harness/SlaveTest.java Base (BASE)
> +++ src/com/sun/jini/qa/harness/SlaveTest.java Locally Modified (Based On LOCAL)
> @@ -26,6 +26,7 @@
> import java.security.PrivilegedAction;
> import java.util.ArrayList;
> import java.util.HashMap;
> +import java.util.List;
> import java.util.logging.Logger;
> import java.util.logging.Level;
> 
> @@ -123,7 +124,7 @@
>   * @param request the <code>SlaveRequest</code> to send
>   */
>  public static void broadcast(SlaveRequest request) {
> -	ArrayList hostList = QAConfig.getConfig().getHostList();
> +	List hostList = QAConfig.getConfig().getHostList();
> 	if (hostList.size() < 2) {
> 	    return;
> 	}
> @@ -147,11 +148,11 @@
>   *        zero or negative, this method returns immediately
>   */ 
>  public static void waitForSlaveDeath(int timeout) {
> -	ArrayList hostList = QAConfig.getConfig().getHostList();
> +	List hostList = QAConfig.getConfig().getHostList();
> 	if (hostList.size() < 2) {
> 	    return;
> 	}
> -	ArrayList slaveList = new ArrayList();
> +	List slaveList = new ArrayList();
> 	for (int i = 1; i < hostList.size(); i++) {
> 	    slaveList.add(hostList.get(i));
> 	}
> Index: src/com/sun/jini/test/spec/discoveryservice/event/MulticastMonitorStop.td
> --- src/com/sun/jini/test/spec/discoveryservice/event/MulticastMonitorStop.td Base (BASE)
> +++ src/com/sun/jini/test/spec/discoveryservice/event/MulticastMonitorStop.td Locally Modified (Based On LOCAL)
> @@ -2,3 +2,7 @@
> testCategories=discoveryservice,discoveryservice_spec,discoveryserviceevent
> include0=../discoveryservice.properties
> include1=simReg6_0regInfo3_0.properties
> +#testjvmargs=-Xdebug,\
> +#-Xrunjdwp:transport=dt_socket+,address=8000+,server=y+,suspend=y,\
> +#-Djava.security.debug=access:failure,\
> +#${testjvmargs}
> \ No newline at end of file
> Running com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td
> Time is Thu Nov 22 07:16:24 CST 2012
> Starting test in separate process with command:
> /usr/lib/jvm/jdk1.7.0_06/jre/bin/java -Djava.security.manager=org.apache.river.api.security.CombinerSecurityManager -Djava.security.policy=file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/harness/policy/defaulttest.policy -cp /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jiniharness.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jinitests.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-platform.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-lib.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/high-scale-lib.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/reference-collections-1.0.1.jar -client -Djava.ext.dirs=/usr/lib/jvm/jdk1.7.0_06/jre/lib/ext:/usr/java/packages/lib/ext:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib-ext:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib-ext -Dcom.sun.jini.jsk.port=9080 -Dcom.sun.jini.qa.port=9081 -Dcom.sun.jini.jsk.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm -Dcom.sun.jini.qa.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa -Dcom.sun.jini.qa.harness.harnessJar=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jiniharness.jar -Dcom.sun.jini.qa.harness.testJar=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jinitests.jar -Dcom.sun.jini.qa.harness.runjiniserver=true -Dcom.sun.jini.qa.harness.runkitserver=true -Djava.security.properties=file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/harness/trust/dynamic-policy.properties -Dcom.sun.jini.qa.harness.testhosts= -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/resources/qa1.logging -Dcom.sun.jini.test.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa -Dcom.sun.jini.test.port=9082 -Dcom.sun.jini.qa.harness.policies=file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/resources/jinitest.policy -Djava.ext.dirs=/usr/lib/jvm/jdk1.7.0_06/jre/lib/ext:/usr/java/packages/lib/ext:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib-ext:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib-ext com.sun.jini.qa.harness.MasterTest com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td 
> 
> TIME: 7:16:25 AM
> 
> QAConfig.loadTestConfiguration FINER: Test Configuration options:
> QAConfig.loadTestConfiguration FINER:    -
> QAConfig.loadTestConfiguration FINER:    net.jini.discovery.LookupDiscovery.multicastAnnouncementInterval = 20000
> QAConfig.loadTestConfiguration FINER:    multicast.ttl = 0
> MasterTest.doTest INFO: 
> ============================== CALLING SETUP() ==============================
> 
> AdminManager.startService FINE: starting qaClassServer
> AdminManager.getAdmin FINEST: getAdmin called with prefix qaClassServer
> FINE: 
> FINE: Parameters for qaClassServer(.0):
> FINE:      type              : classServer
> FINE:      impl              : com.sun.jini.tool.ClassServer
> FINE:      directory         : /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib
> FINE:      options           : 
> AdminManager.startService FINE: starting jiniClassServer
> AdminManager.getAdmin FINEST: getAdmin called with prefix jiniClassServer
> Nov 22, 2012 7:16:25 AM com.sun.jini.tool.ClassServer run
> INFO: ClassServer started [[/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/], port 9081]
> FINE: 
> FINE: Parameters for jiniClassServer(.0):
> FINE:      type              : classServer
> FINE:      impl              : com.sun.jini.tool.ClassServer
> FINE:      directory         : /home/jenkins/jenkins-slave/workspace/River-QA-arm/lib-dl
> FINE:      options           : 
> AdminManager.startService FINE: starting testClassServer
> AdminManager.getAdmin FINEST: getAdmin called with prefix testClassServer
> Nov 22, 2012 7:16:25 AM com.sun.jini.tool.ClassServer run
> INFO: ClassServer started [[/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib-dl/], port 9080]
> FINE: 
> FINE: Parameters for testClassServer(.0):
> FINE:      type              : classServer
> FINE:      impl              : com.sun.jini.tool.ClassServer
> FINE:      directory         : /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib
> FINE:      options           : 
> BaseQATest.setup FINE:  setup()
> BaseQATest.getSetupInfo FINE:  ----- Harness Info ----- 
> BaseQATest.getSetupInfo FINE:  harness codebase         -- no codebase
> Nov 22, 2012 7:16:25 AM com.sun.jini.tool.ClassServer run
> INFO: ClassServer started [[/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/], port 9082]
> BaseQATest.getSetupInfo FINE:  harness classpath        -- /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jiniharness.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jinitests.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-platform.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-lib.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/high-scale-lib.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/reference-collections-1.0.1.jar
> BaseQATest.getSetupInfo FINE:  net.jini.discovery.debug        -- false
> BaseQATest.getSetupInfo FINE:  com.sun.jini.reggie.proxy.debug -- false
> BaseQATest.getSetupInfo FINE:  com.sun.jini.join.debug         -- false
> BaseQATest.getSetupInfo FINE:  com.sun.jini.sdm.debug          -- false
> BaseQATest.getSetupInfo FINE:  max secs event wait             -- 180
> BaseQATest.getSetupInfo FINE:  ----- Lookup Service Info ----- 
> BaseQATest.getSetupInfo FINE:  # of lookup services to start            -- 3
> BaseQATest.getSetupInfo FINE:  # of additional lookup services to start -- 0
> BaseQATest.getSetupInfo FINE:  seconds to wait for discovery            -- 180
> BaseQATest.getSetupInfo FINE:  discard if no announcements in (nSecs =) -- 20
> QAConfig.getServiceHost FINE: Selecting service host
> QAConfig.getServiceHost FINE: Not distributed - selecting this host
> BaseQATest.getTestLocator FINER: getServiceHost returned null
> QAConfig.getServiceHost FINE: Selecting service host
> QAConfig.getServiceHost FINE: Not distributed - selecting this host
> BaseQATest.getTestLocator FINER: getServiceHost returned null
> QAConfig.getServiceHost FINE: Selecting service host
> QAConfig.getServiceHost FINE: Not distributed - selecting this host
> BaseQATest.getTestLocator FINER: getServiceHost returned null
> BaseQATest.startLookup FINE:  starting lookup service 0
> BaseQATest.startLookup FINEST: je-cal-17.apache.org:4160
> AdminManager.startService FINE: starting net.jini.core.lookup.ServiceRegistrar
> AdminManager.getAdmin FINEST: getAdmin called with prefix net.jini.core.lookup.ServiceRegistrar
> QAConfig.getServiceHost FINE: Selecting service host
> QAConfig.getServiceHost FINE: Not distributed - selecting this host
> AbstractServiceAdmin.addServiceExporter FINER: no exporter definition provided
> NonActivatableServiceStarterAdmin.getGroup FINER: Creating shared group
> AdminManager.startService FINE: starting nonActivatableGroup
> AdminManager.getAdmin FINEST: getAdmin called with prefix nonActivatableGroup
> NonActivatableGroupAdmin.start FINER: NonActivatableGroup exec command line: '/usr/lib/jvm/jdk1.7.0_06/jre/bin/java -Djava.rmi.server.codebase=http://je-cal-17:9081/nonactivatablegroup-dl.jar -Djava.security.policy=file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/harness/policy/defaultnonactvm.policy -server -Djava.ext.dirs=/usr/lib/jvm/jdk1.7.0_06/jre/lib/ext:/usr/java/packages/lib/ext:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib-ext:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib-ext -Dcom.sun.jini.jsk.port=9080 -Dcom.sun.jini.qa.port=9081 -Dcom.sun.jini.jsk.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm -Dcom.sun.jini.qa.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa -Dcom.sun.jini.qa.harness.harnessJar=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jiniharness.jar -Dcom.sun.jini.qa.harness.testJar=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jinitests.jar -Dcom.sun.jini.qa.harness.runjiniserver=true -Dcom.sun.jini.qa.harness.runkitserver=true -Djava.security.properties=file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/harness/trust/dynamic-policy.properties -Dcom.sun.jini.qa.harness.testhosts= -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/resources/qa1.logging -Dcom.sun.jini.test.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa -Dcom.sun.jini.test.port=9082 -Dcom.sun.jini.qa.harness.policies=file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/resources/jinitest.policy -cp /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/nonactivatablegroup.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/start.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-platform.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/reference-collections-1.0.1.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/high-scale-lib.jar com.sun.jini.qa.harness.NonActivatableGroupImpl'
> FINE: 
> FINE: Parameters for nonActivatableGroup(.0):
> FINE:      type              : nonactivatablegroup
> FINE:      codebase          : http://je-cal-17:9081/nonactivatablegroup-dl.jar
> FINE:      impl              : com.sun.jini.qa.harness.NonActivatableGroupImpl
> FINE:      policy file       : file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/harness/policy/defaultnonactvm.policy
> FINE:      classpath         : /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/nonactivatablegroup.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/start.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-platform.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/reference-collections-1.0.1.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/high-scale-lib.jar
> FINE:      options           : -server
> FINE:      properties        : -Djava.ext.dirs=/usr/lib/jvm/jdk1.7.0_06/jre/lib/ext:/usr/java/packages/lib/ext:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib-ext:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib-ext
> FINE:                        : -Dcom.sun.jini.jsk.port=9080
> FINE:                        : -Dcom.sun.jini.qa.port=9081
> FINE:                        : -Dcom.sun.jini.jsk.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm
> FINE:                        : -Dcom.sun.jini.qa.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa
> FINE:                        : -Dcom.sun.jini.qa.harness.harnessJar=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jiniharness.jar
> FINE:                        : -Dcom.sun.jini.qa.harness.testJar=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/lib/jinitests.jar
> FINE:                        : -Dcom.sun.jini.qa.harness.runjiniserver=true
> FINE:                        : -Dcom.sun.jini.qa.harness.runkitserver=true
> FINE:                        : -Djava.security.properties=file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/harness/trust/dynamic-policy.properties
> FINE:                        : -Dcom.sun.jini.qa.harness.testhosts=
> FINE:                        : -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/resources/qa1.logging
> FINE:                        : -Dcom.sun.jini.test.home=/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa
> FINE:                        : -Dcom.sun.jini.test.port=9082
> FINE:                        : -Dcom.sun.jini.qa.harness.policies=file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/resources/jinitest.policy
> QAConfig.getComponentFile FINEST: failed existance check on com/sun/jini/test/share/simulator.policy
> QAConfig.getComponentFile FINEST: failed existance check on /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/spec/lookupdiscovery/com/sun/jini/test/share/simulator.policy
> FINE: 
> FINE: Parameters for net.jini.core.lookup.ServiceRegistrar(.0):
> FINE:      type              : transient
> FINE:      codebase          : http://je-cal-17:9082/qa1-lookup-simulator-dl.jar
> FINE:      impl              : com.sun.jini.test.services.lookupsimulator.LookupSimulatorImpl
> FINE:      component name    : com.sun.jini.reggie
> FINE:      policy file       : file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> FINE:      classpath         : lib/qa1-lookup-simulator.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-platform.jar
> FINE:      service conf file : -
> FINE:      starter conf file : -
> FINE:      proxy preparer    : test.reggiePreparer
> FINE:      option args 0     : -
> FINE:      option args 1     : com.sun.jini.reggie.initialMemberGroups = new String[]{"LDGroup0_A_je-cal-17_1353590184346", "LDGroup0_B_je-cal-17_1353590184346", "LDGroup0_C_je-cal-17_1353590184346"}
> FINE:      option args 2     : net.jini.discovery.LookupDiscovery.multicastAnnouncementInterval=20000
> FINE:      option args 3     : com.sun.jini.reggie.multicastAnnouncementInterval=20000
> FINE:      option args 4     : multicast.ttl=0
> FINE: 
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: fiddler
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: fiddler
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: reggie
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: reggie
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: group
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: group
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mercury
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mercury
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: norm
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: norm
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mahalo
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mahalo
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: outrigger
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: outrigger
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: test
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: test
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> LocatorsUtil.displayLocator FINE:   locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.startLookup FINE:    memberGroup(s) = LDGroup0_A_je-cal-17_1353590184346,LDGroup0_B_je-cal-17_1353590184346,LDGroup0_C_je-cal-17_1353590184346
> BaseQATest.startInitLookups FINEST: service host is 'je-cal-17.apache.org', this host is 'je-cal-17'
> BaseQATest.startInitLookups FINEST: init locator 0 = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.portInUse FINE: port in use: 4160
> BaseQATest.startLookup FINE:  starting lookup service 1
> BaseQATest.startLookup FINEST: je-cal-17.apache.org:0
> AdminManager.startService FINE: starting net.jini.core.lookup.ServiceRegistrar
> AdminManager.getAdmin FINEST: getAdmin called with prefix net.jini.core.lookup.ServiceRegistrar
> QAConfig.getServiceHost FINE: Selecting service host
> QAConfig.getServiceHost FINE: Not distributed - selecting this host
> AbstractServiceAdmin.addServiceExporter FINER: no exporter definition provided
> QAConfig.getComponentFile FINEST: failed existance check on com/sun/jini/test/share/simulator.policy
> QAConfig.getComponentFile FINEST: failed existance check on /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/spec/lookupdiscovery/com/sun/jini/test/share/simulator.policy
> FINE: 
> FINE: Parameters for net.jini.core.lookup.ServiceRegistrar.1:
> FINE:      type              : transient
> FINE:      codebase          : http://je-cal-17:9082/qa1-lookup-simulator-dl.jar
> FINE:      impl              : com.sun.jini.test.services.lookupsimulator.LookupSimulatorImpl
> FINE:      component name    : com.sun.jini.reggie
> FINE:      policy file       : file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> FINE:      classpath         : lib/qa1-lookup-simulator.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-platform.jar
> FINE:      service conf file : -
> FINE:      starter conf file : -
> FINE:      proxy preparer    : test.reggiePreparer
> FINE:      option args 0     : -
> FINE:      option args 1     : com.sun.jini.reggie.initialMemberGroups = new String[]{"LDGroup1_A_je-cal-17_1353590184346", "LDGroup1_B_je-cal-17_1353590184346", "LDGroup1_C_je-cal-17_1353590184346", "LDGroup1_D_je-cal-17_1353590184346"}
> FINE:      option args 2     : net.jini.discovery.LookupDiscovery.multicastAnnouncementInterval=20000
> FINE:      option args 3     : com.sun.jini.reggie.multicastAnnouncementInterval=20000
> FINE:      option args 4     : multicast.ttl=0
> FINE: 
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mahalo
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mahalo
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: fiddler
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: fiddler
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: norm
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: norm
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mercury
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mercury
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: outrigger
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: outrigger
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: group
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: group
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: reggie
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: reggie
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: test
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: test
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> DiscoveryProtocolSimulator$UnicastThread.<init> FINE: Failed to bind to default port
> java.net.BindException: Address already in use
> 	at java.net.PlainSocketImpl.socketBind(Native Method)
> 	at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
> 	at java.net.ServerSocket.bind(ServerSocket.java:376)
> 	at java.net.ServerSocket.<init>(ServerSocket.java:237)
> 	at java.net.ServerSocket.<init>(ServerSocket.java:128)
> 	at com.sun.jini.test.share.DiscoveryProtocolSimulator$UnicastThread.<init>(DiscoveryProtocolSimulator.java:576)
> 	at com.sun.jini.test.share.DiscoveryProtocolSimulator.init(DiscoveryProtocolSimulator.java:1111)
> 	at com.sun.jini.test.share.DiscoveryProtocolSimulator.<init>(DiscoveryProtocolSimulator.java:250)
> 	at com.sun.jini.test.share.BaseQATest.startLookup(BaseQATest.java:1657)
> 	at com.sun.jini.test.share.BaseQATest.startInitLookups(BaseQATest.java:1538)
> 	at com.sun.jini.test.share.BaseQATest.setup(BaseQATest.java:1063)
> 	at com.sun.jini.test.spec.lookupdiscovery.AbstractBaseTest.setup(AbstractBaseTest.java:76)
> 	at com.sun.jini.test.spec.lookupdiscovery.Discovered.setup(Discovered.java:69)
> 	at com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:219)
> 	at com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> LocatorsUtil.displayLocator FINE:   locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.startLookup FINE:    memberGroup(s) = LDGroup1_A_je-cal-17_1353590184346,LDGroup1_B_je-cal-17_1353590184346,LDGroup1_C_je-cal-17_1353590184346,LDGroup1_D_je-cal-17_1353590184346
> BaseQATest.startInitLookups FINEST: service host is 'je-cal-17.apache.org', this host is 'je-cal-17'
> BaseQATest.startInitLookups FINEST: init locator 1 = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.portInUse FINE: port in use: 4160
> BaseQATest.startLookup FINE:  starting lookup service 2
> BaseQATest.startLookup FINEST: je-cal-17.apache.org:0
> AdminManager.startService FINE: starting net.jini.core.lookup.ServiceRegistrar
> AdminManager.getAdmin FINEST: getAdmin called with prefix net.jini.core.lookup.ServiceRegistrar
> QAConfig.getServiceHost FINE: Selecting service host
> QAConfig.getServiceHost FINE: Not distributed - selecting this host
> AbstractServiceAdmin.addServiceExporter FINER: no exporter definition provided
> QAConfig.getComponentFile FINEST: failed existance check on com/sun/jini/test/share/simulator.policy
> QAConfig.getComponentFile FINEST: failed existance check on /home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/spec/lookupdiscovery/com/sun/jini/test/share/simulator.policy
> FINE: 
> FINE: Parameters for net.jini.core.lookup.ServiceRegistrar.2:
> FINE:      type              : transient
> FINE:      codebase          : http://je-cal-17:9082/qa1-lookup-simulator-dl.jar
> FINE:      impl              : com.sun.jini.test.services.lookupsimulator.LookupSimulatorImpl
> FINE:      component name    : com.sun.jini.reggie
> FINE:      policy file       : file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> FINE:      classpath         : lib/qa1-lookup-simulator.jar:/home/jenkins/jenkins-slave/workspace/River-QA-arm/lib/jsk-platform.jar
> FINE:      service conf file : -
> FINE:      starter conf file : -
> FINE:      proxy preparer    : test.reggiePreparer
> FINE:      option args 0     : -
> FINE:      option args 1     : com.sun.jini.reggie.initialMemberGroups = new String[]{"LDGroup2_A_je-cal-17_1353590184346", "LDGroup2_B_je-cal-17_1353590184346", "LDGroup2_C_je-cal-17_1353590184346"}
> FINE:      option args 2     : net.jini.discovery.LookupDiscovery.multicastAnnouncementInterval=20000
> FINE:      option args 3     : com.sun.jini.reggie.multicastAnnouncementInterval=20000
> FINE:      option args 4     : multicast.ttl=0
> FINE: 
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: fiddler
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: fiddler
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: norm
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: norm
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: outrigger
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: outrigger
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: test
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: test
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mercury
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mercury
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: group
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: group
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mahalo
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: mahalo
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> NonActGrp-out: Problem parsing policy: file:/home/jenkins/jenkins-slave/workspace/River-QA-arm/qa/src/com/sun/jini/test/share/simulator.policy
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: reggie
> NonActGrp-out: org.apache.river.api.security.PolicyUtils$ExpansionFailedException: Unknown key: reggie
> NonActGrp-out: 	at org.apache.river.api.security.PolicyUtils.expand(PolicyUtils.java:167)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.resolveGrant(DefaultPolicyParser.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.DefaultPolicyParser.parse(DefaultPolicyParser.java:140)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:415)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile$2.run(ConcurrentPolicyFile.java:401)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.initialize(ConcurrentPolicyFile.java:400)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:234)
> NonActGrp-out: 	at org.apache.river.api.security.ConcurrentPolicyFile.<init>(ConcurrentPolicyFile.java:214)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> NonActGrp-out: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> NonActGrp-out: 	at java.lang.Class.newInstance0(Class.java:372)
> NonActGrp-out: 	at java.lang.Class.newInstance(Class.java:325)
> NonActGrp-out: 	at net.jini.security.policy.PolicyFileProvider.<init>(PolicyFileProvider.java:196)
> NonActGrp-out: 	at com.sun.jini.start.NonActivatableServiceDescriptor.create(NonActivatableServiceDescriptor.java:626)
> NonActGrp-out: 	at com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl.startService(NonActivatableGroupImpl.java:182)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> NonActGrp-out: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> NonActGrp-out: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> NonActGrp-out: 	at java.lang.reflect.Method.invoke(Method.java:601)
> NonActGrp-out: 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> NonActGrp-out: 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> NonActGrp-out: 	at java.security.AccessController.doPrivileged(Native Method)
> NonActGrp-out: 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> NonActGrp-out: 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> NonActGrp-out: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> NonActGrp-out: 	at java.lang.Thread.run(Thread.java:722)
> DiscoveryProtocolSimulator$UnicastThread.<init> FINE: Failed to bind to default port
> java.net.BindException: Address already in use
> 	at java.net.PlainSocketImpl.socketBind(Native Method)
> 	at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
> 	at java.net.ServerSocket.bind(ServerSocket.java:376)
> 	at java.net.ServerSocket.<init>(ServerSocket.java:237)
> 	at java.net.ServerSocket.<init>(ServerSocket.java:128)
> 	at com.sun.jini.test.share.DiscoveryProtocolSimulator$UnicastThread.<init>(DiscoveryProtocolSimulator.java:576)
> 	at com.sun.jini.test.share.DiscoveryProtocolSimulator.init(DiscoveryProtocolSimulator.java:1111)
> 	at com.sun.jini.test.share.DiscoveryProtocolSimulator.<init>(DiscoveryProtocolSimulator.java:250)
> 	at com.sun.jini.test.share.BaseQATest.startLookup(BaseQATest.java:1657)
> 	at com.sun.jini.test.share.BaseQATest.startInitLookups(BaseQATest.java:1538)
> 	at com.sun.jini.test.share.BaseQATest.setup(BaseQATest.java:1063)
> 	at com.sun.jini.test.spec.lookupdiscovery.AbstractBaseTest.setup(AbstractBaseTest.java:76)
> 	at com.sun.jini.test.spec.lookupdiscovery.Discovered.setup(Discovered.java:69)
> 	at com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:219)
> 	at com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> LocatorsUtil.displayLocator FINE:   locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.startLookup FINE:    memberGroup(s) = LDGroup2_A_je-cal-17_1353590184346,LDGroup2_B_je-cal-17_1353590184346,LDGroup2_C_je-cal-17_1353590184346
> BaseQATest.startInitLookups FINEST: service host is 'je-cal-17.apache.org', this host is 'je-cal-17'
> BaseQATest.startInitLookups FINEST: init locator 2 = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> AbstractBaseTest.setup FINE: creating a lookup discovery initially configured to discover NO_GROUPS
> MasterTest.doTest INFO: 
> =============================== CALLING RUN() ===============================
> 
> Discovered.run FINE: run()
> AbstractBaseTest.doDiscovery FINE: set groups to discover -- 
> AbstractBaseTest.doDiscovery FINE:    LDGroup0_A_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup0_B_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup0_C_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup1_A_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup1_B_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup1_C_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup1_D_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup2_A_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup2_B_je-cal-17_1353590184346
> AbstractBaseTest.doDiscovery FINE:    LDGroup2_C_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:  for DISCOVERY events -- waiting at least 30 seconds, but no more than 180 seconds ...
> DiscoveryProtocolSimulator$MulticastThread.run FINE: Received valid multicast for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> DiscoveryProtocolSimulator$MulticastThread.run FINE: Received valid multicast for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> DiscoveryProtocolSimulator$MulticastThread.run FINE: Received valid multicast for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> DiscoveryProtocolSimulator$MulticastThread.run FINE: Received valid multicast for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> DiscoveryProtocolSimulator$MulticastThread.run FINE: Received valid multicast for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> DiscoveryProtocolSimulator$AddressTask.run FINE: Responding to multicast with unicast from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> DiscoveryProtocolSimulator$AddressTask.run FINE: Responding to multicast with unicast from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> DiscoveryProtocolSimulator$AddressTask.run FINE: Responding to multicast with unicast from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> DiscoveryProtocolSimulator$MulticastThread.run FINE: Received valid multicast for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> DiscoveryProtocolSimulator.respond FINE: Responded to unicast request for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> DiscoveryProtocolSimulator.respond FINE: Responded to unicast request for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> DiscoveryProtocolSimulator.respond FINE: Responded to unicast request for ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest$LookupListener.discovered FINE:  discovery event received -- 1 lookup(s)
> BaseQATest$LookupListener.discovered FINE:      regs[0] is in lookupList
> LocatorsUtil.displayLocator FINE:   discovered locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest$LookupListener.discovered FINE:    discovered member group(s) = LDGroup0_A_je-cal-17_1353590184346,LDGroup0_B_je-cal-17_1353590184346,LDGroup0_C_je-cal-17_1353590184346
> BaseQATest$LookupListener.discovered FINE:  number of currently discovered lookup(s) = 1
> BaseQATest$LookupListener.discovered FINE:  discovery event received -- 1 lookup(s)
> BaseQATest$LookupListener.discovered FINE:      regs[0] is in lookupList
> LocatorsUtil.displayLocator FINE:   discovered locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest$LookupListener.discovered FINE:    discovered member group(s) = LDGroup2_A_je-cal-17_1353590184346,LDGroup2_B_je-cal-17_1353590184346,LDGroup2_C_je-cal-17_1353590184346
> BaseQATest$LookupListener.discovered FINE:  number of currently discovered lookup(s) = 2
> BaseQATest$LookupListener.discovered FINE:  discovery event received -- 1 lookup(s)
> BaseQATest$LookupListener.discovered FINE:      regs[0] is in lookupList
> LocatorsUtil.displayLocator FINE:   discovered locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest$LookupListener.discovered FINE:    discovered member group(s) = LDGroup1_A_je-cal-17_1353590184346,LDGroup1_B_je-cal-17_1353590184346,LDGroup1_C_je-cal-17_1353590184346,LDGroup1_D_je-cal-17_1353590184346
> BaseQATest$LookupListener.discovered FINE:  number of currently discovered lookup(s) = 3
> 
> TIME: 7:16:50 AM
> 
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> 
> TIME: 7:17:01 AM
> 
> BaseQATest.waitForDiscovery FINE:  initial wait period complete ... waiting at most 150 more seconds ...
> BaseQATest.waitForDiscovery FINE:    discoveredMap.size == 3
> BaseQATest.waitForDiscovery FINE:    discoveredMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[0] == LDGroup1_A_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[1] == LDGroup1_B_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[2] == LDGroup1_C_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[3] == LDGroup1_D_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:    discoveredMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[0] == LDGroup2_A_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[1] == LDGroup2_B_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[2] == LDGroup2_C_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:    discoveredMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[0] == LDGroup0_A_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[1] == LDGroup0_B_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      discoveredMap.groups[2] == LDGroup0_C_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:    expectedDiscoveredMap.size == 3
> BaseQATest.waitForDiscovery FINE:    expectedDiscoveredMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[0] == LDGroup1_A_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[1] == LDGroup1_B_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[2] == LDGroup1_C_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[3] == LDGroup1_D_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:    expectedDiscoveredMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[0] == LDGroup2_A_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[1] == LDGroup2_B_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[2] == LDGroup2_C_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:    expectedDiscoveredMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[0] == LDGroup0_A_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[1] == LDGroup0_B_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:      expectedDiscoveredMap.groups[2] == LDGroup0_C_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByLoc FINE:  comparing locators ... 
> BaseQATest.locGroupsMapsEqualByLoc FINE:  locators set 0 -- 
> BaseQATest.locGroupsMapsEqualByLoc FINE:     ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.locGroupsMapsEqualByLoc FINE:     ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.locGroupsMapsEqualByLoc FINE:     ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.locGroupsMapsEqualByLoc FINE:  locators set 1 -- 
> BaseQATest.locGroupsMapsEqualByLoc FINE:     ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.locGroupsMapsEqualByLoc FINE:     ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.locGroupsMapsEqualByLoc FINE:     ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.locGroupsMapsEqualByGroups FINE:  comparing group sets of each lookup ... 
> BaseQATest.locGroupsMapsEqualByGroups FINE:     set 0 -- locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[0] = LDGroup1_A_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[1] = LDGroup1_B_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[2] = LDGroup1_C_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[3] = LDGroup1_D_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:     set 1 -- locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[0] = LDGroup1_A_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[1] = LDGroup1_B_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[2] = LDGroup1_C_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[3] = LDGroup1_D_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:     set 0 -- locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[0] = LDGroup2_A_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[1] = LDGroup2_B_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[2] = LDGroup2_C_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:     set 1 -- locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[0] = LDGroup2_A_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[1] = LDGroup2_B_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[2] = LDGroup2_C_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:     set 0 -- locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[0] = LDGroup0_A_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[1] = LDGroup0_B_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[2] = LDGroup0_C_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:     set 1 -- locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[0] = LDGroup0_A_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[1] = LDGroup0_B_je-cal-17_1353590184346
> BaseQATest.locGroupsMapsEqualByGroups FINE:       group[2] = LDGroup0_C_je-cal-17_1353590184346
> BaseQATest.waitForDiscovery FINE:    events expected (3) == events received (3), all locators equal
> BaseQATest.waitForDiscovery FINE:  DISCOVERY wait period complete
> BaseQATest.waitForDiscovery FINE:  3 discovery event(s) expected, 3 discovery event(s) received
> BaseQATest.verifyAnnouncementsSent FINE:  number of announcements to wait for    -- 3
> BaseQATest.verifyAnnouncementsSent FINE:  number of intervals to wait through    -- 3
> BaseQATest.verifyAnnouncementsSent FINE:  gen 0 - waiting ... announcements so far -- 2
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> DiscoveryProtocolSimulator$AnnounceThread.announce FINE: Sending packets from ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> 
> TIME: 7:17:21 AM
> 
> BaseQATest.verifyAnnouncementsSent FINE:  gen 0 - waiting ... announcements so far -- 3
> BaseQATest.verifyAnnouncementsSent FINE:  gen 0 - wait complete ... announcements  -- 3
> BaseQATest.verifyAnnouncementsSent FINE:  gen 1 - waiting ... announcements so far -- 3
> BaseQATest.verifyAnnouncementsSent FINE:  gen 1 - wait complete ... announcements  -- 3
> BaseQATest.verifyAnnouncementsSent FINE:  gen 2 - waiting ... announcements so far -- 3
> BaseQATest.verifyAnnouncementsSent FINE:  gen 2 - wait complete ... announcements  -- 3
> BaseQATest.stopAnnouncements FINE:  stop multicast announcements from lookup service 0 ...
> DiscoveryProtocolSimulator.stopAnnouncements FINE:    stopAnnouncements entered
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      interrupted all threads
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      terminated task manager
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      unicastRequestThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      multicastRequestThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      multicastAnnouncementThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      close all request sockets
> DiscoveryProtocolSimulator.stopAnnouncements FINE:    stopAnnouncements exited
> BaseQATest.stopAnnouncements FINE:  stop multicast announcements from lookup service 1 ...
> DiscoveryProtocolSimulator.stopAnnouncements FINE:    stopAnnouncements entered
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      interrupted all threads
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      terminated task manager
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      unicastRequestThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      multicastRequestThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      multicastAnnouncementThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      close all request sockets
> DiscoveryProtocolSimulator.stopAnnouncements FINE:    stopAnnouncements exited
> BaseQATest.stopAnnouncements FINE:  stop multicast announcements from lookup service 2 ...
> DiscoveryProtocolSimulator.stopAnnouncements FINE:    stopAnnouncements entered
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      interrupted all threads
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      terminated task manager
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      unicastRequestThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      multicastRequestThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      multicastAnnouncementThread.join()
> DiscoveryProtocolSimulator.stopAnnouncements FINE:      close all request sockets
> DiscoveryProtocolSimulator.stopAnnouncements FINE:    stopAnnouncements exited
> BaseQATest.replaceMemberGroups FINE:  lookup service 0 - replacing member groups with -- 
> BaseQATest.replaceMemberGroups FINE:  failed to change member groups for lookup service 0
> java.rmi.ConnectIOException: I/O exception connecting to BasicObjectEndpoint[37abfa80-a6e1-4340-9cdb-37caccf52d5a,TcpEndpoint[64.20.246.146:46371]]; nested exception is: 
> 	java.io.IOException: interrupt waiting for connection header
> 	at net.jini.jeri.BasicInvocationHandler.wrapSafeIOException(BasicInvocationHandler.java:893)
> 	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:711)
> 	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethod(BasicInvocationHandler.java:659)
> 	at net.jini.jeri.BasicInvocationHandler.invoke(BasicInvocationHandler.java:528)
> 	at com.sun.jini.test.services.lookupsimulator.$Proxy0.getMemberGroups(Unknown Source)
> 	at com.sun.jini.test.services.lookupsimulator.LookupSimulatorProxy.getMemberGroups(LookupSimulatorProxy.java:144)
> 	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2578)
> 	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2628)
> 	at com.sun.jini.test.share.BaseQATest.replaceMemberGroups(BaseQATest.java:2660)
> 	at com.sun.jini.test.spec.lookupdiscovery.MulticastMonitorStopReplace.run(MulticastMonitorStopReplace.java:103)
> 	at com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:254)
> 	at com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
> Caused by: java.io.IOException: interrupt waiting for connection header
> 	at com.sun.jini.jeri.internal.mux.Mux.start(Mux.java:238)
> 	at com.sun.jini.jeri.internal.mux.MuxClient.start(MuxClient.java:35)
> 	at net.jini.jeri.connection.ConnectionManager$OutboundMux.newRequest(ConnectionManager.java:375)
> 	at net.jini.jeri.connection.ConnectionManager$ReqIterator.next(ConnectionManager.java:665)
> 	at net.jini.jeri.BasicObjectEndpoint$1.next(BasicObjectEndpoint.java:370)
> 	at net.jini.jeri.BasicInvocationHandler.invokeRemoteMethodOnce(BasicInvocationHandler.java:708)
> 	... 10 more
> Caused by: java.lang.InterruptedException
> 	at java.lang.Object.wait(Native Method)
> 	at com.sun.jini.jeri.internal.mux.Mux.start(Mux.java:230)
> 	... 15 more
> BaseQATest.replaceMemberGroups FINE:  lookup service 1 - replacing member groups with -- 
> BaseQATest.replaceMemberGroups FINE:    newGroups[0] = LDGroup2_A_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[1] = LDGroup2_B_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[2] = LDGroup2_C_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:  lookup service 2 - replacing member groups with -- 
> BaseQATest.replaceMemberGroups FINE:    newGroups[0] = LDGroup1_A_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[1] = LDGroup1_B_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[2] = LDGroup1_C_je-cal-17_1353590184346_new
> BaseQATest.replaceMemberGroups FINE:    newGroups[3] = LDGroup1_D_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:  for DISCARD events -- waiting at least 30 seconds, but no more than 180 seconds ...
> 
> TIME: 7:17:51 AM
> 
> BaseQATest.waitForDiscard FINE:  initial wait period complete ... waiting at most 150 more seconds ...
> BaseQATest.waitForDiscard FINE:    discardedMap.size == 0
> BaseQATest.waitForDiscard FINE:    expectedDiscardedMap.size == 3
> BaseQATest.waitForDiscard FINE:    expectedDiscardedMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[0] == LDGroup1_A_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[1] == LDGroup1_B_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[2] == LDGroup1_C_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[3] == LDGroup1_D_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:    expectedDiscardedMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[0] == LDGroup2_A_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[1] == LDGroup2_B_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[2] == LDGroup2_C_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:    expectedDiscardedMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[0] == LDGroup0_A_je-cal-17_1353590184346
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[1] == LDGroup0_B_je-cal-17_1353590184346
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[2] == LDGroup0_C_je-cal-17_1353590184346
> 
> TIME: 7:18:11 AM
> 
> BaseQATest$LookupListener.discarded FINE:  discard event received -- 1 lookup(s)
> BaseQATest$LookupListener.discarded FINE:    discarded locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest$LookupListener.discarded FINE:    discarded member group(s) = LDGroup1_A_je-cal-17_1353590184346_new,LDGroup1_B_je-cal-17_1353590184346_new,LDGroup1_C_je-cal-17_1353590184346_new,LDGroup1_D_je-cal-17_1353590184346_new
> BaseQATest$LookupListener.discarded FINE:  number of currently discovered lookup(s) = 2
> BaseQATest$LookupListener.discarded FINE:  discard event received -- 1 lookup(s)
> BaseQATest$LookupListener.discarded FINE:    discarded locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest$LookupListener.discarded FINE:    discarded member group(s) = LDGroup2_A_je-cal-17_1353590184346_new,LDGroup2_B_je-cal-17_1353590184346_new,LDGroup2_C_je-cal-17_1353590184346_new
> BaseQATest$LookupListener.discarded FINE:  number of currently discovered lookup(s) = 1
> BaseQATest.waitForDiscard FINE:    discardedMap.size == 2
> BaseQATest.waitForDiscard FINE:    discardedMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.waitForDiscard FINE:      discardedMap.groups[0] == LDGroup1_A_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      discardedMap.groups[1] == LDGroup1_B_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      discardedMap.groups[2] == LDGroup1_C_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      discardedMap.groups[3] == LDGroup1_D_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:    discardedMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.waitForDiscard FINE:      discardedMap.groups[0] == LDGroup2_A_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      discardedMap.groups[1] == LDGroup2_B_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      discardedMap.groups[2] == LDGroup2_C_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:    expectedDiscardedMap.size == 3
> BaseQATest.waitForDiscard FINE:    expectedDiscardedMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:41765/], [null]]
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[0] == LDGroup1_A_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[1] == LDGroup1_B_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[2] == LDGroup1_C_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[3] == LDGroup1_D_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:    expectedDiscardedMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:36067/], [null]]
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[0] == LDGroup2_A_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[1] == LDGroup2_B_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[2] == LDGroup2_C_je-cal-17_1353590184346_new
> BaseQATest.waitForDiscard FINE:    expectedDiscardedMap.locator = ConstrainableLookupLocator[[jini://je-cal-17.apache.org:4160/], [null]]
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[0] == LDGroup0_A_je-cal-17_1353590184346
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[1] == LDGroup0_B_je-cal-17_1353590184346
> BaseQATest.waitForDiscard FINE:      expectedDiscardedMap.groups[2] == LDGroup0_C_je-cal-17_1353590184346
> 
> TIME: 7:20:20 AM
> 
> BaseQATest.waitForDiscard FINE:    events expected (3) != events received (2)
> BaseQATest.waitForDiscard FINE:  DISCARD wait period complete
> com.sun.jini.qa.harness.TestException: discard failed -- waited 150 seconds (2 minutes) -- 3 discard event(s) expected, 2 discard event(s) received
> 	at com.sun.jini.test.share.BaseQATest.waitForDiscard(BaseQATest.java:2025)
> 	at com.sun.jini.test.spec.lookupdiscovery.MulticastMonitorStopReplace.run(MulticastMonitorStopReplace.java:109)
> 	at com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:254)
> 	at com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)
> MasterTest.doTest INFO: 
> ============================ CALLING TEARDOWN() =============================
> 
> AbstractBaseTest.tearDown FINE: tearDown - terminating LookupDiscovery instance 0
> BaseQATest.tearDown FINE:  tearDown - terminating lookup service(s)
> QATest.tearDown FINE: Destroying remaining managed services
> AdminManager.destroyService FINE: destroying service: class com.sun.jini.test.services.lookupsimulator.LookupSimulatorConstrainableProxy
> AdminManager.destroyService FINE: destroying service: class com.sun.jini.test.services.lookupsimulator.LookupSimulatorConstrainableProxy
> AdminManager.destroyService FINE: destroying service: class com.sun.jini.test.services.lookupsimulator.LookupSimulatorConstrainableProxy
> AdminManager.destroyService FINE: destroying service: class com.sun.jini.qa.harness.NonActivatableGroupImpl$GroupImpl_Stub
> AdminManager.destroyService FINE: destroying service: class com.sun.jini.tool.ClassServer
> Nov 22, 2012 7:20:24 AM com.sun.jini.tool.ClassServer terminate
> INFO: ClassServer terminated [port 9082]
> AdminManager.destroyService FINE: destroying service: class com.sun.jini.tool.ClassServer
> Nov 22, 2012 7:20:24 AM com.sun.jini.tool.ClassServer terminate
> INFO: ClassServer terminated [port 9082]
> Nov 22, 2012 7:20:24 AM com.sun.jini.tool.ClassServer terminate
> INFO: ClassServer terminated [port 9081]
> AdminManager.destroyService FINE: destroying service: class com.sun.jini.tool.ClassServer
> Nov 22, 2012 7:20:24 AM com.sun.jini.tool.ClassServer terminate
> INFO: ClassServer terminated [port 9081]
> Nov 22, 2012 7:20:24 AM com.sun.jini.tool.ClassServer terminate
> INFO: ClassServer terminated [port 9080]
> Nov 22, 2012 7:20:24 AM com.sun.jini.tool.ClassServer terminate
> INFO: ClassServer terminated [port 9080]
> 
> TIME: 7:20:24 AM
> 
> Test process was destroyed and returned code 1
> com/sun/jini/test/spec/lookupdiscovery/MulticastMonitorStopReplace.td
> Test Failed: Test Failed: com.sun.jini.qa.harness.TestException: discard failed -- waited 150 seconds (2 minutes) -- 3 discard event(s) expected, 2 discard event(s) received
>