River Scalability - TransactionManager Service Mahalo

[Peter Firmstone <ji...@zeus.net.au>]

This is a brief comparison between our current release River 2.2.2 and 
what I'm playing with in River qa-refactor in skunk.

The test run:

com/sun/jini/test/impl/mahalo/RandomStressTest.td

The tests proceeds for about 1 minute, both tests pass.

The test description states:

/This class subjects the TransactionManager to randomly selected 
stresses in an effort to subject it to the types of load which may be 
encountered in a real situation.

/The test runs 1000 random tasks from a thread pool (TaskManager), then 
sleeps for a designated period, wakes up then finishes.

This is the result in the remote JVM which those random tasks are being 
executed on.

Test analysis for River 2.2.2:

    All 4 cores running at 100%
    Live threads 1002 (mostly TaskManager task threads, 16 of which
    spend most of the test in 100% monitor state)
    Daemon threads 54

    Hot Spots:
    com.sun.jini.thread.RetryTask.run()                                
                                 5,151ms(32.1%), 71 invocations
    net.jini.jeri.connection.ConnectionManager$Outbound$Input.read()   
    3189ms(19.9%), 133 invocations
    com.sun.jini.jeri.internal.runtime.Target$2.run()                   
                        1958ms (12.2%), 54 invocations
    com.sun.jini.thread.TaskManager$TaskThread.run()                   
                  1,558ms (9.7%), 2 invocations
    net.jini.jeri.BasicObjectEndpoint.writeObject(                   
                               934ms(5.8%), 103 invocations



Test analysis for River (skunk) qa-refactor:

    All 4 cores running at 100%
    Live threads 188 (with only 3 threads spending about 1.6% in monitor
    state)
    Daemon threads 183

    Hot Spots:
    net.jini.jeri.BasicInvocationDispatcher.invoke(                   
                             937,503ms (90.2%), 2116 invocations
    com.sun.jini.thread.RetryTask.run()                               
                                   42,804ms (4.1%) 16,649 invocations
    net.jini.jeri.connection.ConnectionManager$Outbound$Input.read()   
    26,653ms (2.3%) 35,932 invocations
    com.sun.jini.thread.ThreadPool$Worker.run()                       
                         10,385 ms (1%) 16 invocations
    net.jini.jeri.BasicObjectEndpoint.writeObject(                   
                                7,682 ms (0.7%) 17,542 invocations


The real question is, why does River qa-refactor perform 17,542 
writeObject invocations, with 188 threads, while River 2.2.2 only 
performs 103 writeObject invocations while using over 1000 threads?

Why does River qa-refactor RetryTask run 16,649 times (2.57ms each), but 
River 2.2.2 only runs 71 times  (72.55ms each) as the primary hotspot?

Obviously something that is run much more is going to benefit from the 
hotspot compiler, but the numbers and thread counts are difficult to 
explain, other than River 2.2.2 is spending most of it's time bogged 
down with contention?

Anyone have time to investigate?

Regards,

Peter.