You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Peter Firmstone <ji...@zeus.net.au> on 2014/05/12 16:42:56 UTC
JERI Scalability Testing
Thought you may find these results interesting, just killed some more
latent concurrency bugs in JERI.
* Mahalo stress tests are now running with 0% contention at close to
raw socket speed.
* ClassLoading is thread confined for each classloader to avoid
contention.
* All underlying infrastructure takes advantage of modern Executors
and concurrency utils.
* TaskManager has been replaced and is no longer needed.
* There are no unnecessary DNS calls, only those required to make
connections.
* Security checks have an imperceptible impact on performance.
* RFC3986 compliant Uri class uses bitshift operations for ASCII
string manipulation.
I need someone with access to decent hardware to really test this out.
Hot Spots - Method Self time [%] Self time Self time (CPU) Samples
java.net.ServerSocket.accept() 59.933640 161513.736 ms 161513.736 ms 6
java.net.DatagramSocket.receive() 29.966820 80756.868 ms 80756.868 ms 3
java.io.ObjectInputStream.defaultReadObject() 2.706319 7293.194 ms
7293.194 ms 134
java.lang.Class.forName() 2.082170 5611.19 ms 5611.19 ms 9
com.sun.jini.start.AggregatePolicyProvider.getContext() 0.925748
2602.776 ms 2494.776 ms 7
java.util.concurrent.FutureTask.get() 0.911941 182311.234 ms
2457.569 ms 271
java.lang.ClassLoader.loadClass() 0.732275 1973.389 ms 1973.389 ms 7
java.util.Collections$UnmodifiableCollection.isEmpty() 0.516405
1391.647 ms 1391.647 ms 1
com.sun.jini.jeri.internal.mux.Session$MuxInputStream.read() 0.516405
182643.391 ms 1391.647 ms 246
java.io.ObjectOutputStream.writeObject() 0.481305 2885.993 ms
1297.057 ms 76
java.lang.System.identityHashCode[native]() 0.459557 1238.449 ms
1238.449 ms 1
java.lang.SecurityManager.checkConnect() 0.115316 310.762 ms
310.762 ms 3
java.lang.Long.toHexString() 0.110310 297.271 ms 297.271 ms 1
java.io.ObjectOutputStream.defaultWriteObject() 0.095964 38327.99 ms
258.61 ms 71
com.sun.jini.jeri.internal.runtime.Util.unmarshalValue() 0.091908
247.68 ms 247.68 ms 133
au.net.zeus.collection.AbstractReferenceComparator.compare() 0.079535
214.336 ms 214.336 ms 2
java.lang.reflect.Proxy.getInvocationHandler() 0.071852 193.631 ms
193.631 ms 1
com.sun.jini.jeri.internal.runtime.Util.marshalValue() 0.058676
1049.831 ms 158.124 ms 78
java.util.concurrent.AbstractExecutorService.submit() 0.040076 108.0
ms 108.0 ms 1
java.io.ObjectInputStream.<init>() 0.022718 61.223 ms 61.223 ms 1
com.sun.jini.start.ActivateWrapper$ExportClassLoader.toString()
0.022597 60.895 ms 60.895 ms 2
java.security.AccessController.doPrivileged[native]() 0.022204
59.837 ms 59.837 ms 340
java.lang.reflect.Method.invoke() 0.020569 31369.303 ms 55.431 ms 244
org.apache.river.api.net.RFC3986URLClassLoader$5.run() 0.015687
42.274 ms 42.274 ms 2
com.sun.jini.jeri.internal.runtime.Target.dispatch() 0 0.0 ms 0.0
ms 474
Re: JERI Scalability Testing
Posted by Bryan Thompson <br...@systap.com>.
+1
> On May 12, 2014, at 11:03 PM, Gregg Wonderly <gr...@wonderly.org> wrote:
>
> All of these things are the reason why we just need to take Peter’s hard work an go with it. There is really no reason to look back. The performance issues and security problems that Peter has so patiently waded through have needed attention for a long time. Only people trying to actually stress and use Jini to it’s full capacity will have any chance at seeing these things be big barriers, but that doesn’t discount their importance to even small applications.
>
> Gregg Wonderly
>
>> On May 12, 2014, at 6:17 PM, Peter <ji...@zeus.net.au> wrote:
>>
>> Hi Simon,
>>
>> http://svn.apache.org/viewvc/river/jtsk/skunk/qa_refactor/trunk/src/net/jini/loader/ClassLoading.java?view=markup
>>
>> During stress test profiling I found Class.forName to be heavily contended (the method with a ClassLoader parameter).
>>
>> Uncontended synchronization is very fast, so rather than employ parallel ClassLoading (which requires a lock for every class and isn't part of the jvm or lang specs), I decided to try thread confinement instead (for each ClassLoader). Before I found class loading to be a bottle neck, I had to fix a number of other throughput bottlenecks however:
>>
>> 1. Security Policy provider would cause contention while it performed dns calls. Dns calls are still made, but far fewer and concurrent security checks are now non blocking. The new policy provider takes advantage of immutability, how often do you change your policy files?
>> 2. SecureClassLoader uses CodeSource in a loader map as keys, causing multiple dns lookup calls, fixed that with a RFC3986 compliant URI class to replace URL based keys in maps.
>> 3. PreferredClassLoader also used URL's as keys in maps.
>> 4. Exising thread pools didn't take advantage of concurrent utilities and TaskManager performed very poorly when it's task queue became large, since dependant tasks had to synchronize and iterate over the whole queue to find preceeding tasks.
>>
>> All fixed now, just need to reduce network traffic as Sockets are now the bottleneck.
>>
>> After I finish with latent bugs, I'll investigate using a provider to plug in various serialization frameworks that are available now.
>>
>> That and codebase provisioning should make River really sing.
>>
>> The stress tests were devised by the original Jini team, they're probably the closest thing I've got to deployment scenario's.
>>
>> Regards,
>>
>> Peter.
>> ----- Original message -----
>>> Peter,
>>>
>>> With apologies that this it totally off-topic, but you made a comment
>>> here that I'd really like to understand better:
>>>
>>> * ClassLoading is thread confined for each classloader to avoid
>>>> contention.
>>>
>>>
>>> Without impinging on your time unduly, are you able to point me at
>>> something that would allow me to discover what this is about? It sounds
>>> interesting, but I have no idea how this would work.
>>>
>>> And, I quite understand if you remain silent :)
>>>
>>> Cheers,
>>> Simon
>
Re: JERI Scalability Testing
Posted by Gregg Wonderly <gr...@wonderly.org>.
All of these things are the reason why we just need to take Peter’s hard work an go with it. There is really no reason to look back. The performance issues and security problems that Peter has so patiently waded through have needed attention for a long time. Only people trying to actually stress and use Jini to it’s full capacity will have any chance at seeing these things be big barriers, but that doesn’t discount their importance to even small applications.
Gregg Wonderly
On May 12, 2014, at 6:17 PM, Peter <ji...@zeus.net.au> wrote:
> Hi Simon,
>
> http://svn.apache.org/viewvc/river/jtsk/skunk/qa_refactor/trunk/src/net/jini/loader/ClassLoading.java?view=markup
>
> During stress test profiling I found Class.forName to be heavily contended (the method with a ClassLoader parameter).
>
> Uncontended synchronization is very fast, so rather than employ parallel ClassLoading (which requires a lock for every class and isn't part of the jvm or lang specs), I decided to try thread confinement instead (for each ClassLoader). Before I found class loading to be a bottle neck, I had to fix a number of other throughput bottlenecks however:
>
> 1. Security Policy provider would cause contention while it performed dns calls. Dns calls are still made, but far fewer and concurrent security checks are now non blocking. The new policy provider takes advantage of immutability, how often do you change your policy files?
> 2. SecureClassLoader uses CodeSource in a loader map as keys, causing multiple dns lookup calls, fixed that with a RFC3986 compliant URI class to replace URL based keys in maps.
> 3. PreferredClassLoader also used URL's as keys in maps.
> 4. Exising thread pools didn't take advantage of concurrent utilities and TaskManager performed very poorly when it's task queue became large, since dependant tasks had to synchronize and iterate over the whole queue to find preceeding tasks.
>
> All fixed now, just need to reduce network traffic as Sockets are now the bottleneck.
>
> After I finish with latent bugs, I'll investigate using a provider to plug in various serialization frameworks that are available now.
>
> That and codebase provisioning should make River really sing.
>
> The stress tests were devised by the original Jini team, they're probably the closest thing I've got to deployment scenario's.
>
> Regards,
>
> Peter.
> ----- Original message -----
>> Peter,
>>
>> With apologies that this it totally off-topic, but you made a comment
>> here that I'd really like to understand better:
>>
>> * ClassLoading is thread confined for each classloader to avoid
>>> contention.
>>
>>
>> Without impinging on your time unduly, are you able to point me at
>> something that would allow me to discover what this is about? It sounds
>> interesting, but I have no idea how this would work.
>>
>> And, I quite understand if you remain silent :)
>>
>> Cheers,
>> Simon
>
Re: JERI Scalability Testing
Posted by Peter <ji...@zeus.net.au>.
Hi Simon,
http://svn.apache.org/viewvc/river/jtsk/skunk/qa_refactor/trunk/src/net/jini/loader/ClassLoading.java?view=markup
During stress test profiling I found Class.forName to be heavily contended (the method with a ClassLoader parameter).
Uncontended synchronization is very fast, so rather than employ parallel ClassLoading (which requires a lock for every class and isn't part of the jvm or lang specs), I decided to try thread confinement instead (for each ClassLoader). Before I found class loading to be a bottle neck, I had to fix a number of other throughput bottlenecks however:
1. Security Policy provider would cause contention while it performed dns calls. Dns calls are still made, but far fewer and concurrent security checks are now non blocking. The new policy provider takes advantage of immutability, how often do you change your policy files?
2. SecureClassLoader uses CodeSource in a loader map as keys, causing multiple dns lookup calls, fixed that with a RFC3986 compliant URI class to replace URL based keys in maps.
3. PreferredClassLoader also used URL's as keys in maps.
4. Exising thread pools didn't take advantage of concurrent utilities and TaskManager performed very poorly when it's task queue became large, since dependant tasks had to synchronize and iterate over the whole queue to find preceeding tasks.
All fixed now, just need to reduce network traffic as Sockets are now the bottleneck.
After I finish with latent bugs, I'll investigate using a provider to plug in various serialization frameworks that are available now.
That and codebase provisioning should make River really sing.
The stress tests were devised by the original Jini team, they're probably the closest thing I've got to deployment scenario's.
Regards,
Peter.
----- Original message -----
> Peter,
>
> With apologies that this it totally off-topic, but you made a comment
> here that I'd really like to understand better:
>
> * ClassLoading is thread confined for each classloader to avoid
> > contention.
>
>
> Without impinging on your time unduly, are you able to point me at
> something that would allow me to discover what this is about? It sounds
> interesting, but I have no idea how this would work.
>
> And, I quite understand if you remain silent :)
>
> Cheers,
> Simon
Re: JERI Scalability Testing
Posted by Simon Roberts <si...@dancingcloudservices.com>.
Peter,
With apologies that this it totally off-topic, but you made a comment here
that I'd really like to understand better:
* ClassLoading is thread confined for each classloader to avoid
> contention.
Without impinging on your time unduly, are you able to point me at
something that would allow me to discover what this is about? It sounds
interesting, but I have no idea how this would work.
And, I quite understand if you remain silent :)
Cheers,
Simon