You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Rainer Jung <to...@kippdata.de> on 2006/02/25 21:19:01 UTC
DTrace analysis of TC-Cluster Multicast
Hi,
I started to trace TC cluster with DTrace. I did a first simple exam for
the mcast subcomplex. CPU usage and especially elapsed times might be
slightly larger than expected, because I used the more intrusive
extended DTrace probes.
All tests were done with JDK 1.6.0 build 72 on Solaris 10 SPARC, using a
T2000 system (8 cores with 4 Threads each).
Method allocation elapsed cpu
mcast/McastServiceImpl.send() 1328B 40ms 6.7ms
mcast/McastServiceImpl.receive() 744-760B 23-510ms 3.5ms
So this does not look very bad, but I inspected, where cpu time is used
and objects get allocated:
1) send cpu time, elapsed time and allocation, per send
cpu elapsed alloc
6.7ms 40.3ms 1328B McastServiceImpl.send()
6.3ms 37.6ms 1296B McastMember.getData()
1.3ms 7.6ms 136B ... getName().getBytes();
0.9ms 5.3ms 120B ... getDomain().getBytes();
3.6ms 21.2ms 880B ... java.net.InetAddress.getByName(host)...
It's interesting, that the last three statements are responsible for 85%
of the allocation and 90% of the cpu usage. This seems unnecessary,
because for sending multicast heartbeat, name, domain and host do not
change, so they could be buffered easily.
2) receive cpu time, elapsed time and allocation, per receive
cpu elapsed alloc
3.5ms 23-510ms 744-760B McastServiceImpl.receive()
0.1ms 3-485ms 0B socket.receive()
2.8ms 17.9ms 640B McastMember.getMember()
1.4ms 8.1ms 160B new String(named),
1.0ms 5.6ms 128B new String(domaind),
0.1ms 0.6ms 112B addressToString(addr),
0.3ms 2.7ms 16B ... membership.memberAlive(m) [Only 50%]
0.2ms 2.3ms 32B ... membership.expire(timeToExpiration);
...
The calls to memberAlive are much faster in ~50% of the calls.
Here most interesting is the fact, that the string construction takes
70% of total cpu time. I inspected, why this is so. For the next trace I
had to increase the depth I traced, so elapsed times go a little up. The
columns are:
- ">" or "<": ">" means start of method, "<" end of method
- stack depth relative to the receive method
- cpu time
- elapsed time
- allocated bytes
- class and method
> 3 - - - java/lang/String.<init>
> 4 - - - java/lang/String.<init>
> 5 - - - java/lang/StringCoding.decode
> 6 - - - java/lang/StringCoding.decode
> 7 - - - java/lang/StringCoding.deref
< 7 0.1ms 0.7ms 0 java/lang/StringCoding.deref
> 7 - - - java/lang/StringCoding$StringDecoder.decode
> 8 - - - java/nio/ByteBuffer.wrap
< 8 0.1ms 0.5ms 48 java/nio/ByteBuffer.wrap
> 8 - - - java/nio/CharBuffer.wrap
< 8 0.1ms 0.5ms 48 java/nio/CharBuffer.wrap
> 8 - - - java/nio/charset/CharsetDecoder.decode
< 8 0.7ms 4.4ms 0 java/nio/charset/CharsetDecoder.decode
< 7 1.1ms 7.8ms 160 java/lang/StringCoding$StringDecoder.decode
< 6 1.2ms 9.1ms 160 java/lang/StringCoding.decode
< 5 1.2ms 9.5ms 160 java/lang/StringCoding.decode
< 4 1.3ms 10.1ms 160 java/lang/String.<init>
< 3 1.3ms 10.2ms 160 java/lang/String.<init>
So we can see, that most of the cpu time is needed inside
- java/nio/charset/CharsetDecoder.decode: 0.7ms
- java/lang/StringCoding.deref: 0.1ms
- java/nio/ByteBuffer.wrap: 0.1ms
- java/nio/CharBuffer.wrap: 0.1ms
Maybe it would help to use a more byte-oriented format in the heartbeat
messages and to not convert from it. String conversions might be
restrictable to the case when logging in debug mode.
Cheers,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: DTrace analysis of TC-Cluster Multicast
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
correctness can be achieved in many different ways, synchronization
just happens to be the easiest one.
The build is working, including session replication (modules
groupcom,ha,storeconfig-ha), just uncomment one line in
container/modules/build.properties and tomcat builds the new modules
instead of the old.
I have a lot of refactoring and optimization left to do.
Filip
Rainer Jung wrote:
> I agree, heartbeat is not that important w.r.t. performance. I used it
> as a simple test for how DTrace will be able to help. And at least in
> that nearly trivial test case the results are already interesting.
>
> Another thing that DTrace can check is lock contention. So there's
> more to come :)
>
> Do you have a working build of groupcom? I could trace some of the
> group communication.
>
> I hope you will not sacrifice synchronization correctness for
> performance ...
>
> Rainer
>
> Filip Hanik - Dev Lists wrote:
>> Cool Rainer, interesting numbers. Can you run the same numbers on the
>> new module groupcom.
>> I have done some optimizations there, for example, the name isn't
>> broadcasted. And I am about to remove the domain, instead of having a
>> default one. also, I agree, lets not translate the 4byte IP and the
>> 4byte port when receiving it. instead, member comparisons can be done
>> on the byte array level, excellent suggestion.
>>
>> After that I won't be spending to much focus on this issue, as a
>> heartbeat do not really have to be optimized. The heartbeat runs
>> *very* infrequently and it not a critical piece.
>>
>> Instead I am gonna start reworking the data senders, there are way
>> too many synchronized calls locking down the sockets, the code is
>> somewhat cluttered then I am gonna move on to serialization. In the
>> current cluster module, too much gets serialized, in the new module
>> "ha" I have already removed a lot of serialization and will continue
>> stripping it out.
>>
>> Filip
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: DTrace analysis of TC-Cluster Multicast
Posted by Rainer Jung <to...@kippdata.de>.
I agree, heartbeat is not that important w.r.t. performance. I used it
as a simple test for how DTrace will be able to help. And at least in
that nearly trivial test case the results are already interesting.
Another thing that DTrace can check is lock contention. So there's more
to come :)
Do you have a working build of groupcom? I could trace some of the group
communication.
I hope you will not sacrifice synchronization correctness for
performance ...
Rainer
Filip Hanik - Dev Lists wrote:
> Cool Rainer, interesting numbers. Can you run the same numbers on the
> new module groupcom.
> I have done some optimizations there, for example, the name isn't
> broadcasted. And I am about to remove the domain, instead of having a
> default one. also, I agree, lets not translate the 4byte IP and the
> 4byte port when receiving it. instead, member comparisons can be done on
> the byte array level, excellent suggestion.
>
> After that I won't be spending to much focus on this issue, as a
> heartbeat do not really have to be optimized. The heartbeat runs *very*
> infrequently and it not a critical piece.
>
> Instead I am gonna start reworking the data senders, there are way too
> many synchronized calls locking down the sockets, the code is somewhat
> cluttered then I am gonna move on to serialization. In the current
> cluster module, too much gets serialized, in the new module "ha" I have
> already removed a lot of serialization and will continue stripping it out.
>
> Filip
>
> Rainer Jung wrote:
>> Hi,
>>
>> I started to trace TC cluster with DTrace. I did a first simple exam for
>> the mcast subcomplex. CPU usage and especially elapsed times might be
>> slightly larger than expected, because I used the more intrusive
>> extended DTrace probes.
>>
>> All tests were done with JDK 1.6.0 build 72 on Solaris 10 SPARC, using a
>> T2000 system (8 cores with 4 Threads each).
>>
>> Method allocation elapsed cpu
>> mcast/McastServiceImpl.send() 1328B 40ms 6.7ms
>> mcast/McastServiceImpl.receive() 744-760B 23-510ms 3.5ms
>>
>> So this does not look very bad, but I inspected, where cpu time is used
>> and objects get allocated:
>>
>> 1) send cpu time, elapsed time and allocation, per send
>>
>> cpu elapsed alloc
>> 6.7ms 40.3ms 1328B McastServiceImpl.send()
>>
>> 6.3ms 37.6ms 1296B McastMember.getData()
>>
>> 1.3ms 7.6ms 136B ... getName().getBytes();
>> 0.9ms 5.3ms 120B ... getDomain().getBytes();
>> 3.6ms 21.2ms 880B ... java.net.InetAddress.getByName(host)...
>>
>> It's interesting, that the last three statements are responsible for 85%
>> of the allocation and 90% of the cpu usage. This seems unnecessary,
>> because for sending multicast heartbeat, name, domain and host do not
>> change, so they could be buffered easily.
>>
>> 2) receive cpu time, elapsed time and allocation, per receive
>>
>> cpu elapsed alloc
>> 3.5ms 23-510ms 744-760B McastServiceImpl.receive()
>>
>> 0.1ms 3-485ms 0B socket.receive()
>> 2.8ms 17.9ms 640B McastMember.getMember()
>>
>> 1.4ms 8.1ms 160B new String(named),
>> 1.0ms 5.6ms 128B new String(domaind),
>> 0.1ms 0.6ms 112B addressToString(addr),
>>
>> 0.3ms 2.7ms 16B ... membership.memberAlive(m) [Only 50%]
>> 0.2ms 2.3ms 32B ... membership.expire(timeToExpiration);
>> ...
>>
>> The calls to memberAlive are much faster in ~50% of the calls.
>>
>> Here most interesting is the fact, that the string construction takes
>> 70% of total cpu time. I inspected, why this is so. For the next trace I
>> had to increase the depth I traced, so elapsed times go a little up. The
>> columns are:
>>
>> - ">" or "<": ">" means start of method, "<" end of method
>> - stack depth relative to the receive method
>> - cpu time
>> - elapsed time
>> - allocated bytes
>> - class and method
>>
>>> 3 - - - java/lang/String.<init>
>>> 4 - - - java/lang/String.<init>
>>> 5 - - - java/lang/StringCoding.decode
>>> 6 - - - java/lang/StringCoding.decode
>>> 7 - - - java/lang/StringCoding.deref
>> < 7 0.1ms 0.7ms 0 java/lang/StringCoding.deref
>>> 7 - - - java/lang/StringCoding$StringDecoder.decode
>>> 8 - - - java/nio/ByteBuffer.wrap
>> < 8 0.1ms 0.5ms 48 java/nio/ByteBuffer.wrap
>>> 8 - - - java/nio/CharBuffer.wrap
>> < 8 0.1ms 0.5ms 48 java/nio/CharBuffer.wrap
>>> 8 - - - java/nio/charset/CharsetDecoder.decode
>> < 8 0.7ms 4.4ms 0 java/nio/charset/CharsetDecoder.decode
>> < 7 1.1ms 7.8ms 160 java/lang/StringCoding$StringDecoder.decode
>> < 6 1.2ms 9.1ms 160 java/lang/StringCoding.decode
>> < 5 1.2ms 9.5ms 160 java/lang/StringCoding.decode
>> < 4 1.3ms 10.1ms 160 java/lang/String.<init>
>> < 3 1.3ms 10.2ms 160 java/lang/String.<init>
>>
>> So we can see, that most of the cpu time is needed inside
>>
>> - java/nio/charset/CharsetDecoder.decode: 0.7ms
>> - java/lang/StringCoding.deref: 0.1ms
>> - java/nio/ByteBuffer.wrap: 0.1ms
>> - java/nio/CharBuffer.wrap: 0.1ms
>>
>> Maybe it would help to use a more byte-oriented format in the heartbeat
>> messages and to not convert from it. String conversions might be
>> restrictable to the case when logging in debug mode.
>>
>> Cheers,
>>
>> Rainer
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: DTrace analysis of TC-Cluster Multicast
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
Cool Rainer, interesting numbers. Can you run the same numbers on the
new module groupcom.
I have done some optimizations there, for example, the name isn't
broadcasted. And I am about to remove the domain, instead of having a
default one. also, I agree, lets not translate the 4byte IP and the
4byte port when receiving it. instead, member comparisons can be done on
the byte array level, excellent suggestion.
After that I won't be spending to much focus on this issue, as a
heartbeat do not really have to be optimized. The heartbeat runs *very*
infrequently and it not a critical piece.
Instead I am gonna start reworking the data senders, there are way too
many synchronized calls locking down the sockets, the code is somewhat
cluttered then I am gonna move on to serialization. In the current
cluster module, too much gets serialized, in the new module "ha" I have
already removed a lot of serialization and will continue stripping it out.
Filip
Rainer Jung wrote:
> Hi,
>
> I started to trace TC cluster with DTrace. I did a first simple exam for
> the mcast subcomplex. CPU usage and especially elapsed times might be
> slightly larger than expected, because I used the more intrusive
> extended DTrace probes.
>
> All tests were done with JDK 1.6.0 build 72 on Solaris 10 SPARC, using a
> T2000 system (8 cores with 4 Threads each).
>
> Method allocation elapsed cpu
> mcast/McastServiceImpl.send() 1328B 40ms 6.7ms
> mcast/McastServiceImpl.receive() 744-760B 23-510ms 3.5ms
>
> So this does not look very bad, but I inspected, where cpu time is used
> and objects get allocated:
>
> 1) send cpu time, elapsed time and allocation, per send
>
> cpu elapsed alloc
> 6.7ms 40.3ms 1328B McastServiceImpl.send()
>
> 6.3ms 37.6ms 1296B McastMember.getData()
>
> 1.3ms 7.6ms 136B ... getName().getBytes();
> 0.9ms 5.3ms 120B ... getDomain().getBytes();
> 3.6ms 21.2ms 880B ... java.net.InetAddress.getByName(host)...
>
> It's interesting, that the last three statements are responsible for 85%
> of the allocation and 90% of the cpu usage. This seems unnecessary,
> because for sending multicast heartbeat, name, domain and host do not
> change, so they could be buffered easily.
>
> 2) receive cpu time, elapsed time and allocation, per receive
>
> cpu elapsed alloc
> 3.5ms 23-510ms 744-760B McastServiceImpl.receive()
>
> 0.1ms 3-485ms 0B socket.receive()
> 2.8ms 17.9ms 640B McastMember.getMember()
>
> 1.4ms 8.1ms 160B new String(named),
> 1.0ms 5.6ms 128B new String(domaind),
> 0.1ms 0.6ms 112B addressToString(addr),
>
> 0.3ms 2.7ms 16B ... membership.memberAlive(m) [Only 50%]
> 0.2ms 2.3ms 32B ... membership.expire(timeToExpiration);
> ...
>
> The calls to memberAlive are much faster in ~50% of the calls.
>
> Here most interesting is the fact, that the string construction takes
> 70% of total cpu time. I inspected, why this is so. For the next trace I
> had to increase the depth I traced, so elapsed times go a little up. The
> columns are:
>
> - ">" or "<": ">" means start of method, "<" end of method
> - stack depth relative to the receive method
> - cpu time
> - elapsed time
> - allocated bytes
> - class and method
>
>> 3 - - - java/lang/String.<init>
>> 4 - - - java/lang/String.<init>
>> 5 - - - java/lang/StringCoding.decode
>> 6 - - - java/lang/StringCoding.decode
>> 7 - - - java/lang/StringCoding.deref
> < 7 0.1ms 0.7ms 0 java/lang/StringCoding.deref
>> 7 - - - java/lang/StringCoding$StringDecoder.decode
>> 8 - - - java/nio/ByteBuffer.wrap
> < 8 0.1ms 0.5ms 48 java/nio/ByteBuffer.wrap
>> 8 - - - java/nio/CharBuffer.wrap
> < 8 0.1ms 0.5ms 48 java/nio/CharBuffer.wrap
>> 8 - - - java/nio/charset/CharsetDecoder.decode
> < 8 0.7ms 4.4ms 0 java/nio/charset/CharsetDecoder.decode
> < 7 1.1ms 7.8ms 160 java/lang/StringCoding$StringDecoder.decode
> < 6 1.2ms 9.1ms 160 java/lang/StringCoding.decode
> < 5 1.2ms 9.5ms 160 java/lang/StringCoding.decode
> < 4 1.3ms 10.1ms 160 java/lang/String.<init>
> < 3 1.3ms 10.2ms 160 java/lang/String.<init>
>
> So we can see, that most of the cpu time is needed inside
>
> - java/nio/charset/CharsetDecoder.decode: 0.7ms
> - java/lang/StringCoding.deref: 0.1ms
> - java/nio/ByteBuffer.wrap: 0.1ms
> - java/nio/CharBuffer.wrap: 0.1ms
>
> Maybe it would help to use a more byte-oriented format in the heartbeat
> messages and to not convert from it. String conversions might be
> restrictable to the case when logging in debug mode.
>
> Cheers,
>
> Rainer
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org