You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Glenn Nielsen <gl...@mail.more.net> on 2003/12/10 04:33:07 UTC
[TBM] Thread Benchmark example test results
I have posted some example Thread Benchmark test results at:
http://www.apache.org/~glenn/
TBM still needs alot of work, there are only 6 benchmark tests so far.
Three of them are for formatting a date, the results are interesting.
Regards,
Glenn
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [TBM] Thread Benchmark example test results
Posted by Glenn Nielsen <gl...@mail.more.net>.
Brian S O'Neill wrote:
> The BenchTest has a few flaws. I wrote a few new tests and was shocked
> by the numbers. Running with two threads, each with half the iterations
> was over one hundred times slower than the single thread test. This
> makes no sense.
>
Yes, this is a work in progress. Thanks for the feedback.
> The main problem is the warmup thread. After you call join on the
> thread, it is useless. I would expect calling start again to throw an
> exception, but instead nothing seems to happen. Either way, thread 0 is
> always dead. This throws off the results for the single thread test, and
> the normalized results are way off too.
Not doing a warmup could also impact the test results. It would increase
the time it takes for the single thread test to complete due to HotSpot
optimizing the code.
I modified the code to instantiate a seperate object for doing the warmup.
When I ran the tests it verified your conclusion that thread[0] was not being
executed. Wierd, that is not what I would have expected. Do you know why
this is the case?
>
> The second problem is including garbage collection times as part of the
> test. I don't recommend forcing a call to Runtime.gc(). If you're trying
> to determine how much time is spent collecting garbage, run with
> verbose:gc. If you want to see how many objects are allocated, run with
> a profiler.
If you run these tests with a profiler that could also impact the results.
The instrumentation the profiler uses will affect the benchmark results.
Yes, trying to get data on the impact of GC is problematic, I mention
that in the docs. There are many other things which can impact the
performance such as using different JVM args like -server, heap size,
gc options. Getting accurate data on the impact of GC may require
capturing the output from -verbose:gc and post processing it.
I still think it is a good idea to try and include in the benchmark
tests the impact of GC on performance.
>
> When I commented out the warmup section and all the gc() related calls,
> the results were more like what I would expect. Doubling the threads and
> halving the iterations resulted in performance which was the same or
> only slightly worse. After these changes, here are my results:
>
GC does have a significant impact on performance. Not performing the
Full GC prior to starting the test could impact the test results.
For example, you execute a series of benchmark tests, during the
execution of the single thread for a benchmark a Full GC is triggered.
That would significantly impact the time taken to execute the
benchmark with a single thread.
Developing accurate benchmarks is challenging, especially when you
are trying to benchmark java code running within a JVM.
Regards,
Glenn
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [TBM] Thread Benchmark example test results
Posted by Brian S O'Neill <br...@earthlink.net>.
The BenchTest has a few flaws. I wrote a few new tests and was shocked
by the numbers. Running with two threads, each with half the iterations
was over one hundred times slower than the single thread test. This
makes no sense.
The main problem is the warmup thread. After you call join on the
thread, it is useless. I would expect calling start again to throw an
exception, but instead nothing seems to happen. Either way, thread 0 is
always dead. This throws off the results for the single thread test, and
the normalized results are way off too.
The second problem is including garbage collection times as part of the
test. I don't recommend forcing a call to Runtime.gc(). If you're trying
to determine how much time is spent collecting garbage, run with
verbose:gc. If you want to see how many objects are allocated, run with
a profiler.
When I commented out the warmup section and all the gc() related calls,
the results were more like what I would expect. Doubling the threads and
halving the iterations resulted in performance which was the same or
only slightly worse. After these changes, here are my results:
JVM: Sun Microsystems Inc. 1.4.2
OS: Windows 2000 5.0 x86
Test Name: Baseline No Operation
Test Class: org.apache.commons.tbm.bench.BenchNOP
Threads Memory Used GC Time (ms) Total Time (ms) Normalized
1 0 0 0 1.00
2 0 0 0 ?
4 0 0 16 ?
8 0 0 15 ?
16 0 0 0 ?
32 0 0 78 ?
64 0 0 79 ?
Test Name: TimeZone.getDefault()
Test Class: org.apache.commons.tbm.bench.datetime.DefaultTimeZone
Threads Memory Used GC Time (ms) Total Time (ms) Normalized
1 0 0 110 1.00
2 0 0 171 1.55
4 0 0 391 3.55
8 0 0 547 4.97
16 0 0 593 5.39
32 0 0 782 7.11
64 0 0 171 1.55
Test Name: Construct SimpleDateFormat() and format new Date()
Test Class: org.apache.commons.tbm.bench.datetime.SimpleDateFormat
Threads Memory Used GC Time (ms) Total Time (ms) Normalized
1 0 0 6188 1.00
2 0 0 6031 0.97
4 0 0 6391 1.03
8 0 0 8031 1.30
16 0 0 9578 1.55
32 0 0 9750 1.58
64 0 0 12125 1.96
Test Name: Jakarta Commons-Lang FastDateFormat
Test Class: org.apache.commons.tbm.bench.datetime.CommonsLangFastDateFormat
Threads Memory Used GC Time (ms) Total Time (ms) Normalized
1 0 0 1563 1.00
2 0 0 1531 0.98
4 0 0 1734 1.11
8 0 0 2516 1.61
16 0 0 2468 1.58
32 0 0 3110 1.99
64 0 0 3437 2.20
Test Name: ThreadLocal SimpleDateFormat() and format new Date()
Test Class:
org.apache.commons.tbm.bench.datetime.ThreadLocalSimpleDateFormat
Threads Memory Used GC Time (ms) Total Time (ms) Normalized
1 0 0 719 1.00
2 0 0 719 1.00
4 0 0 718 1.00
8 0 0 735 1.02
16 0 0 750 1.04
32 0 0 781 1.09
64 0 0 813 1.13
Test Name: java.sql.Timestamp.toString()
Test Class: org.apache.commons.tbm.bench.datetime.TimestampToString
Threads Memory Used GC Time (ms) Total Time (ms) Normalized
1 0 0 3859 1.00
2 0 0 4344 1.13
4 0 0 7734 2.00
8 0 0 8281 2.15
16 0 0 8672 2.25
32 0 0 10906 2.83
64 0 0 12890 3.34
Glenn Nielsen wrote:
> I have posted some example Thread Benchmark test results at:
>
> http://www.apache.org/~glenn/
>
>
> TBM still needs alot of work, there are only 6 benchmark tests so far.
> Three of them are for formatting a date, the results are interesting.
>
> Regards,
>
> Glenn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org