You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openjpa.apache.org by "David Ezzio (asmtp)" <de...@bea.com> on 2007/06/01 15:25:26 UTC
Re: Comparison Metrics for OpenJPA's ConcurrentHashMap
Hi Marc,
I still was not able to get HighScaleLib to work. It produces a
SecurityException when attempting to get the Unsafe object. I decided to
avoid changing the relevant security policy.
On the other hand, I did test Emory University's backport of the
java.util.concurrent package. This provides to Java 1.4 a
ConcurrentHashMap implementation that is compatible with the one found
in Java 5 and 6.
I also realized that I could get another metric from my numbers, the
percentage of time the threads were suspended during the metered
operations. Then I reran the tests for Emory's backport using fewer
threads. In the classic pattern of an overloaded CPU, the higher thread
count both lowers throughput and increases response time. (Throughput
is yet another number I haven't extracted from the data, although it is
obvious when running the tests.)
All of the previous tests were run with 5 writing and 10 reading
threads. Only backport was additionally tested with 2 writing and 4
reading threads. The suspended percentage is approximate since the
adding and updating tests have slightly different numbers.
Implementation Add Remove Update Find-a Find-r Find-u Suspended
--------------+------+-------+-------+-------+-------+-------+---------
synchronized 103 35 50 40 37 54
concurrent 13.2 6.4 6.1 0.6 0.3 1.1
OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
Backport (5/10) 43.2 36.8 40.4 0.3 0.2 0.5 62%
Backport (2/4) 6.1 3.1 3.3 0.3 0.2 0.3 4%
David
David Ezzio (asmtp) wrote:
> Hi Marc,
>
> I did plug it in, but it failed straightaway on a security issue. I
> should probably read its documentation. :) I'll try it again along with
> the backport lib done by Emory U.
>
> David
>
>
> Marc Prud'hommeaux wrote:
>> David-
>>
>> That is very interesting.
>>
>> Did you also take a look at the one at
>> http://sourceforge.net/projects/high-scale-lib ? They say its
>> performance only shines for high thread/cpu counts, but it might be
>> interesting to see where its numbers lie in the range.
>>
>>
>>
>> On May 29, 2007, at 11:01 AM, David Ezzio (asmtp) wrote:
>>
>>> Recently, I did some testing of Map implementations under concurrency.
>>>
>>> My primary purpose was to verify the reliability of OpenJPA's
>>> ConcurrentHashMap implementation. As I got into it, I saw the
>>> opportunity to get some performance metrics out of the test.
>>>
>>> The biggest part of my task was coming up with a reliable and useful
>>> testing framework. I design it with the following two factors in
>>> mind: First, I wanted to test the edge conditions where an entry had
>>> just been added or removed or where a key's value had just been
>>> updated. The idea is that a number of threads add, remove, and update
>>> entries, while other threads check to see if these recent
>>> modifications are visible (or in the case of removals, not visible).
>>> Second, I wanted the testing framework itself to be free of
>>> synchronization. If the testing framework used synchronization then
>>> it would tend to serialize the readers and writers and thereby mask
>>> concurrency issues in the map implementation under test.
>>>
>>> The testing framework uses a non-synchronizing, non-blocking FIFO
>>> queue as the mechanism for the writing threads to communicate their
>>> recent modifications to the reading threads.
>>>
>>> To prevent writing threads from overwriting recent modifications
>>> before they could be read and verified, the testing framework walks
>>> the hash map keys in in a linear (or in the case of updates,
>>> circular) order. By using a hash map with a large enough capacity,
>>> readers have the time to verify the recent modifications before the
>>> writer threads come back to modify that part of the key space again.
>>>
>>> Using an adapter for the map implementation, the testing framework
>>> starts five writer threads and ten reader threads at the same time.
>>> These threads run wide open for 30 seconds, except that the readers
>>> will give up their time slice if they find nothing on the queue. The
>>> HashMaps were all sized for the needed capacity upon creation, so no
>>> resizing occurred during testing.
>>>
>>> I got some interesting results.
>>>
>>> Four implementations were tested, Java's unsynchronized HashMap
>>> implementation, Java's synchronized HashMap implementation, Java's
>>> ConcurrentHashMap implementation, and OpenJPA's ConcurrentHashMap
>>> implementation.
>>>
>>> Only Java's unsynchronized HashMap failed, as expected, under test.
>>> Under test, this implementation demonstrates its inability to handle
>>> concurrency. The other three implementations worked flawlessly under
>>> test.
>>>
>>> The java.util.concurrent.ConcurrentHashMap implementation (available
>>> with Java 5 and 6) was the fastest implementation tested.
>>>
>>> Java's synchronized wrapper for the HashMap implementation is one to
>>> two orders of magnitude slower than Java's ConcurrentHashMap
>>> implementation.
>>>
>>> OpenJPA's ConcurrentHashMap compares equally with Java's
>>> ConcurrentHashMap in find operations and is 2-4 times slower in
>>> mutating operations.
>>>
>>> Implementation Add Remove Update Find-a Find-r Find-u
>>> ---------------+------+-------+--------+-------+-------+------
>>> synchronized 103 35 50 40 37 54
>>> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
>>> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
>>>
>>>
>>> Legend:
>>>
>>> synchronized:
>>> java.util.Collections.synchronizedMap(new java.util.HashMap())
>>> concurrent: java.util.concurrent.ConcurrentHashMap
>>> OpenJPA: org.apache.openjpa.lib.util.concurrent.ConcurrentHashMap
>>>
>>> Add: time for average add operation
>>> Remove: time for average remove operation
>>> Update: time for average update of new value for existing key
>>> Find-a: time to find a recent addition
>>> Find-r: time to NOT find a recent removal
>>> Find-u: time to find a recent update
>>>
>>> These times (in microseconds) are representative, but are not the
>>> average of several runs. The tests were run on a Dell Dual Core
>>> laptop under Windows. The performance meter was pegged during the tests.
>>>
>>> David Ezzio
>>>
>>>
>>
>>
>
>
Re: Comparison Metrics for OpenJPA's ConcurrentHashMap
Posted by Patrick Linskey <pl...@gmail.com>.
Incidentally, I'm working on code that allows for toggling of
concurrent map providers in different contexts. We could probably
adapt this to be more general-purpose. However, currently, this is not
something that can happen via our current configuration mechanisms,
since some of the key maps in question are constructed before any
configuration bootstrapping happens.
-Patrick
On 6/4/07, Dain Sundstrom <da...@iq80.com> wrote:
> I think this thread shows that there are some critical maps at the
> core of JPA, and it shows that the optimal map for an environment is
> debatable. Is this something we could make configurable? I'm
> personally very very skeptical of micro-benchmarks, and won't be
> convinced which is best until I see a full application benchmark
> (that is representative of my application).
>
> -dain
>
> On Jun 4, 2007, at 5:34 AM, David Ezzio wrote:
>
> > Two
> >
> > Patrick Linskey wrote:
> >> Hi,
> >>
> >> How many cores / CPUs were you using in these tests?
> >>
> >> -Patrick
> >>
> >> On 6/1/07, David Ezzio (asmtp) <de...@bea.com> wrote:
> >>> Hi Marc,
> >>>
> >>> I still was not able to get HighScaleLib to work. It produces a
> >>> SecurityException when attempting to get the Unsafe object. I
> >>> decided to
> >>> avoid changing the relevant security policy.
> >>>
> >>> On the other hand, I did test Emory University's backport of the
> >>> java.util.concurrent package. This provides to Java 1.4 a
> >>> ConcurrentHashMap implementation that is compatible with the one
> >>> found
> >>> in Java 5 and 6.
> >>>
> >>> I also realized that I could get another metric from my numbers, the
> >>> percentage of time the threads were suspended during the metered
> >>> operations. Then I reran the tests for Emory's backport using fewer
> >>> threads. In the classic pattern of an overloaded CPU, the higher
> >>> thread
> >>> count both lowers throughput and increases response time.
> >>> (Throughput
> >>> is yet another number I haven't extracted from the data, although
> >>> it is
> >>> obvious when running the tests.)
> >>>
> >>> All of the previous tests were run with 5 writing and 10 reading
> >>> threads. Only backport was additionally tested with 2 writing and 4
> >>> reading threads. The suspended percentage is approximate since the
> >>> adding and updating tests have slightly different numbers.
> >>>
> >>> Implementation Add Remove Update Find-a Find-r Find-u
> >>> Suspended
> >>> --------------+------+-------+-------+-------+-------+-------
> >>> +---------
> >>> synchronized 103 35 50 40 37 54
> >>> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
> >>> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
> >>> Backport (5/10) 43.2 36.8 40.4 0.3 0.2 0.5 62%
> >>> Backport (2/4) 6.1 3.1 3.3 0.3 0.2 0.3 4%
> >>>
> >>> David
> >>>
> >>>
> >>> David Ezzio (asmtp) wrote:
> >>>> Hi Marc,
> >>>>
> >>>> I did plug it in, but it failed straightaway on a security
> >>>> issue. I
> >>>> should probably read its documentation. :) I'll try it again along
> >>> with
> >>>> the backport lib done by Emory U.
> >>>>
> >>>> David
> >>>>
> >>>>
> >>>> Marc Prud'hommeaux wrote:
> >>>>> David-
> >>>>>
> >>>>> That is very interesting.
> >>>>>
> >>>>> Did you also take a look at the one at
> >>>>> http://sourceforge.net/projects/high-scale-lib ? They say its
> >>>>> performance only shines for high thread/cpu counts, but it
> >>>>> might be
> >>>>> interesting to see where its numbers lie in the range.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On May 29, 2007, at 11:01 AM, David Ezzio (asmtp) wrote:
> >>>>>
> >>>>>> Recently, I did some testing of Map implementations under
> >>> concurrency.
> >>>>>>
> >>>>>> My primary purpose was to verify the reliability of OpenJPA's
> >>>>>> ConcurrentHashMap implementation. As I got into it, I saw the
> >>>>>> opportunity to get some performance metrics out of the test.
> >>>>>>
> >>>>>> The biggest part of my task was coming up with a reliable and
> >>>>>> useful
> >>>>>> testing framework. I design it with the following two factors in
> >>>>>> mind: First, I wanted to test the edge conditions where an
> >>>>>> entry had
> >>>>>> just been added or removed or where a key's value had just been
> >>>>>> updated. The idea is that a number of threads add, remove, and
> >>>>>> update
> >>>>>> entries, while other threads check to see if these recent
> >>>>>> modifications are visible (or in the case of removals, not
> >>>>>> visible).
> >>>>>> Second, I wanted the testing framework itself to be free of
> >>>>>> synchronization. If the testing framework used synchronization
> >>>>>> then
> >>>>>> it would tend to serialize the readers and writers and thereby
> >>>>>> mask
> >>>>>> concurrency issues in the map implementation under test.
> >>>>>>
> >>>>>> The testing framework uses a non-synchronizing, non-blocking FIFO
> >>>>>> queue as the mechanism for the writing threads to communicate
> >>>>>> their
> >>>>>> recent modifications to the reading threads.
> >>>>>>
> >>>>>> To prevent writing threads from overwriting recent modifications
> >>>>>> before they could be read and verified, the testing framework
> >>>>>> walks
> >>>>>> the hash map keys in in a linear (or in the case of updates,
> >>>>>> circular) order. By using a hash map with a large enough
> >>>>>> capacity,
> >>>>>> readers have the time to verify the recent modifications
> >>>>>> before the
> >>>>>> writer threads come back to modify that part of the key space
> >>>>>> again.
> >>>>>>
> >>>>>> Using an adapter for the map implementation, the testing
> >>>>>> framework
> >>>>>> starts five writer threads and ten reader threads at the same
> >>>>>> time.
> >>>>>> These threads run wide open for 30 seconds, except that the
> >>>>>> readers
> >>>>>> will give up their time slice if they find nothing on the
> >>>>>> queue. The
> >>>>>> HashMaps were all sized for the needed capacity upon creation,
> >>>>>> so no
> >>>>>> resizing occurred during testing.
> >>>>>>
> >>>>>> I got some interesting results.
> >>>>>>
> >>>>>> Four implementations were tested, Java's unsynchronized HashMap
> >>>>>> implementation, Java's synchronized HashMap implementation,
> >>>>>> Java's
> >>>>>> ConcurrentHashMap implementation, and OpenJPA's ConcurrentHashMap
> >>>>>> implementation.
> >>>>>>
> >>>>>> Only Java's unsynchronized HashMap failed, as expected, under
> >>>>>> test.
> >>>>>> Under test, this implementation demonstrates its inability to
> >>>>>> handle
> >>>>>> concurrency. The other three implementations worked flawlessly
> >>>>>> under
> >>>>>> test.
> >>>>>>
> >>>>>> The java.util.concurrent.ConcurrentHashMap implementation
> >>>>>> (available
> >>>>>> with Java 5 and 6) was the fastest implementation tested.
> >>>>>>
> >>>>>> Java's synchronized wrapper for the HashMap implementation is
> >>>>>> one to
> >>>>>> two orders of magnitude slower than Java's ConcurrentHashMap
> >>>>>> implementation.
> >>>>>>
> >>>>>> OpenJPA's ConcurrentHashMap compares equally with Java's
> >>>>>> ConcurrentHashMap in find operations and is 2-4 times slower in
> >>>>>> mutating operations.
> >>>>>>
> >>>>>> Implementation Add Remove Update Find-a Find-r Find-u
> >>>>>> ---------------+------+-------+--------+-------+-------+------
> >>>>>> synchronized 103 35 50 40 37 54
> >>>>>> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
> >>>>>> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
> >>>>>>
> >>>>>>
> >>>>>> Legend:
> >>>>>>
> >>>>>> synchronized:
> >>>>>> java.util.Collections.synchronizedMap(new java.util.HashMap())
> >>>>>> concurrent: java.util.concurrent.ConcurrentHashMap
> >>>>>> OpenJPA: org.apache.openjpa.lib.util.concurrent.ConcurrentHashMap
> >>>>>>
> >>>>>> Add: time for average add operation
> >>>>>> Remove: time for average remove operation
> >>>>>> Update: time for average update of new value for existing key
> >>>>>> Find-a: time to find a recent addition
> >>>>>> Find-r: time to NOT find a recent removal
> >>>>>> Find-u: time to find a recent update
> >>>>>>
> >>>>>> These times (in microseconds) are representative, but are not the
> >>>>>> average of several runs. The tests were run on a Dell Dual Core
> >>>>>> laptop under Windows. The performance meter was pegged during the
> >>> tests.
> >>>>>>
> >>>>>> David Ezzio
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
> >
> > Notice: This email message, together with any attachments, may
> > contain information of BEA Systems, Inc., its subsidiaries
> > and affiliated entities, that may be confidential, proprietary,
> > copyrighted and/or legally privileged, and is intended solely for
> > the use of the individual or entity named in this message. If you
> > are not the intended recipient, and have received this message in
> > error, please immediately return this by email and then delete it.
>
>
--
Patrick Linskey
202 669 5907
Re: Comparison Metrics for OpenJPA's ConcurrentHashMap
Posted by Dain Sundstrom <da...@iq80.com>.
I think this thread shows that there are some critical maps at the
core of JPA, and it shows that the optimal map for an environment is
debatable. Is this something we could make configurable? I'm
personally very very skeptical of micro-benchmarks, and won't be
convinced which is best until I see a full application benchmark
(that is representative of my application).
-dain
On Jun 4, 2007, at 5:34 AM, David Ezzio wrote:
> Two
>
> Patrick Linskey wrote:
>> Hi,
>>
>> How many cores / CPUs were you using in these tests?
>>
>> -Patrick
>>
>> On 6/1/07, David Ezzio (asmtp) <de...@bea.com> wrote:
>>> Hi Marc,
>>>
>>> I still was not able to get HighScaleLib to work. It produces a
>>> SecurityException when attempting to get the Unsafe object. I
>>> decided to
>>> avoid changing the relevant security policy.
>>>
>>> On the other hand, I did test Emory University's backport of the
>>> java.util.concurrent package. This provides to Java 1.4 a
>>> ConcurrentHashMap implementation that is compatible with the one
>>> found
>>> in Java 5 and 6.
>>>
>>> I also realized that I could get another metric from my numbers, the
>>> percentage of time the threads were suspended during the metered
>>> operations. Then I reran the tests for Emory's backport using fewer
>>> threads. In the classic pattern of an overloaded CPU, the higher
>>> thread
>>> count both lowers throughput and increases response time.
>>> (Throughput
>>> is yet another number I haven't extracted from the data, although
>>> it is
>>> obvious when running the tests.)
>>>
>>> All of the previous tests were run with 5 writing and 10 reading
>>> threads. Only backport was additionally tested with 2 writing and 4
>>> reading threads. The suspended percentage is approximate since the
>>> adding and updating tests have slightly different numbers.
>>>
>>> Implementation Add Remove Update Find-a Find-r Find-u
>>> Suspended
>>> --------------+------+-------+-------+-------+-------+-------
>>> +---------
>>> synchronized 103 35 50 40 37 54
>>> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
>>> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
>>> Backport (5/10) 43.2 36.8 40.4 0.3 0.2 0.5 62%
>>> Backport (2/4) 6.1 3.1 3.3 0.3 0.2 0.3 4%
>>>
>>> David
>>>
>>>
>>> David Ezzio (asmtp) wrote:
>>>> Hi Marc,
>>>>
>>>> I did plug it in, but it failed straightaway on a security
>>>> issue. I
>>>> should probably read its documentation. :) I'll try it again along
>>> with
>>>> the backport lib done by Emory U.
>>>>
>>>> David
>>>>
>>>>
>>>> Marc Prud'hommeaux wrote:
>>>>> David-
>>>>>
>>>>> That is very interesting.
>>>>>
>>>>> Did you also take a look at the one at
>>>>> http://sourceforge.net/projects/high-scale-lib ? They say its
>>>>> performance only shines for high thread/cpu counts, but it
>>>>> might be
>>>>> interesting to see where its numbers lie in the range.
>>>>>
>>>>>
>>>>>
>>>>> On May 29, 2007, at 11:01 AM, David Ezzio (asmtp) wrote:
>>>>>
>>>>>> Recently, I did some testing of Map implementations under
>>> concurrency.
>>>>>>
>>>>>> My primary purpose was to verify the reliability of OpenJPA's
>>>>>> ConcurrentHashMap implementation. As I got into it, I saw the
>>>>>> opportunity to get some performance metrics out of the test.
>>>>>>
>>>>>> The biggest part of my task was coming up with a reliable and
>>>>>> useful
>>>>>> testing framework. I design it with the following two factors in
>>>>>> mind: First, I wanted to test the edge conditions where an
>>>>>> entry had
>>>>>> just been added or removed or where a key's value had just been
>>>>>> updated. The idea is that a number of threads add, remove, and
>>>>>> update
>>>>>> entries, while other threads check to see if these recent
>>>>>> modifications are visible (or in the case of removals, not
>>>>>> visible).
>>>>>> Second, I wanted the testing framework itself to be free of
>>>>>> synchronization. If the testing framework used synchronization
>>>>>> then
>>>>>> it would tend to serialize the readers and writers and thereby
>>>>>> mask
>>>>>> concurrency issues in the map implementation under test.
>>>>>>
>>>>>> The testing framework uses a non-synchronizing, non-blocking FIFO
>>>>>> queue as the mechanism for the writing threads to communicate
>>>>>> their
>>>>>> recent modifications to the reading threads.
>>>>>>
>>>>>> To prevent writing threads from overwriting recent modifications
>>>>>> before they could be read and verified, the testing framework
>>>>>> walks
>>>>>> the hash map keys in in a linear (or in the case of updates,
>>>>>> circular) order. By using a hash map with a large enough
>>>>>> capacity,
>>>>>> readers have the time to verify the recent modifications
>>>>>> before the
>>>>>> writer threads come back to modify that part of the key space
>>>>>> again.
>>>>>>
>>>>>> Using an adapter for the map implementation, the testing
>>>>>> framework
>>>>>> starts five writer threads and ten reader threads at the same
>>>>>> time.
>>>>>> These threads run wide open for 30 seconds, except that the
>>>>>> readers
>>>>>> will give up their time slice if they find nothing on the
>>>>>> queue. The
>>>>>> HashMaps were all sized for the needed capacity upon creation,
>>>>>> so no
>>>>>> resizing occurred during testing.
>>>>>>
>>>>>> I got some interesting results.
>>>>>>
>>>>>> Four implementations were tested, Java's unsynchronized HashMap
>>>>>> implementation, Java's synchronized HashMap implementation,
>>>>>> Java's
>>>>>> ConcurrentHashMap implementation, and OpenJPA's ConcurrentHashMap
>>>>>> implementation.
>>>>>>
>>>>>> Only Java's unsynchronized HashMap failed, as expected, under
>>>>>> test.
>>>>>> Under test, this implementation demonstrates its inability to
>>>>>> handle
>>>>>> concurrency. The other three implementations worked flawlessly
>>>>>> under
>>>>>> test.
>>>>>>
>>>>>> The java.util.concurrent.ConcurrentHashMap implementation
>>>>>> (available
>>>>>> with Java 5 and 6) was the fastest implementation tested.
>>>>>>
>>>>>> Java's synchronized wrapper for the HashMap implementation is
>>>>>> one to
>>>>>> two orders of magnitude slower than Java's ConcurrentHashMap
>>>>>> implementation.
>>>>>>
>>>>>> OpenJPA's ConcurrentHashMap compares equally with Java's
>>>>>> ConcurrentHashMap in find operations and is 2-4 times slower in
>>>>>> mutating operations.
>>>>>>
>>>>>> Implementation Add Remove Update Find-a Find-r Find-u
>>>>>> ---------------+------+-------+--------+-------+-------+------
>>>>>> synchronized 103 35 50 40 37 54
>>>>>> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
>>>>>> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
>>>>>>
>>>>>>
>>>>>> Legend:
>>>>>>
>>>>>> synchronized:
>>>>>> java.util.Collections.synchronizedMap(new java.util.HashMap())
>>>>>> concurrent: java.util.concurrent.ConcurrentHashMap
>>>>>> OpenJPA: org.apache.openjpa.lib.util.concurrent.ConcurrentHashMap
>>>>>>
>>>>>> Add: time for average add operation
>>>>>> Remove: time for average remove operation
>>>>>> Update: time for average update of new value for existing key
>>>>>> Find-a: time to find a recent addition
>>>>>> Find-r: time to NOT find a recent removal
>>>>>> Find-u: time to find a recent update
>>>>>>
>>>>>> These times (in microseconds) are representative, but are not the
>>>>>> average of several runs. The tests were run on a Dell Dual Core
>>>>>> laptop under Windows. The performance meter was pegged during the
>>> tests.
>>>>>>
>>>>>> David Ezzio
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
> Notice: This email message, together with any attachments, may
> contain information of BEA Systems, Inc., its subsidiaries
> and affiliated entities, that may be confidential, proprietary,
> copyrighted and/or legally privileged, and is intended solely for
> the use of the individual or entity named in this message. If you
> are not the intended recipient, and have received this message in
> error, please immediately return this by email and then delete it.
Re: Comparison Metrics for OpenJPA's ConcurrentHashMap
Posted by David Ezzio <de...@bea.com>.
Two
Patrick Linskey wrote:
> Hi,
>
> How many cores / CPUs were you using in these tests?
>
> -Patrick
>
> On 6/1/07, David Ezzio (asmtp) <de...@bea.com> wrote:
>> Hi Marc,
>>
>> I still was not able to get HighScaleLib to work. It produces a
>> SecurityException when attempting to get the Unsafe object. I decided to
>> avoid changing the relevant security policy.
>>
>> On the other hand, I did test Emory University's backport of the
>> java.util.concurrent package. This provides to Java 1.4 a
>> ConcurrentHashMap implementation that is compatible with the one found
>> in Java 5 and 6.
>>
>> I also realized that I could get another metric from my numbers, the
>> percentage of time the threads were suspended during the metered
>> operations. Then I reran the tests for Emory's backport using fewer
>> threads. In the classic pattern of an overloaded CPU, the higher thread
>> count both lowers throughput and increases response time. (Throughput
>> is yet another number I haven't extracted from the data, although it is
>> obvious when running the tests.)
>>
>> All of the previous tests were run with 5 writing and 10 reading
>> threads. Only backport was additionally tested with 2 writing and 4
>> reading threads. The suspended percentage is approximate since the
>> adding and updating tests have slightly different numbers.
>>
>> Implementation Add Remove Update Find-a Find-r Find-u Suspended
>> --------------+------+-------+-------+-------+-------+-------+---------
>> synchronized 103 35 50 40 37 54
>> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
>> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
>> Backport (5/10) 43.2 36.8 40.4 0.3 0.2 0.5 62%
>> Backport (2/4) 6.1 3.1 3.3 0.3 0.2 0.3 4%
>>
>> David
>>
>>
>> David Ezzio (asmtp) wrote:
>> > Hi Marc,
>> >
>> > I did plug it in, but it failed straightaway on a security issue. I
>> > should probably read its documentation. :) I'll try it again along
>> with
>> > the backport lib done by Emory U.
>> >
>> > David
>> >
>> >
>> > Marc Prud'hommeaux wrote:
>> >> David-
>> >>
>> >> That is very interesting.
>> >>
>> >> Did you also take a look at the one at
>> >> http://sourceforge.net/projects/high-scale-lib ? They say its
>> >> performance only shines for high thread/cpu counts, but it might be
>> >> interesting to see where its numbers lie in the range.
>> >>
>> >>
>> >>
>> >> On May 29, 2007, at 11:01 AM, David Ezzio (asmtp) wrote:
>> >>
>> >>> Recently, I did some testing of Map implementations under
>> concurrency.
>> >>>
>> >>> My primary purpose was to verify the reliability of OpenJPA's
>> >>> ConcurrentHashMap implementation. As I got into it, I saw the
>> >>> opportunity to get some performance metrics out of the test.
>> >>>
>> >>> The biggest part of my task was coming up with a reliable and useful
>> >>> testing framework. I design it with the following two factors in
>> >>> mind: First, I wanted to test the edge conditions where an entry had
>> >>> just been added or removed or where a key's value had just been
>> >>> updated. The idea is that a number of threads add, remove, and update
>> >>> entries, while other threads check to see if these recent
>> >>> modifications are visible (or in the case of removals, not visible).
>> >>> Second, I wanted the testing framework itself to be free of
>> >>> synchronization. If the testing framework used synchronization then
>> >>> it would tend to serialize the readers and writers and thereby mask
>> >>> concurrency issues in the map implementation under test.
>> >>>
>> >>> The testing framework uses a non-synchronizing, non-blocking FIFO
>> >>> queue as the mechanism for the writing threads to communicate their
>> >>> recent modifications to the reading threads.
>> >>>
>> >>> To prevent writing threads from overwriting recent modifications
>> >>> before they could be read and verified, the testing framework walks
>> >>> the hash map keys in in a linear (or in the case of updates,
>> >>> circular) order. By using a hash map with a large enough capacity,
>> >>> readers have the time to verify the recent modifications before the
>> >>> writer threads come back to modify that part of the key space again.
>> >>>
>> >>> Using an adapter for the map implementation, the testing framework
>> >>> starts five writer threads and ten reader threads at the same time.
>> >>> These threads run wide open for 30 seconds, except that the readers
>> >>> will give up their time slice if they find nothing on the queue. The
>> >>> HashMaps were all sized for the needed capacity upon creation, so no
>> >>> resizing occurred during testing.
>> >>>
>> >>> I got some interesting results.
>> >>>
>> >>> Four implementations were tested, Java's unsynchronized HashMap
>> >>> implementation, Java's synchronized HashMap implementation, Java's
>> >>> ConcurrentHashMap implementation, and OpenJPA's ConcurrentHashMap
>> >>> implementation.
>> >>>
>> >>> Only Java's unsynchronized HashMap failed, as expected, under test.
>> >>> Under test, this implementation demonstrates its inability to handle
>> >>> concurrency. The other three implementations worked flawlessly under
>> >>> test.
>> >>>
>> >>> The java.util.concurrent.ConcurrentHashMap implementation (available
>> >>> with Java 5 and 6) was the fastest implementation tested.
>> >>>
>> >>> Java's synchronized wrapper for the HashMap implementation is one to
>> >>> two orders of magnitude slower than Java's ConcurrentHashMap
>> >>> implementation.
>> >>>
>> >>> OpenJPA's ConcurrentHashMap compares equally with Java's
>> >>> ConcurrentHashMap in find operations and is 2-4 times slower in
>> >>> mutating operations.
>> >>>
>> >>> Implementation Add Remove Update Find-a Find-r Find-u
>> >>> ---------------+------+-------+--------+-------+-------+------
>> >>> synchronized 103 35 50 40 37 54
>> >>> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
>> >>> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
>> >>>
>> >>>
>> >>> Legend:
>> >>>
>> >>> synchronized:
>> >>> java.util.Collections.synchronizedMap(new java.util.HashMap())
>> >>> concurrent: java.util.concurrent.ConcurrentHashMap
>> >>> OpenJPA: org.apache.openjpa.lib.util.concurrent.ConcurrentHashMap
>> >>>
>> >>> Add: time for average add operation
>> >>> Remove: time for average remove operation
>> >>> Update: time for average update of new value for existing key
>> >>> Find-a: time to find a recent addition
>> >>> Find-r: time to NOT find a recent removal
>> >>> Find-u: time to find a recent update
>> >>>
>> >>> These times (in microseconds) are representative, but are not the
>> >>> average of several runs. The tests were run on a Dell Dual Core
>> >>> laptop under Windows. The performance meter was pegged during the
>> tests.
>> >>>
>> >>> David Ezzio
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>>
>>
>
>
Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.
Re: Comparison Metrics for OpenJPA's ConcurrentHashMap
Posted by Patrick Linskey <pl...@gmail.com>.
Hi,
How many cores / CPUs were you using in these tests?
-Patrick
On 6/1/07, David Ezzio (asmtp) <de...@bea.com> wrote:
> Hi Marc,
>
> I still was not able to get HighScaleLib to work. It produces a
> SecurityException when attempting to get the Unsafe object. I decided to
> avoid changing the relevant security policy.
>
> On the other hand, I did test Emory University's backport of the
> java.util.concurrent package. This provides to Java 1.4 a
> ConcurrentHashMap implementation that is compatible with the one found
> in Java 5 and 6.
>
> I also realized that I could get another metric from my numbers, the
> percentage of time the threads were suspended during the metered
> operations. Then I reran the tests for Emory's backport using fewer
> threads. In the classic pattern of an overloaded CPU, the higher thread
> count both lowers throughput and increases response time. (Throughput
> is yet another number I haven't extracted from the data, although it is
> obvious when running the tests.)
>
> All of the previous tests were run with 5 writing and 10 reading
> threads. Only backport was additionally tested with 2 writing and 4
> reading threads. The suspended percentage is approximate since the
> adding and updating tests have slightly different numbers.
>
> Implementation Add Remove Update Find-a Find-r Find-u Suspended
> --------------+------+-------+-------+-------+-------+-------+---------
> synchronized 103 35 50 40 37 54
> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
> Backport (5/10) 43.2 36.8 40.4 0.3 0.2 0.5 62%
> Backport (2/4) 6.1 3.1 3.3 0.3 0.2 0.3 4%
>
> David
>
>
> David Ezzio (asmtp) wrote:
> > Hi Marc,
> >
> > I did plug it in, but it failed straightaway on a security issue. I
> > should probably read its documentation. :) I'll try it again along with
> > the backport lib done by Emory U.
> >
> > David
> >
> >
> > Marc Prud'hommeaux wrote:
> >> David-
> >>
> >> That is very interesting.
> >>
> >> Did you also take a look at the one at
> >> http://sourceforge.net/projects/high-scale-lib ? They say its
> >> performance only shines for high thread/cpu counts, but it might be
> >> interesting to see where its numbers lie in the range.
> >>
> >>
> >>
> >> On May 29, 2007, at 11:01 AM, David Ezzio (asmtp) wrote:
> >>
> >>> Recently, I did some testing of Map implementations under concurrency.
> >>>
> >>> My primary purpose was to verify the reliability of OpenJPA's
> >>> ConcurrentHashMap implementation. As I got into it, I saw the
> >>> opportunity to get some performance metrics out of the test.
> >>>
> >>> The biggest part of my task was coming up with a reliable and useful
> >>> testing framework. I design it with the following two factors in
> >>> mind: First, I wanted to test the edge conditions where an entry had
> >>> just been added or removed or where a key's value had just been
> >>> updated. The idea is that a number of threads add, remove, and update
> >>> entries, while other threads check to see if these recent
> >>> modifications are visible (or in the case of removals, not visible).
> >>> Second, I wanted the testing framework itself to be free of
> >>> synchronization. If the testing framework used synchronization then
> >>> it would tend to serialize the readers and writers and thereby mask
> >>> concurrency issues in the map implementation under test.
> >>>
> >>> The testing framework uses a non-synchronizing, non-blocking FIFO
> >>> queue as the mechanism for the writing threads to communicate their
> >>> recent modifications to the reading threads.
> >>>
> >>> To prevent writing threads from overwriting recent modifications
> >>> before they could be read and verified, the testing framework walks
> >>> the hash map keys in in a linear (or in the case of updates,
> >>> circular) order. By using a hash map with a large enough capacity,
> >>> readers have the time to verify the recent modifications before the
> >>> writer threads come back to modify that part of the key space again.
> >>>
> >>> Using an adapter for the map implementation, the testing framework
> >>> starts five writer threads and ten reader threads at the same time.
> >>> These threads run wide open for 30 seconds, except that the readers
> >>> will give up their time slice if they find nothing on the queue. The
> >>> HashMaps were all sized for the needed capacity upon creation, so no
> >>> resizing occurred during testing.
> >>>
> >>> I got some interesting results.
> >>>
> >>> Four implementations were tested, Java's unsynchronized HashMap
> >>> implementation, Java's synchronized HashMap implementation, Java's
> >>> ConcurrentHashMap implementation, and OpenJPA's ConcurrentHashMap
> >>> implementation.
> >>>
> >>> Only Java's unsynchronized HashMap failed, as expected, under test.
> >>> Under test, this implementation demonstrates its inability to handle
> >>> concurrency. The other three implementations worked flawlessly under
> >>> test.
> >>>
> >>> The java.util.concurrent.ConcurrentHashMap implementation (available
> >>> with Java 5 and 6) was the fastest implementation tested.
> >>>
> >>> Java's synchronized wrapper for the HashMap implementation is one to
> >>> two orders of magnitude slower than Java's ConcurrentHashMap
> >>> implementation.
> >>>
> >>> OpenJPA's ConcurrentHashMap compares equally with Java's
> >>> ConcurrentHashMap in find operations and is 2-4 times slower in
> >>> mutating operations.
> >>>
> >>> Implementation Add Remove Update Find-a Find-r Find-u
> >>> ---------------+------+-------+--------+-------+-------+------
> >>> synchronized 103 35 50 40 37 54
> >>> concurrent 13.2 6.4 6.1 0.6 0.3 1.1
> >>> OpenJPA 29.8 26.6 27.9 0.6 0.6 0.6
> >>>
> >>>
> >>> Legend:
> >>>
> >>> synchronized:
> >>> java.util.Collections.synchronizedMap(new java.util.HashMap())
> >>> concurrent: java.util.concurrent.ConcurrentHashMap
> >>> OpenJPA: org.apache.openjpa.lib.util.concurrent.ConcurrentHashMap
> >>>
> >>> Add: time for average add operation
> >>> Remove: time for average remove operation
> >>> Update: time for average update of new value for existing key
> >>> Find-a: time to find a recent addition
> >>> Find-r: time to NOT find a recent removal
> >>> Find-u: time to find a recent update
> >>>
> >>> These times (in microseconds) are representative, but are not the
> >>> average of several runs. The tests were run on a Dell Dual Core
> >>> laptop under Windows. The performance meter was pegged during the tests.
> >>>
> >>> David Ezzio
> >>>
> >>>
> >>
> >>
> >
> >
>
>
--
Patrick Linskey
202 669 5907