You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ccanning <cc...@stubhub.com> on 2016/04/22 01:41:11 UTC

Performance Issue - Threads blocking

We seem to be having some serious performance issues after adding Apache
Ignite Local cache to our APIs'. Looking at a heap dump, we seem to have a
bunch of threads blocked by this lock:

"ajp-0.0.0.0-8009-70" - Thread t@641
   java.lang.Thread.State: RUNNABLE
	at
org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:166)
	at
org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1486)
	at
org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:1830)
	at
org.apache.ignite.internal.binary.BinaryUtils.doReadMap(BinaryUtils.java:1813)
	at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1597)
	at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1646)
	at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read(BinaryFieldAccessor.java:643)
	at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:714)
	at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1450)
	at
org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:537)
	at
org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:117)
	at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:280)
	at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:145)
	at
org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:276)
	at
org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:159)
	at
org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:92)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet0(GridCacheMapEntry.java:862)
	- locked <70d32489> (a
org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet(GridCacheMapEntry.java:669)
	at
org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.getAllInternal(GridLocalAtomicCache.java:587)
	at
org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.get(GridLocalAtomicCache.java:483)
	at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1378)
	at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:864)
	at org.apache.ignite.cache.spring.SpringCache.get(SpringCache.java:52)

**** - locked <70d32489> (a
org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry)

Should this be causing blocking in a high-throughput API? Do you have any
pointers in how we could solve this issue?

Thanks.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance Issue - Threads blocking

Posted by ccanning <cc...@stubhub.com>.
Forgot, out of those requests about 100 objects are the target of 90+% of
requests.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433p4629.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance Issue - Threads blocking

Posted by ccanning <cc...@stubhub.com>.
Thanks Matt, 

you nailed the problem. We have scenario where there are a bunch of hot keys
that are accessed concurrently (about 1.2 billion requests/day but
consistently 150 concurrent users and peaks of 10x). 

For us, this cache is meant to always be read-only and we never need to copy
as data is never dirtied. To help with the problem, I moved all values into
the off-heap memory and only kept the keys in-heap and disabled eagerTTL. 

We are still getting performance problems compared to using ehcache with the
BigMemoryGo off-heap implementation. But, the above made it usable. 

Just to give you performance numbers:

<bean class="org.apache.ignite.configuration.CacheConfiguration"
              p:name="myCache"
              p:cacheMode="LOCAL"
              p:memoryMode="ONHEAP_TIERED"
              p:offHeapMaxMemory="#{256 * 1024 * 1024}"
              p:evictionPolicy-ref="max50Elements"
              p:expiryPolicyFactory-ref="2HourTTL"
              p:statisticsEnabled="true"
              p:managementEnabled="true"
              p:swapEnabled="false"/> 

~ 17 seconds per request with the blocking

<bean class="org.apache.ignite.configuration.CacheConfiguration"
          p:name="myCache"
          p:cacheMode="LOCAL"
          p:atomicityMode="ATOMIC"
          p:memoryMode="OFFHEAP_VALUES"
          p:offHeapMaxMemory="#{256 * 1024 * 1024}"
          p:statisticsEnabled="true"
          p:managementEnabled="true"
          p:swapEnabled="false"
          p:eagerTtl="false"
          p:expiryPolicyFactory-ref="2HourTTL"/>

Median time: 64 ms, 90% 120ms, max time: 1.2 seconds 

The max was cold cache. 

Turning off copy should probably put it in the same ballpark as bigmemory as
that option was called out in their documentation. 

A complete read-only option that only blocks when evisting (if even needed
might provide even more head-room.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433p4628.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance Issue - Threads blocking

Posted by vkulichenko <va...@gmail.com>.
Hi Matt,


Matt Hoffman wrote
> Based on his claim of a lot of threads waiting on the same locks, I
> assumed
> that's what was happening -- high contention for a few cache keys. I don't
> know his use case, but I can imagine cases with a fairly small number of
> very "hot" entries.
> It wouldn't necessarily require very few keys, right? Just high contention
> on a few of them.

This is right, but what are you expectations in this scenario? Concurrent
updates/reads of the same entry have to be synchronized, and Ignite locks
only on per-entry level. In my view, this is the highest concurrency level
possible.





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433p4508.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance Issue - Threads blocking

Posted by vkulichenko <va...@gmail.com>.
Matt Hoffman wrote
> Do you think that would be a candidate for the "Performance tips" page in
> the docs? I know I've referred to that page a few times recently myself.

That's a good point, thanks. I updated the docs.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433p4509.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance Issue - Threads blocking

Posted by Matt Hoffman <ma...@mhoffman.org>.
(Inline)

On Fri, Apr 22, 2016, 4:26 PM vkulichenko <va...@gmail.com>
wrote:
>
> Hi Matt,
>
> I'm confused. The locking does happen on per-entry level, otherwise it's
> impossible to guarantee data consistency. Two concurrent updates or reads
> for the same key will wait for each other on this lock. But this should
not
> cause performance degradation, unless you have very few keys and very high
> contention on them.
>

Based on his claim of a lot of threads waiting on the same locks, I assumed
that's what was happening -- high contention for a few cache keys. I don't
know his use case, but I can imagine cases with a fairly small number of
very "hot" entries.
It wouldn't necessarily require very few keys, right? Just high contention
on a few of them.

> The only thing I see here is that the value is deserialized on read. This
is
> done because JCache requires store-by-value semantics and thus we create a
> copy each time you get the value (by deserializing its binary
> representation). You can override this behavior by setting
> CacheConfiguration.setCopyOnRead(false) property, this should give you
> performance improvement. Only note that it's not safe to modify the
instance
> that you got from cache this way.
>

Do you think that would be a candidate for the "Performance tips" page in
the docs? I know I've referred to that page a few times recently myself.

> -Val
>
>
>
> --
> View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433p4465.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance Issue - Threads blocking

Posted by vkulichenko <va...@gmail.com>.
Hi Matt,

I'm confused. The locking does happen on per-entry level, otherwise it's
impossible to guarantee data consistency. Two concurrent updates or reads
for the same key will wait for each other on this lock. But this should not
cause performance degradation, unless you have very few keys and very high
contention on them.

The only thing I see here is that the value is deserialized on read. This is
done because JCache requires store-by-value semantics and thus we create a
copy each time you get the value (by deserializing its binary
representation). You can override this behavior by setting
CacheConfiguration.setCopyOnRead(false) property, this should give you
performance improvement. Only note that it's not safe to modify the instance
that you got from cache this way.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433p4465.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance Issue - Threads blocking

Posted by Matt Hoffman <ma...@mhoffman.org>.
I'm assuming you're seeing a lot of threads that are BLOCKED waiting on
that locked GridLocalCacheEntry (<70d32489> in that example you pasted
above). Looking at the code, it looks like it does block on individual
cache entries (so two reads of the same key within the same JVM will
block). In your particular example above, the thread in question is
publishing an EVT_CACHE_OBJECT_EXPIRED event. If you don't need that,
turning it off (along with EVT_CACHE_OBJECT_READ) will speed up the time
that the cache entry spends blocking other reads (and speed things up,
generally).
It's locking to make sure it's deserialized from swap once and expired once
(if necessary; looks like it was in this particular case).

matt

On Fri, Apr 22, 2016 at 8:07 AM, Vladimir Ozerov <vo...@gridgain.com>
wrote:

> Hi,
>
> Could you please explain why do you think that the thread is blocked? I
> see it is in a RUNNABLE state.
>
> Vladimir.
>
> On Fri, Apr 22, 2016 at 2:41 AM, ccanning <cc...@stubhub.com> wrote:
>
>> We seem to be having some serious performance issues after adding Apache
>> Ignite Local cache to our APIs'. Looking at a heap dump, we seem to have a
>> bunch of threads blocked by this lock:
>>
>> "ajp-0.0.0.0-8009-70" - Thread t@641
>>    java.lang.Thread.State: RUNNABLE
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:166)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1486)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:1830)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadMap(BinaryUtils.java:1813)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1597)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1646)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read(BinaryFieldAccessor.java:643)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:714)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1450)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:537)
>>         at
>>
>> org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:117)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:280)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:145)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:276)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:159)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:92)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet0(GridCacheMapEntry.java:862)
>>         - locked <70d32489> (a
>> org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet(GridCacheMapEntry.java:669)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.getAllInternal(GridLocalAtomicCache.java:587)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.get(GridLocalAtomicCache.java:483)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1378)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:864)
>>         at
>> org.apache.ignite.cache.spring.SpringCache.get(SpringCache.java:52)
>>
>> **** - locked <70d32489> (a
>> org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry)
>>
>> Should this be causing blocking in a high-throughput API? Do you have any
>> pointers in how we could solve this issue?
>>
>> Thanks.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Performance Issue - Threads blocking

Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi,

Could you please explain why do you think that the thread is blocked? I see
it is in a RUNNABLE state.

Vladimir.

On Fri, Apr 22, 2016 at 2:41 AM, ccanning <cc...@stubhub.com> wrote:

> We seem to be having some serious performance issues after adding Apache
> Ignite Local cache to our APIs'. Looking at a heap dump, we seem to have a
> bunch of threads blocked by this lock:
>
> "ajp-0.0.0.0-8009-70" - Thread t@641
>    java.lang.Thread.State: RUNNABLE
>         at
>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:166)
>         at
>
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1486)
>         at
>
> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:1830)
>         at
>
> org.apache.ignite.internal.binary.BinaryUtils.doReadMap(BinaryUtils.java:1813)
>         at
>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1597)
>         at
>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1646)
>         at
>
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read(BinaryFieldAccessor.java:643)
>         at
>
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:714)
>         at
>
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1450)
>         at
>
> org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:537)
>         at
>
> org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:117)
>         at
>
> org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:280)
>         at
>
> org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:145)
>         at
>
> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:276)
>         at
>
> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:159)
>         at
>
> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:92)
>         at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet0(GridCacheMapEntry.java:862)
>         - locked <70d32489> (a
> org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry)
>         at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet(GridCacheMapEntry.java:669)
>         at
>
> org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.getAllInternal(GridLocalAtomicCache.java:587)
>         at
>
> org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.get(GridLocalAtomicCache.java:483)
>         at
>
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1378)
>         at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:864)
>         at
> org.apache.ignite.cache.spring.SpringCache.get(SpringCache.java:52)
>
> **** - locked <70d32489> (a
> org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry)
>
> Should this be causing blocking in a high-throughput API? Do you have any
> pointers in how we could solve this issue?
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>