[jira] [Created] (IGNITE-14244) Performance degradation of entire ignite because of indexed persisted cache

["Vyacheslav Puzakov (Jira)" <ji...@apache.org>]

Vyacheslav Puzakov created IGNITE-14244:
-------------------------------------------

             Summary: Performance degradation of entire ignite because of indexed persisted cache
                 Key: IGNITE-14244
                 URL: https://issues.apache.org/jira/browse/IGNITE-14244
             Project: Ignite
          Issue Type: Bug
          Components: cache, data structures
    Affects Versions: 2.9.1
         Environment: 2 nodes

56 cpu per node

600gb ram per node

ssd(wal)/hdd(storage)
            Reporter: Vyacheslav Puzakov
         Attachments: 632.zip

We have multiple not persisted caches and *one* with persistenc *on* (needed for events registration)
 
Ignite nodes starting up inside Business module (share single jvm)
All caches are transactional.
 
Persisted cache key, value:
 
key:
long recordId; (unique)
Int type;.       \ affinity
int entityid;  /
 
 
Value:
long evtDate;
String system;
... and other not necessary fields
 
Indexes:
1) EntityId, entityType, system
2) evtDate, system
 
Case:
 
1) Put value on *not* persisted cache,
2) Put new value on *persisted* cache (register event of creation)
3) remove value on *not* persisted cache
4) Put new value on *persisted* cache (register event of delete)
 
On start I start to send 4000 tps of that case.
so 4k put/remove from not persisted cache and 8k values added to persisted cache per second.
 
It running okay until 10 mil of entries added in persisted cache.
After that performance start to slowdown. Transaction timeouts begin showing up (and not only on persisted cache but *on NOT persisted caches TOO*!)
And throughput lowers to 3k tps of 4k tps input (business threads reject 1k tps cause internal threads cant process more because of ignite)
 
On 20 mil - handled tps dropping to 100 tps. And so on.
All business threads waiting on commit (and then after ignite response - failing with timeout)
 
1) Is that because of index update?
2) All requests to not persisted caches slowdown too accordingly (waiting on commit too, even tho its simple get)
 
CPU utilization only around 30%. Memory not swapping, gc is good, cant say about IO but it looks good (any tip how to check that out will be nice)
 
I tried to remove all indexes and it slowdowns too overtime.
 
I attached thread dumps and ignite log of 1 node.
PropagationTaskExecutorThread-poo[]-thread[%d] - business threads that struggling with ignite call, all sit on LockSupport.park.
 
Would be nice to see any help or explanation why persisted cache is so slow :(



--
This message was sent by Atlassian Jira
(v8.3.4#803005)