[jira] [Updated] (IGNITE-14244) Performance degradation of entire ignite because of indexed persisted cache

["Vyacheslav Puzakov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vyacheslav Puzakov updated IGNITE-14244:
----------------------------------------
    Attachment:     (was: 632.zip)

> Performance degradation of entire ignite because of indexed persisted cache
> ---------------------------------------------------------------------------
>
>                 Key: IGNITE-14244
>                 URL: https://issues.apache.org/jira/browse/IGNITE-14244
>             Project: Ignite
>          Issue Type: Bug
>          Components: cache, data structures
>    Affects Versions: 2.9.1
>         Environment: 2 nodes
> 56 cpu per node
> 600gb ram per node
> ssd(wal)/hdd(storage)
>            Reporter: Vyacheslav Puzakov
>            Priority: Blocker
>              Labels: Ignite, performance, persistence
>
> We have multiple not persisted caches and *one* with persistenc *on* (needed for events registration)
>  
> Ignite nodes starting up inside Business module (share single jvm)
> All caches are transactional.
>  
> Persisted cache key, value:
>  
> key:
> long recordId; (unique)
> Int type;.       \ affinity
> int entityid;  /
>  
>  
> Value:
> long evtDate;
> String system;
> ... and other not necessary fields
>  
> Indexes:
> 1) EntityId, entityType, system
> 2) evtDate, system
>  
> Case:
>  
> 1) Put value on *not* persisted cache,
> 2) Put new value on *persisted* cache (register event of creation)
> 3) remove value on *not* persisted cache
> 4) Put new value on *persisted* cache (register event of delete)
>  
> On start I start to send 4000 tps of that case.
> so 4k put/remove from not persisted cache and 8k values added to persisted cache per second.
>  
> It running okay until 10 mil of entries added in persisted cache.
> After that performance start to slowdown. Transaction timeouts begin showing up (and not only on persisted cache but *on NOT persisted caches TOO*!)
> And throughput lowers to 3k tps of 4k tps input (business threads reject 1k tps cause internal threads cant process more because of ignite)
>  
> On 20 mil - handled tps dropping to 100 tps. And so on.
> All business threads waiting on commit (and then after ignite response - failing with timeout)
>  
> 1) Is that because of index update?
> 2) All requests to not persisted caches slowdown too accordingly (waiting on commit too, even tho its simple get)
>  
> CPU utilization only around 30%. Memory not swapping, gc is good, cant say about IO but it looks good (any tip how to check that out will be nice)
>  
> I tried to remove all indexes and it slowdowns too overtime.
>  
> I attached thread dumps and ignite log of 1 node.
> PropagationTaskExecutorThread-poo[]-thread[%d] - business threads that struggling with ignite call, all sit on LockSupport.park.
>  
> Would be nice to see any help or explanation why persisted cache is so slow :(



--
This message was sent by Atlassian Jira
(v8.3.4#803005)