You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "HUANG DUICAN (JIRA)" <ji...@apache.org> on 2019/01/05 03:42:00 UTC
[jira] [Updated] (CASSANDRA-14953) Failed to reclaim the memory and
too many MemtableReclaimMemory pending task
[ https://issues.apache.org/jira/browse/CASSANDRA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
HUANG DUICAN updated CASSANDRA-14953:
-------------------------------------
Description:
We found that Cassandra has a lot of write accumulation in the production environment, and our business has experienced a lot of write failures.
Through the system.log, it was found that MemtableReclaimMemory was pending at the beginning, and then a large number of MutationStage stacks appeared at a certain moment.
Finally, the heap memory is full, the GC time reaches tens of seconds, the node status is DN through nodetool, but the Cassandra process is still running.We killed the node and restarted the node, and the above situation disappeared.
Also the number of Active MemtableReclaimMemory threads seems to stay at 1.
a large number of MutationStage stacks appeared at a certain moment.
long GC time:
Why is this happening?
was:
We found that Cassandra has a lot of write accumulation in the production environment, and our business has experienced a lot of write failures.
Through the system.log, it was found that MemtableReclaimMemory was pending at the beginning, and then a large number of MutationStage stacks appeared at a certain moment.
Finally, the heap memory is full, the GC time reaches tens of seconds, the node status is DN through nodetool, but the Cassandra process is still running.We killed the node and restarted the node, and the above situation disappeared.
Also the number of Active MemtableReclaimMemory threads seems to stay at 1.
!image-2019-01-05-11-36-31-199.png!
a large number of MutationStage stacks appeared at a certain moment.
!image-2019-01-05-11-37-54-253.png!
long GC time:
!image-2019-01-05-11-38-21-711.png!
Why is this happening?
> Failed to reclaim the memory and too many MemtableReclaimMemory pending task
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-14953
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14953
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Memtable
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
> Reporter: HUANG DUICAN
> Priority: Major
> Attachments: cassandra_20190105.zip
>
>
> We found that Cassandra has a lot of write accumulation in the production environment, and our business has experienced a lot of write failures.
> Through the system.log, it was found that MemtableReclaimMemory was pending at the beginning, and then a large number of MutationStage stacks appeared at a certain moment.
> Finally, the heap memory is full, the GC time reaches tens of seconds, the node status is DN through nodetool, but the Cassandra process is still running.We killed the node and restarted the node, and the above situation disappeared.
>
> Also the number of Active MemtableReclaimMemory threads seems to stay at 1.
>
> a large number of MutationStage stacks appeared at a certain moment.
> long GC time:
>
> Why is this happening?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org