You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@ozone.apache.org by "Sammi Chen (Jira)" <ji...@apache.org> on 2022/10/31 07:51:00 UTC

[jira] [Updated] (HDDS-7244) Fix multiple reports queued up from same DN and using up heap

     [ https://issues.apache.org/jira/browse/HDDS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sammi Chen updated HDDS-7244:
-----------------------------
    Fix Version/s: 1.4.0

> Fix multiple reports queued up from same DN and using up heap
> -------------------------------------------------------------
>
>                 Key: HDDS-7244
>                 URL: https://issues.apache.org/jira/browse/HDDS-7244
>             Project: Apache Ozone
>          Issue Type: Improvement
>         Environment: Environment Observation:
>  # Disk is not SSD
>  # DNs- 96, Containers for each DNs on average approx 12K - 15K
>  # Hearbeat initially is 60sec, but same observed with 20 minutes also.
>  # Observed multiple threads waiting on write lock in thread dump, but not a deadlock.
>            Reporter: Sumit Agrawal
>            Assignee: Sumit Agrawal
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 1.4.0
>
>         Attachments: Handling multiple FCR for the datanode in event queue .docx
>
>
> When processing of FCR/ICR reports from datanode is slow, queue keeps getting filled with these reports. Slowness can be due to:
>  * Disk being slow (recommended SSD is not configured)
>  * Number of datanodes and container is huge
>  * Heartbeat from DNs configured with high frequency
>  * Thread contention / starvation where global locks are present
> The issue is observed in combination of multiple condition as given above and reports keeps on appending to EventQueue for processing. This further results in heapdump over a period of time.
> This is observed for both SCM and Recon server



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org