You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ChenXi (Jira)" <ji...@apache.org> on 2023/08/01 08:04:00 UTC
[jira] [Commented] (HDDS-9102) Reduce DN long-tail latency of handle Request
[ https://issues.apache.org/jira/browse/HDDS-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749527#comment-17749527 ]
ChenXi commented on HDDS-9102:
------------------------------
We have found that some GET_BLOCK Requests will take tens of seconds to process which is not normal performance.
After some investigation, we found that the root cause is the {{BlockDeletingService}} thread, which has been holding the Container lock for a long time, causing the CheckContainerIsHealthy function to be unable to acquire the lock.
!image-2023-08-01-15-32-15-327.png|width=1007,height=208!
!image-2023-08-01-15-28-59-694.png!
We can compare the log of BlockDeletingService with the checkContainerHealthy function. Both threads want to acquire the lock of Container 21154, but `checkContainerHealthy` acquires the lock after the BlockDeletingService process ends. The checkContainerHealthy took 20s to acquire the Container lock
!image-2023-08-01-15-32-53-913.png!
> Reduce DN long-tail latency of handle Request
> ----------------------------------------------
>
> Key: HDDS-9102
> URL: https://issues.apache.org/jira/browse/HDDS-9102
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: ChenXi
> Assignee: ChenXi
> Priority: Major
> Attachments: image-2023-08-01-15-28-59-694.png, image-2023-08-01-15-31-54-492.png, image-2023-08-01-15-32-15-327.png, image-2023-08-01-15-32-53-913.png
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org