You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Devesh Kumar Singh (Jira)" <ji...@apache.org> on 2023/12/14 13:35:00 UTC
[jira] [Updated] (HDDS-9883) Recon - Improve the performance of processing of IncrementalContainerReport requests from DN
[ https://issues.apache.org/jira/browse/HDDS-9883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devesh Kumar Singh updated HDDS-9883:
-------------------------------------
Description:
Between SCM *IncrementalContainerReportHandler* and *ReconIncrementalContainerReportHandler* , Recon always connecting to SCM and verifying for each container before adding new container to its own containerStateManager cache. This could be a bottle neck if SCM may respond slow and frequent ICR requests may pile up in queue. So as of now, we'll improve multiple things as part of this JIRA:
* Recon to verify containers in batches from SCM on receive of ICR request from DNs.
* Reduce the scmClient configs for Recon before connecting to SCM:
** hdds.scmclient.rpc.timeout - 1 min (Default value is 15 mins)
** hdds.scmclient.failover.max.retry - 3 (Default value is dynamic and computed, but based on default values, computed value is 15)
Above 2 SCM client configs will be updated to respective new values as mentioned for recon to connect to SCM. These 2 SCM client configs will be exposed and mapped with new recon configs to be able to adjust independently in recon.
*New configs in Recon:*
*
** ozone.recon.scmclient.rpc.timeout
** ozone.recon.scmclient.failover.max.retry
* Merge the Incremental container report (ICR) to existing list of ICR reports.
was:Between SCM *IncrementalContainerReportHandler* and *ReconIncrementalContainerReportHandler* , Recon always connecting to SCM and verifying for each container before adding new container to its own containerStateManager cache. This could be a bottle neck if SCM may respond slow and frequent ICR requests may pile up in queue. So as of now, we'll improve this part in Recon to implement queue based logic for verifying each container from SCM in async way and on receive of ICR request from DN, Recon will first simply add the new container if container not exists and also add the container info to a queue, then later an async task in Recon will process that queue to verify each container from SCM. This will decouple the ICR request processing at Recon and reduce the possibility of bottleneck in processing of ICR requests quickly and eventually reduce the possibility of getting "capacity not available" as ICR requests queue will be processed quickly.
> Recon - Improve the performance of processing of IncrementalContainerReport requests from DN
> --------------------------------------------------------------------------------------------
>
> Key: HDDS-9883
> URL: https://issues.apache.org/jira/browse/HDDS-9883
> Project: Apache Ozone
> Issue Type: Task
> Components: Ozone Recon
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
>
> Between SCM *IncrementalContainerReportHandler* and *ReconIncrementalContainerReportHandler* , Recon always connecting to SCM and verifying for each container before adding new container to its own containerStateManager cache. This could be a bottle neck if SCM may respond slow and frequent ICR requests may pile up in queue. So as of now, we'll improve multiple things as part of this JIRA:
> * Recon to verify containers in batches from SCM on receive of ICR request from DNs.
> * Reduce the scmClient configs for Recon before connecting to SCM:
> ** hdds.scmclient.rpc.timeout - 1 min (Default value is 15 mins)
> ** hdds.scmclient.failover.max.retry - 3 (Default value is dynamic and computed, but based on default values, computed value is 15)
> Above 2 SCM client configs will be updated to respective new values as mentioned for recon to connect to SCM. These 2 SCM client configs will be exposed and mapped with new recon configs to be able to adjust independently in recon.
> *New configs in Recon:*
> *
> ** ozone.recon.scmclient.rpc.timeout
> ** ozone.recon.scmclient.failover.max.retry
> * Merge the Incremental container report (ICR) to existing list of ICR reports.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org