[jira] [Assigned] (HDDS-11508) Decouple delete batch limits from Ratis request size

["Ethan Rose (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ethan Rose reassigned HDDS-11508:
---------------------------------

    Assignee: Sadanand Shenoy

> Decouple delete batch limits from Ratis request size
> ----------------------------------------------------
>
>                 Key: HDDS-11508
>                 URL: https://issues.apache.org/jira/browse/HDDS-11508
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: OM, SCM
>            Reporter: Ethan Rose
>            Assignee: Sadanand Shenoy
>            Priority: Major
>              Labels: pull-request-available
>
> Currently the background deletion services in OM and SCM send one Ratis request per iteration. This can result in very large protos if the task's limits are high, and Ratis may reject them due to their size. HDDS-8977 tried to fix this by check the Ratis message size and falling back to a hardcoded limit of 1000 entries if the message is too big, but this is too conservative and will over-throttle the service under heavy deletion load.
> For each run, these entries should collect as many items as is their configured limit, and then break them into multiple Ratis requests to send out. This way limit per task can be set purely based on the time taken to gather the entries, and not based on size of the resulting proto.
> This Jira may optionally be split into subtasks for each service:
> * OM open key cleanup
> * OM file delete
> * OM directory delete
> * SCM block delete



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org