[jira] [Resolved] (HDDS-13242) getTrashRoots is limited by the max list page size

["Ivan Andika (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ivan Andika resolved HDDS-13242.
--------------------------------
    Fix Version/s: 2.3.0
       Resolution: Fixed

> getTrashRoots is limited by the max list page size
> --------------------------------------------------
>
>                 Key: HDDS-13242
>                 URL: https://issues.apache.org/jira/browse/HDDS-13242
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Ivan Andika
>            Assignee: Ivan Andika
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 2.3.0
>
>
> TrashOzoneFileSystem#listStatus (used by TrashOzoneFileSystem#getTrashRoots) uses Integer.MAX_VALUE as the number of list entries. However, since HDDS-7290, a single listStatus is limited to ozone.fs.listing.page.size.max configuration, which defaults to 5000. 
> Although it is not common to have more than 5000 trash roots (e.g. which implies 5000 unique users deletes to trash in the same bucket). This might cause some trash roots to not be processed. We can first write a test to validate the issue.
> One solution is to use a listing iterator to send multiple smaller listStatus API, similar to OzoneFileStatusIterator. We can improve this iterator by persistently storing the current RocksDB iterator across listStatus calls instead of creating one for every listStatus call.
> Note: The FS client implementation BasicRootedOzoneFileSystem#listStatusAdapter (used by BasicRootedOzoneFileSystem#getTrashRoots) also default to 5000, so it will also have the same issues. However, since getTrashRoots are only used for server-side (OM) trash cleanup, the client-side getTrashRoots are rarely used.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org