You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ivan Andika (Jira)" <ji...@apache.org> on 2025/07/30 03:54:00 UTC
[jira] [Commented] (HDDS-12882) S3G should consider S3 listObjects max-keys parameter
[ https://issues.apache.org/jira/browse/HDDS-12882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18010794#comment-18010794 ]
Ivan Andika commented on HDDS-12882:
------------------------------------
Goofys uses listObjects with maxKeys equals to 1 or 2 as part of its directory mechanism
* MaxKeys = 1 is used in Inode#LookUpInodeDir
** [https://github.com/kahing/goofys/blob/350ff312abaa1abcf21c5a06e143c7edffe9e2f4/internal/dir.go#L1311]
** Used in
*** Inode#LookUpInodeMaybeDir
**** Inode#LookUp
***** Goofys#LookUpInode (which implements FileSystem#LookUpInode interface)
****** LookUpInodeOp look up a child by name within a parent directory
******* Notable attributes
******** Parent InodeID: The ID of the directory inode to which the child belongs
******** Name: The name of the child of interest, relative to the parent
** This would be involved in directory traversal and might be called for each file
*** Which is dangerous since this might cause OM to load 1000 OmKeyInfo per directory access
* MaxKeys = 2 is used in Inode#isEmptyDir (dir.go)
** [https://github.com/kahing/goofys/blob/350ff312abaa1abcf21c5a06e143c7edffe9e2f4/internal/dir.go#L977]
** Used in
*** Inode#RmDir
*** Inode#Rename
** This operation might not be done as often
We might need to handle these special cases. We can maybe start by handling MaxKeys=1 first since this should be used more often.
> S3G should consider S3 listObjects max-keys parameter
> -----------------------------------------------------
>
> Key: HDDS-12882
> URL: https://issues.apache.org/jira/browse/HDDS-12882
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We have a situation where OM crash with heap space OOM because S3 user keeps sending a lot listObjectsV2 with max-keys=1 parameter as a way to check the object's owner (although GetObjectAcl should be a better alternative).
> We see that the KeyIterator implementation always send listStatus with maxKeys specified by ozone.client.list.cache (default is 1000). The default ozone.client.list.cache of 1000 is also the same as the "max-keys" specified by S3G, so normally it will be ok. If there are 1000 files inside this directory, the S3 client will send 1000 S3 listObjectsV2 (with max-keys=1), and OM has to load 1000 * 1000 = 1,000,000 OmKeyInfo entries to its memory. This causes OM heap space to be OOM.
> We need to take into account this max-keys sent by S3. One possible idea is to use the S3 max-keys if it's smaller than the configured ozone.client.list.cache. This way, OM will only load the necessary OmKeyInfo to memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org