[jira] [Created] (HDDS-12882) S3G should consider S3 max-keys parameter

["Ivan Andika (Jira)" <ji...@apache.org>]

Ivan Andika created HDDS-12882:
----------------------------------

             Summary: S3G should consider S3 max-keys parameter
                 Key: HDDS-12882
                 URL: https://issues.apache.org/jira/browse/HDDS-12882
             Project: Apache Ozone
          Issue Type: Sub-task
            Reporter: Ivan Andika
            Assignee: Ivan Andika


We have a situation where OM crash because S3 user keeps sending a lot listObjectsV2 with max-keys=1 parameter.

The code looks something like this:
{code:java}
s3_parquet_path = f"s3a://bucket/dir/*.parquet"
parquet_tab = spark.read.format("parquet").load(s3_parquet_path)
parquet_count = parquet_tab.count(){code}
We see that the KeyIterator implementation always send listStatus with maxKeys specified by ozone.client.list.cache (default is 1000). The default ozone.client.list.cache is also the same as the "max-keys" specified by S3G. If there are 1000 files inside this directory, the S3 client will send 1000 S3 listObjectsV2 (with max-keys=1), and OM has to load 1000 * 1000 = 1,000,000 OmKeyInfo entries to its memory. This causes OM heap space to be OOM.

We need to take into account this max-keys sent by S3. One possible idea is to use the S3 max-keys if it's smaller than the configured ozone.client.list.cache. This way, OM will only load the necessary OmKeyInfo to memory.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org