You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Gargi-jais11 (via GitHub)" <gi...@apache.org> on 2026/06/23 08:28:27 UTC

[PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Gargi-jais11 opened a new pull request, #10589:
URL: https://github.com/apache/ozone/pull/10589

   ## What changes were proposed in this pull request?
   s3-tests `test_bucket_create_special_key_names` fails because a key named      `   **' '**` is created but not returned by ListObjects. `OmMetadataManagerImpl.getOzoneKey()` uses `StringUtils.isNotBlank()`, which treats whitespace-only keys as absent, so the object is stored under /volume/bucket instead of /volume/bucket/.
   List object uses prefix `/volume/bucket/`, so the key is omitted. Fix key encoding to allow whitespace-only names
   
   ```
   Expected: [' ', '"', '$', '%', '&', "'", '<', '>', '_', '_ ', '_ _', '__']
   Actual:   ['"', '$', '%', '&', "'", '<', '>', '_', '_ ', '_ _'] 
   
     # missing leading ' '
   
   At index 0: ' ' != '"' 
   ```
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-15618
   
   ## How was this patch tested?
   
   Added unit test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #10589:
URL: https://github.com/apache/ozone/pull/10589#issuecomment-4830941263

   Thanks @Gargi-jais11 for the patch and @chungen0126 for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10589:
URL: https://github.com/apache/ozone/pull/10589#discussion_r3460981875


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java:
##########
@@ -1294,4 +1294,29 @@ public void testGetMultipartUploadKeys() throws Exception {
 
     assertEquals(25, noPagination.size());
   }
+
+  @Test
+  public void testListKeysSpecialKeyNames() throws Exception {
+    List<String> keyNames = Arrays.asList(" ", "\"",
+        "$", "%", "&", "'", "<", ">", "_", "_ ", "_ _", "__");
+
+    String volumeName = "volumeA";
+    String bucketName = "bucketA";
+    OMRequestTestUtils.addVolumeToDB(volumeName, omMetadataManager);
+    addBucketsToCache(volumeName, bucketName);
+
+    assertEquals("/volumeA/bucketA/ ",
+        omMetadataManager.getOzoneKey(volumeName, bucketName, " "));
+
+    for (int i = 0; i < keyNames.size(); i++) {
+      addKeysToOM(volumeName, bucketName, keyNames.get(i), i);
+    }

Review Comment:
   I'd prefer if we actually have a S3 integration test to prove that Ozone actually accepts a blank key (I'm not sure if it does). If not, then the S3 single white space S3 object key path will never be exercised.



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java:
##########
@@ -1294,4 +1294,29 @@ public void testGetMultipartUploadKeys() throws Exception {
 
     assertEquals(25, noPagination.size());
   }
+
+  @Test
+  public void testListKeysSpecialKeyNames() throws Exception {
+    List<String> keyNames = Arrays.asList(" ", "\"",
+        "$", "%", "&", "'", "<", ">", "_", "_ ", "_ _", "__");
+
+    String volumeName = "volumeA";
+    String bucketName = "bucketA";
+    OMRequestTestUtils.addVolumeToDB(volumeName, omMetadataManager);
+    addBucketsToCache(volumeName, bucketName);
+
+    assertEquals("/volumeA/bucketA/ ",
+        omMetadataManager.getOzoneKey(volumeName, bucketName, " "));
+
+    for (int i = 0; i < keyNames.size(); i++) {
+      addKeysToOM(volumeName, bucketName, keyNames.get(i), i);
+    }

Review Comment:
   I'd prefer if we actually have an S3 integration test to prove that Ozone actually accepts a blank key (I'm not sure if it does). If not, then the S3 single white space S3 object key path will never be exercised.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 merged PR #10589:
URL: https://github.com/apache/ozone/pull/10589


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10589:
URL: https://github.com/apache/ozone/pull/10589#discussion_r3460981875


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java:
##########
@@ -1294,4 +1294,29 @@ public void testGetMultipartUploadKeys() throws Exception {
 
     assertEquals(25, noPagination.size());
   }
+
+  @Test
+  public void testListKeysSpecialKeyNames() throws Exception {
+    List<String> keyNames = Arrays.asList(" ", "\"",
+        "$", "%", "&", "'", "<", ">", "_", "_ ", "_ _", "__");
+
+    String volumeName = "volumeA";
+    String bucketName = "bucketA";
+    OMRequestTestUtils.addVolumeToDB(volumeName, omMetadataManager);
+    addBucketsToCache(volumeName, bucketName);
+
+    assertEquals("/volumeA/bucketA/ ",
+        omMetadataManager.getOzoneKey(volumeName, bucketName, " "));
+
+    for (int i = 0; i < keyNames.size(); i++) {
+      addKeysToOM(volumeName, bucketName, keyNames.get(i), i);
+    }

Review Comment:
   I'd prefer if we actually have an S3 integration test to prove that Ozone actually accepts a blank key (I'm not sure if it does). If OM does not accept a blanc key, then the S3 single white space S3 object key path will never be exercised.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10589:
URL: https://github.com/apache/ozone/pull/10589#discussion_r3460981875


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java:
##########
@@ -1294,4 +1294,29 @@ public void testGetMultipartUploadKeys() throws Exception {
 
     assertEquals(25, noPagination.size());
   }
+
+  @Test
+  public void testListKeysSpecialKeyNames() throws Exception {
+    List<String> keyNames = Arrays.asList(" ", "\"",
+        "$", "%", "&", "'", "<", ">", "_", "_ ", "_ _", "__");
+
+    String volumeName = "volumeA";
+    String bucketName = "bucketA";
+    OMRequestTestUtils.addVolumeToDB(volumeName, omMetadataManager);
+    addBucketsToCache(volumeName, bucketName);
+
+    assertEquals("/volumeA/bucketA/ ",
+        omMetadataManager.getOzoneKey(volumeName, bucketName, " "));
+
+    for (int i = 0; i < keyNames.size(); i++) {
+      addKeysToOM(volumeName, bucketName, keyNames.get(i), i);
+    }

Review Comment:
   I'd prefer if we actually have an S3 integration test to prove that Ozone actually accepts a blank key (I'm not sure if it does). If OM does not accept a blanc key, then the S3 single white space case will never be exercised.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10589:
URL: https://github.com/apache/ozone/pull/10589#discussion_r3490050932


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java:
##########
@@ -1294,4 +1294,29 @@ public void testGetMultipartUploadKeys() throws Exception {
 
     assertEquals(25, noPagination.size());
   }
+
+  @Test
+  public void testListKeysSpecialKeyNames() throws Exception {
+    List<String> keyNames = Arrays.asList(" ", "\"",
+        "$", "%", "&", "'", "<", ">", "_", "_ ", "_ _", "__");
+
+    String volumeName = "volumeA";
+    String bucketName = "bucketA";
+    OMRequestTestUtils.addVolumeToDB(volumeName, omMetadataManager);
+    addBucketsToCache(volumeName, bucketName);
+
+    assertEquals("/volumeA/bucketA/ ",
+        omMetadataManager.getOzoneKey(volumeName, bucketName, " "));
+
+    for (int i = 0; i < keyNames.size(); i++) {
+      addKeysToOM(volumeName, bucketName, keyNames.get(i), i);
+    }

Review Comment:
   I have added integration test. Ozone actually accepts a blank key, below I have given evidence of it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on PR #10589:
URL: https://github.com/apache/ozone/pull/10589#issuecomment-4829305883

   > Thanks @Gargi-jais11 for working on this.
   > 
   > As far as I know, HDFS doesn't support filenames with spaces. I'm concerning about how this change impact that. Have you looked into it?
   
   I have tested creating hdfs filenames with spaces and it does support. Evidence:
   ```
   [datanode -1 ~]# echo "test" | hdfs dfs -put - "/tmp/ "
   
   [datanode-1 ~]# hdfs dfs -ls "/tmp" | cat -A
   Found 1 items$
   -rw-r--r--   3 hdfs supergroup          5 2026-06-29 05:25 /tmp/ $
   ```
   
   and even ozone supports creating key with space :
   ```
   ozone sh key put /vol1/buck1/" " test
   [datanode-1 ~]# ozone sh key list /vol1/buck1
   [ {
     "volumeName" : "vol1",
     "bucketName" : "buck1",
     "name" : " ",
     "owner" : "om",
     "dataSize" : 11,
     "creationTime" : "2026-06-29T05:28:46.616Z",
     "modificationTime" : "2026-06-29T05:28:48.179Z",
     "replicationConfig" : {
       "replicationFactor" : "THREE",
       "requiredNodes" : 3,
       "minimumNodes" : 1,
       "replicationType" : "RATIS"
     },
     "metadata" : { },
     "tags" : { },
     "file" : true
   } ]
   ```
   The S3 issue was not “Ozone/HDFS rejects space keys.” It was a **create vs list mismatch** caused by a bug in how OM builds the internal metadata key. OM treated " " as “no key segment” and stored it at /vol1/buck1 instead of /vol1/buck1/.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on PR #10589:
URL: https://github.com/apache/ozone/pull/10589#issuecomment-4786735697

   Thanks @Gargi-jais11 for working on this. 
   
   As far as I know, HDFS doesn't support filenames with spaces. I'm concerning about how this change impact that. Have you looked into it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15618. Fix ListObjects to include single whitespace-only S3 object keys [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10589:
URL: https://github.com/apache/ozone/pull/10589#discussion_r3460981875


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java:
##########
@@ -1294,4 +1294,29 @@ public void testGetMultipartUploadKeys() throws Exception {
 
     assertEquals(25, noPagination.size());
   }
+
+  @Test
+  public void testListKeysSpecialKeyNames() throws Exception {
+    List<String> keyNames = Arrays.asList(" ", "\"",
+        "$", "%", "&", "'", "<", ">", "_", "_ ", "_ _", "__");
+
+    String volumeName = "volumeA";
+    String bucketName = "bucketA";
+    OMRequestTestUtils.addVolumeToDB(volumeName, omMetadataManager);
+    addBucketsToCache(volumeName, bucketName);
+
+    assertEquals("/volumeA/bucketA/ ",
+        omMetadataManager.getOzoneKey(volumeName, bucketName, " "));
+
+    for (int i = 0; i < keyNames.size(); i++) {
+      addKeysToOM(volumeName, bucketName, keyNames.get(i), i);
+    }

Review Comment:
   I'd prefer if we actually have an S3 integration test to prove that Ozone actually accepts a blank key (I'm not sure if it does). If OM does not accept a blank key, then the S3 single white space case will never be exercised.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org