You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Gargi-jais11 (via GitHub)" <gi...@apache.org> on 2026/06/23 06:03:39 UTC
[PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Gargi-jais11 opened a new pull request, #10586:
URL: https://github.com/apache/ozone/pull/10586
## What changes were proposed in this pull request?
Ozone S3 Gateway fails s3-tests ListObjects cases that AWS S3 passes:
- `test_bucket_list_encoding_basic` — With EncodingType=url, keys and CommonPrefixes containing spaces are encoded with + (Java URLEncoder form encoding) instead of AWS-style %20.
Example: prefix **quux ab/** is returned as **quux+ab/** but should be **quux%20ab/**.
- `test_bucket_list_delimiter_empty` — When **Delimiter=''** is sent, listing behavior is correct (all keys returned, no CommonPrefixes), but the response incorrectly includes a Delimiter field. AWS omits Delimiter from the XML when the client passes an empty delimiter.
- `test_bucket_list_prefix_unreadable` — ListObjects with **Prefix='\x0a' (newline)** should echo the prefix in the response and return empty Contents/CommonPrefixes. Ozone may not preserve or echo the control-character prefix correctly.
https://ozone.s3.peterxcli.dev/#latest-run-section
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15600
## How was this patch tested?
**Before fix:**
1. encoding type:
```
bash-5.1$ aws s3api list-objects --bucket buck1 --encoding-type url --delimiter / --endpoint-url http://s3g:9878/
{
"Contents": [
{
"Key": "asdf%2Bb",
"LastModified": "2026-06-19T09:51:39.569Z",
"ETag": "\"f75b8179e4bbe7e2b4a074dcef62de95\"",
"Size": 8,
"StorageClass": "STANDARD",
"Owner": {
"DisplayName": "testuser",
"ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
}
}
],
"CommonPrefixes": [
{
"Prefix": "foo/"
},
{
"Prefix": "foo%2B1/"
},
{
"Prefix": "quux+ab/". <----------------- wrong output
}
],
"RequestCharged": null,
"Prefix": ""
}
```
2. empty delimiter:
```
bash-5.1$ aws --debug s3api list-objects \
--bucket buck1 \
--delimiter "" \
--endpoint-url http://s3g:9878 2>&1 \
| grep -oE '<Prefix>[^<]*</Prefix>|<Delimiter>[^<]*</Delimiter>|<KeyCount>[^<]*</KeyCount>'
<Prefix></Prefix>
<KeyCount>0</KeyCount>
<Delimiter></Delimiter>. <---------- should not be present
```
3 echoed prefix always url-encoded:
bash-5.1$ aws s3api list-objects \
--bucket buck1 \
--prefix $'\n' \
--endpoint-url http://s3g:9878/
{
"RequestCharged": null,
"Prefix": "%0A" <------------------ wrong behaviour
}
**After fix:**
1. encoding-type:
```
bash-5.1$ aws s3api list-objects --bucket buck1 --encoding-type url --delimiter / --endpoint-url http://s3g:9878/
{
"Contents": [
{
"Key": "asdf%2Bb",
"LastModified": "2026-06-19T09:08:58.933Z",
"ETag": "\"f75b8179e4bbe7e2b4a074dcef62de95\"",
"Size": 8,
"StorageClass": "STANDARD",
"Owner": {
"DisplayName": "testuser",
"ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
}
}
],
"CommonPrefixes": [
{
"Prefix": "foo/"
},
{
"Prefix": "foo%2B1/"
},
{
"Prefix": "quux%20ab/" <------------------------correct output
}
],
"RequestCharged": null,
"Prefix": null
}
```
2. empty delimeter :
```
bash-5.1$ aws --debug s3api list-objects \
--bucket buck1 \
--delimiter "" \
--endpoint-url http://s3g:9878 2>&1 \
| grep -oE '<Prefix>[^<]*</Prefix>|<Delimiter>[^<]*</Delimiter>|<KeyCount>[^<]*</KeyCount>'
<KeyCount>0</KeyCount>. <-------------------- no empty delimiter present
```
3. echoed prefix should not be url encoded:
```
bash-5.1$ aws s3api list-objects \
--bucket buck1 \
--prefix $'\n' \
--endpoint-url http://s3g:9878/
{
"RequestCharged": null,
"Prefix": "\n"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on PR #10586:
URL: https://github.com/apache/ozone/pull/10586#issuecomment-4966050748
@Russole Please take a look, I have updated the Pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3490105640
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -62,6 +62,17 @@ public static String urlEncode(String str)
return URLEncoder.encode(str, UTF_8.name());
}
+ /**
+ * Percent-encode a string for S3 {@code encoding-type=url} responses.
+ *
+ * <p>Unlike {@link URLEncoder} (application/x-www-form-urlencoded), AWS S3
+ * uses percent-encoding where spaces are {@code %20}, not {@code +}.
+ */
+ public static String s3urlEncode(String str)
+ throws UnsupportedEncodingException {
+ return urlEncode(str).replace("+", "%20");
+ }
Review Comment:
@adoroszlai I need to determine whether `PercentCodec` produces the same encoding semantics that AWS S3 expects. Currently I am gong through some of the documentation. If there is no issue then I will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3503402269
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java:
##########
@@ -157,17 +159,21 @@ Response handleGetRequest(S3RequestContext context, String bucketName) throws IO
if (encodingType != null && !encodingType.equals(ENCODING_TYPE)) {
throw S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, encodingType);
}
-
// If you specify the encoding-type request parameter,should return
- // encoded key name values in the following response elements:
- // Delimiter, Prefix, Key, and StartAfter.
+ // encoded key name values in the following response elements: Delimiter,
+ // Key, and StartAfter. The echoed Prefix request parameter is returned without URL encoding.
//
// For detail refer:
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
ListObjectResponse response = new ListObjectResponse();
- response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ // AWS omits Delimiter from the response when the client passes delimiter= or does not specify delimiter at all.
+ if (StringUtils.isNotEmpty(delimiter)) {
+ response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ }
response.setName(bucketName);
- response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));
+ if (prefixSpecified) {
+ response.setPrefix(EncodingTypeObject.createNullable(prefix, null));
+ }
Review Comment:
I wonder how to fix this part when encoding-type is not mentioned in the request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Russole (via GitHub)" <gi...@apache.org>.
Russole commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3499999214
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java:
##########
@@ -157,17 +159,21 @@ Response handleGetRequest(S3RequestContext context, String bucketName) throws IO
if (encodingType != null && !encodingType.equals(ENCODING_TYPE)) {
throw S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, encodingType);
}
-
// If you specify the encoding-type request parameter,should return
- // encoded key name values in the following response elements:
- // Delimiter, Prefix, Key, and StartAfter.
+ // encoded key name values in the following response elements: Delimiter,
+ // Key, and StartAfter. The echoed Prefix request parameter is returned without URL encoding.
//
// For detail refer:
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
ListObjectResponse response = new ListObjectResponse();
- response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ // AWS omits Delimiter from the response when the client passes delimiter= or does not specify delimiter at all.
+ if (StringUtils.isNotEmpty(delimiter)) {
+ response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ }
response.setName(bucketName);
- response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));
+ if (prefixSpecified) {
+ response.setPrefix(EncodingTypeObject.createNullable(prefix, null));
+ }
Review Comment:
`Prefix` should also be encoded when `encoding-type=url` is specified.
AWS docs say `EncodingType` applies to `Delimiter`, `Prefix`, `Key`, and `StartAfter`:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
Local test:
```bash
aws s3api list-objects \
--bucket buck1 \
--prefix $'\n' \
--encoding-type url \
--endpoint-url http://localhost:9878/
```
Actual:
```json
"Prefix": "\n"
```
Expected:
```json
"Prefix": "%0A"
```
Looks like `Prefix` is created with a `null` encoding type here. Should this use `encodingType` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on PR #10586:
URL: https://github.com/apache/ozone/pull/10586#issuecomment-4976923119
Thanks @adoroszlai , @Russole and @chungen0126 for reviewing the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3457986688
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -62,6 +62,17 @@ public static String urlEncode(String str)
return URLEncoder.encode(str, UTF_8.name());
}
+ /**
+ * Percent-encode a string for S3 {@code encoding-type=url} responses.
+ *
+ * <p>Unlike {@link URLEncoder} (application/x-www-form-urlencoded), AWS S3
+ * uses percent-encoding where spaces are {@code %20}, not {@code +}.
+ */
+ public static String s3urlEncode(String str)
+ throws UnsupportedEncodingException {
+ return urlEncode(str).replace("+", "%20");
+ }
Review Comment:
Can we use `PercentCodec` from `commons-codec`?
https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/net/PercentCodec.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on PR #10586:
URL: https://github.com/apache/ozone/pull/10586#issuecomment-4840734937
@ivandika3 please review this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 merged PR #10586:
URL: https://github.com/apache/ozone/pull/10586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3492994265
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -62,6 +62,17 @@ public static String urlEncode(String str)
return URLEncoder.encode(str, UTF_8.name());
}
+ /**
+ * Percent-encode a string for S3 {@code encoding-type=url} responses.
+ *
+ * <p>Unlike {@link URLEncoder} (application/x-www-form-urlencoded), AWS S3
+ * uses percent-encoding where spaces are {@code %20}, not {@code +}.
+ */
+ public static String s3urlEncode(String str)
+ throws UnsupportedEncodingException {
+ return urlEncode(str).replace("+", "%20");
+ }
Review Comment:
Thanks for checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3568285728
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -1095,11 +1095,11 @@ public ListKeysResult listKeys(String volumeName, String bucketName,
} else {
// This allows us to seek directly to the first key with the right prefix.
seekKey = getOzoneKey(volumeName, bucketName,
- StringUtils.isNotBlank(keyPrefix) ? keyPrefix : OM_KEY_PREFIX);
+ StringUtils.isNotEmpty(keyPrefix) ? keyPrefix : OM_KEY_PREFIX);
Review Comment:
Thanks for pointing out this. Added the test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3507364293
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java:
##########
@@ -157,17 +159,21 @@ Response handleGetRequest(S3RequestContext context, String bucketName) throws IO
if (encodingType != null && !encodingType.equals(ENCODING_TYPE)) {
throw S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, encodingType);
}
-
// If you specify the encoding-type request parameter,should return
- // encoded key name values in the following response elements:
- // Delimiter, Prefix, Key, and StartAfter.
+ // encoded key name values in the following response elements: Delimiter,
+ // Key, and StartAfter. The echoed Prefix request parameter is returned without URL encoding.
//
// For detail refer:
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
ListObjectResponse response = new ListObjectResponse();
- response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ // AWS omits Delimiter from the response when the client passes delimiter= or does not specify delimiter at all.
+ if (StringUtils.isNotEmpty(delimiter)) {
+ response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ }
response.setName(bucketName);
- response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));
+ if (prefixSpecified) {
+ response.setPrefix(EncodingTypeObject.createNullable(prefix, null));
+ }
Review Comment:
but both the cases are not handled properly in case of null encoding-type its giving **%0A**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3507709386
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java:
##########
@@ -157,17 +159,21 @@ Response handleGetRequest(S3RequestContext context, String bucketName) throws IO
if (encodingType != null && !encodingType.equals(ENCODING_TYPE)) {
throw S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, encodingType);
}
-
// If you specify the encoding-type request parameter,should return
- // encoded key name values in the following response elements:
- // Delimiter, Prefix, Key, and StartAfter.
+ // encoded key name values in the following response elements: Delimiter,
+ // Key, and StartAfter. The echoed Prefix request parameter is returned without URL encoding.
//
// For detail refer:
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
ListObjectResponse response = new ListObjectResponse();
- response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ // AWS omits Delimiter from the response when the client passes delimiter= or does not specify delimiter at all.
+ if (StringUtils.isNotEmpty(delimiter)) {
+ response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ }
response.setName(bucketName);
- response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));
+ if (prefixSpecified) {
+ response.setPrefix(EncodingTypeObject.createNullable(prefix, null));
+ }
Review Comment:
To identify the root cause of this, I have added few logs around`response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));`
**with encoding-type**: correct response
```
bash-5.1$ aws s3api list-objects \
--bucket buck1 \
--prefix $'\n' \
--encoding-type url \
--endpoint-url http://s3g:9878/
{
"RequestCharged": null,
"Prefix": "%0A"
}
```
Related logs for this request:
```
prefixSpecified=true, prefix='\n', encodingType='url'
createNullable: value='\n', encodingType='url'
EncodingTypeObject ctor: value='\n', encodingType='url'
prefixObj.value='\n'
prefixObj.encodingType='url'
"GET /buck1?encoding-type=url&prefix=%0A HTTP/1.1" 200 304
```
**Without Encoding-Type**: output is wrong
```
bash-5.1$ aws s3api list-objects --bucket buck1 --prefix $'\n' --endpoint-url http://s3g:9878/
{
"RequestCharged": null,
"Prefix": "%0A"
}
```
Related logs to this commands:
```
prefixSpecified=true, prefix='\n', encodingType='url'
createNullable: value='\n', encodingType='url'
EncodingTypeObject ctor: value='\n', encodingType='url'
prefixObj.value='\n'
prefixObj.encodingType='url'
// the HTTP request logged by S3G is:
"GET /buck1?prefix=%0A&encoding-type=url HTTP/1.1" 200 304
```
This is the important finding:
The second request was sent without `--encoding-type url`, but S3G is processing a request that contains encoding-type=url.
Because of that:
- encodingType is never null.
- EncodingTypeObject.createNullable() is always invoked with "url".
- The response is therefore always URL-encoded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3491467391
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -62,6 +62,17 @@ public static String urlEncode(String str)
return URLEncoder.encode(str, UTF_8.name());
}
+ /**
+ * Percent-encode a string for S3 {@code encoding-type=url} responses.
+ *
+ * <p>Unlike {@link URLEncoder} (application/x-www-form-urlencoded), AWS S3
+ * uses percent-encoding where spaces are {@code %20}, not {@code +}.
+ */
+ public static String s3urlEncode(String str)
+ throws UnsupportedEncodingException {
+ return urlEncode(str).replace("+", "%20");
+ }
Review Comment:
@adoroszlai
On the basis of local testing I found that`PercentCodec` alone does not encode ASCII specials like **=,** which S3 requires. Using it correctly needs configuring **alwaysEncodeChars** to mirror URLEncoder's unsafe set. Since the only S3 difference from URLEncoder is %20 vs +, urlEncode(str).replace("+", "%20") is simpler and equivalent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3503388894
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java:
##########
@@ -157,17 +159,21 @@ Response handleGetRequest(S3RequestContext context, String bucketName) throws IO
if (encodingType != null && !encodingType.equals(ENCODING_TYPE)) {
throw S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, encodingType);
}
-
// If you specify the encoding-type request parameter,should return
- // encoded key name values in the following response elements:
- // Delimiter, Prefix, Key, and StartAfter.
+ // encoded key name values in the following response elements: Delimiter,
+ // Key, and StartAfter. The echoed Prefix request parameter is returned without URL encoding.
//
// For detail refer:
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
ListObjectResponse response = new ListObjectResponse();
- response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ // AWS omits Delimiter from the response when the client passes delimiter= or does not specify delimiter at all.
+ if (StringUtils.isNotEmpty(delimiter)) {
+ response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ }
response.setName(bucketName);
- response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));
+ if (prefixSpecified) {
+ response.setPrefix(EncodingTypeObject.createNullable(prefix, null));
+ }
Review Comment:
@Russole thanks for bringing this in view. Will revert the changes for this but now the problem is `test_bucket_list_prefix_unreadable` this test since the encoding type is not there so it should not encode it.
With original :
```
if (prefixSpecified) {
response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));
}
```
**Local test:**
```
bash-5.1$ aws s3api list-objects \
--bucket buck1 \
--prefix $'\n' \
--endpoint-url http://s3g:9878/
```
**Actual is:**
```
{
"RequestCharged": null,
"Prefix": "%0A" --------------------> it is encoded without encoding type mentioned in the request
}
```
**Expected :**
```
{
"RequestCharged": null,
"Prefix": "\n"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Russole (via GitHub)" <gi...@apache.org>.
Russole commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3507224613
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java:
##########
@@ -157,17 +159,21 @@ Response handleGetRequest(S3RequestContext context, String bucketName) throws IO
if (encodingType != null && !encodingType.equals(ENCODING_TYPE)) {
throw S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, encodingType);
}
-
// If you specify the encoding-type request parameter,should return
- // encoded key name values in the following response elements:
- // Delimiter, Prefix, Key, and StartAfter.
+ // encoded key name values in the following response elements: Delimiter,
+ // Key, and StartAfter. The echoed Prefix request parameter is returned without URL encoding.
//
// For detail refer:
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
ListObjectResponse response = new ListObjectResponse();
- response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ // AWS omits Delimiter from the response when the client passes delimiter= or does not specify delimiter at all.
+ if (StringUtils.isNotEmpty(delimiter)) {
+ response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ }
response.setName(bucketName);
- response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));
+ if (prefixSpecified) {
+ response.setPrefix(EncodingTypeObject.createNullable(prefix, null));
+ }
Review Comment:
If `queryParams().get(QueryParams.PREFIX)` already returns the decoded value `"\n"`, then no extra request-decoding fix is needed here.
Passing `encodingType` should handle both cases:
- when `encodingType == null`, `Prefix` is marshalled as `"\n"`
- when `encodingType == "url"`, `Prefix` is marshalled as `"%0A"`
So using `EncodingTypeObject.createNullable(prefix, encodingType)` should not encode `Prefix` when `encoding-type` is absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3510929187
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java:
##########
@@ -157,17 +159,21 @@ Response handleGetRequest(S3RequestContext context, String bucketName) throws IO
if (encodingType != null && !encodingType.equals(ENCODING_TYPE)) {
throw S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, encodingType);
}
-
// If you specify the encoding-type request parameter,should return
- // encoded key name values in the following response elements:
- // Delimiter, Prefix, Key, and StartAfter.
+ // encoded key name values in the following response elements: Delimiter,
+ // Key, and StartAfter. The echoed Prefix request parameter is returned without URL encoding.
//
// For detail refer:
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
ListObjectResponse response = new ListObjectResponse();
- response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ // AWS omits Delimiter from the response when the client passes delimiter= or does not specify delimiter at all.
+ if (StringUtils.isNotEmpty(delimiter)) {
+ response.setDelimiter(EncodingTypeObject.createNullable(delimiter, encodingType));
+ }
response.setName(bucketName);
- response.setPrefix(EncodingTypeObject.createNullable(prefix, encodingType));
+ if (prefixSpecified) {
+ response.setPrefix(EncodingTypeObject.createNullable(prefix, null));
+ }
Review Comment:
@Russole for now I have reverted the fix as this URL-encoded seems to be coming from aws s3 itself. Which is verified from the above experiment which I did locally. We can come up for this test fix later. Focusing this PR to the two fix only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10586:
URL: https://github.com/apache/ozone/pull/10586#discussion_r3458179991
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -62,6 +62,17 @@ public static String urlEncode(String str)
return URLEncoder.encode(str, UTF_8.name());
}
+ /**
+ * Percent-encode a string for S3 {@code encoding-type=url} responses.
+ *
+ * <p>Unlike {@link URLEncoder} (application/x-www-form-urlencoded), AWS S3
+ * uses percent-encoding where spaces are {@code %20}, not {@code +}.
+ */
+ public static String s3urlEncode(String str)
+ throws UnsupportedEncodingException {
+ return urlEncode(str).replace("+", "%20");
+ }
Review Comment:
Okay let me check and get back to u.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org