You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/12/25 17:02:02 UTC
[GitHub] [ozone] xichen01 opened a new pull request, #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter
xichen01 opened a new pull request, #4127:
URL: https://github.com/apache/ozone/pull/4127
## What changes were proposed in this pull request?
Support AWS s3 ListObjects API's encodingType request parameter
refer:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
current Ozone will encode `key` and `prefix` in the `ListObjects` response in the default, however, the AWS s3 has a request parameter of `encoding-type` that can control whether to encode fields such as key in the response
### Why need add this feature
The Ozone default encoding key can cause some bugs
For example, when using go-sdk `ListObjectsV2`, the encoded key is returned
this is the example code:
you can put some keys that contain special characters in the key name (such as "="), execute this code and check the return value, you will find that special characters are encoded. (If using AWS s3 key name will be returned correctly)
```golang
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
const (
AccessKeyID = "id"
SecretAccessKey = "key"
BucketRegion = "none"
BucketName = "bucket1"
Endpoint = "http://localhost:9878"
)
func ListItems(client *s3.S3, bucketName string, prefix string) (*s3.ListObjectsV2Output, error) {
res, err := client.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String(bucketName),
Prefix: aws.String(prefix),
})
if err != nil {
return nil, err
}
return res, nil
}
func main() {
// create session
sess := session.Must(session.NewSession(aws.NewConfig().
WithRegion(BucketRegion).
WithCredentials(credentials.NewStaticCredentials(AccessKeyID, SecretAccessKey, "")).
WithEndpoint(Endpoint).
WithS3ForcePathStyle(true)))
s3session := s3.New(sess)
prefixName := ""
bucketObjects, err := ListItems(s3session, BucketName, prefixName)
if err != nil {
fmt.Printf("Couldn't retrieve bucket items: %v", err)
return
}
for _, item := range bucketObjects.Contents {
fmt.Printf("Name: %s, Last Modified: %s\n", *item.Key, *item.LastModified)
}
}
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7710
## How was this patch tested?
- create a bucket and put some keys
```shell
[root@Linux /root/ozone-1.3.0-SNAPSHOT]% bin/ozone sh bucket create s3v/bucket1
[root@Linux /root/ozone-1.3.0-SNAPSHOT]% bin/ozone sh key put /s3v/bucket1/file=124 ~/testfile
[root@Linux /root/ozone-1.3.0-SNAPSHOT]% bin/ozone sh key put /s3v/bucket1/file==124 ~/testfile
[root@Linux /root/ozone-1.3.0-SNAPSHOT]
```
- Prepare a postman APP
- configure the postman APP's S3 Authorization
<img width="1260" alt="image" src="https://user-images.githubusercontent.com/32928346/209475686-a2840e47-f494-4a41-8610-16d05bd00b46.png">
- Create a `ListObjects` request using postman as shown below
If you specify the encoding-type request parameter, S3 includes this element in the response, and returns encoded key name values in the following response elements:
Get URL: `http://s3.localhost:9878/bucket1/?encoding-type=url&prefix=file=&start-after=file=&delimiter==&list-type=2`
<img width="974" alt="image" src="https://user-images.githubusercontent.com/32928346/209475244-e89e151a-bcce-4dea-98ed-3f2d687e4fae.png">
- Send the `ListObjects` request again, but without the `encodingtype` parameter
If you do not specify the encoding-type request parameter, Amazon S3 does not include this element in the response, and returns unencoded key name values in the following response elements:
Get URL: `http://s3.localhost:9878/bucket1/?prefix=file=&start-after=file=&delimiter==&list-type=2`
<img width="988" alt="image" src="https://user-images.githubusercontent.com/32928346/209475288-f76e4e46-d7e2-489f-b929-3bd121ecacf1.png">
- if the encoding-type request parameter is invalid
<img width="913" alt="image" src="https://user-images.githubusercontent.com/32928346/209475831-020d681f-16a3-41af-a3bb-bd74117e4a47.png">
### refer AWS s3's response:
I verified the AWS response using the postman
the key in the bucket
```shell
[root@VM-8-3-centos ~]$ aws s3 ls s3://pony-bucket1/
2022-12-23 15:08:39 2884 file=124
2022-12-23 15:19:23 234 file==hosts
[root@VM-8-3-centos ~]$
```
If you specify the encoding-type request parameter, Amazon S3 includes this element in the response, and returns encoded key name values in the following response elements:
`Delimiter`, `Prefix`, `Key`, and `StartAfter`.
<img width="1201" alt="image" src="https://user-images.githubusercontent.com/32928346/209474637-16961931-deb6-4924-b1e2-ec177e1e3629.png">
If you do not specify the encoding-type request parameter, Amazon S3 does not include this element in the response, and returns unencoded key name values in the following response elements:
<img width="1003" alt="image" src="https://user-images.githubusercontent.com/32928346/209474508-01d9898a-2436-42b4-8277-b1d3055fb60d.png">
if the encoding-type request parameter is invalid
<img width="929" alt="image" src="https://user-images.githubusercontent.com/32928346/209475771-fe2c4641-523e-4181-a7e5-1d98419fb3bd.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xichen01 commented on pull request #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #4127:
URL: https://github.com/apache/ozone/pull/4127#issuecomment-1419355073
@neils-dev @DaveTeng0 Another case that can reproduce this bug is using the PySpark to access the Ozone key that contains "=" in the key.
- connect the Ozone S3g
```bash
[root@VM-8-3-centos ~]$ bin/pyspark --conf spark.s3a.enabled=true --conf spark.hadoop.fs.s3a.path.style.access=true --conf spark.hadoop.fs.s3a.bucket.bucket1.access.key=id --conf spark.hadoop.fs.s3a.bucket.bucket1.secret.key=secret --conf spark.hadoop.fs.s3a.bucket.bucket1.endpoint=http://localhost:9878 --conf "spark.driver.extraClassPath=/root/hadoop-3.3.4/share/hadoop/tools/lib/*" --conf spark.hadoop.fs.s3a.change.detection.version.required=false --conf spark.hadoop.fs.s3a.change.detection.mode=none --conf spark.hadoop.fs.s3a.bucket.probe=0
```
- Read keys with "=" in the key name
```bash
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.3.1
/_/
SparkSession available as 'spark'.
>>> path = "s3a://bucket1/dt=2022"
>>> df = spark.read.text(path)
23/02/106 21:33:51 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/root/spark-3.3.1-bin-hadoop3/python/pyspark/sql/readwriter.py", line 421, in text
return self._df(self._jreader.text(self._spark._sc._jvm.PythonUtils.toSeq(paths)))
File "/root/spark-3.3.1-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
File "/root/spark-3.3.1-bin-hadoop3/python/pyspark/sql/utils.py", line 190, in deco
return f(*a, **kw)
File "/root/spark-3.3.1-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o47.text.
: java.io.FileNotFoundException: No such file or directory: s3a://bucket1/dt%3D2022
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:3300)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$20(S3AFileSystem.java:3264)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$21(S3AFileSystem.java:3263)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3262)
at org.apache.spark.util.HadoopFSUtils$.listLeafFiles(HadoopFSUtils.scala:225)
at org.apache.spark.util.HadoopFSUtils$.$anonfun$listLeafFiles$7(HadoopFSUtils.scala:281)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
at org.apache.spark.util.HadoopFSUtils$.listLeafFiles(HadoopFSUtils.scala:271)
at org.apache.spark.util.HadoopFSUtils$.$anonfun$parallelListLeafFilesInternal$1(HadoopFSUtils.scala:95)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.spark.util.HadoopFSUtils$.parallelListLeafFilesInternal(HadoopFSUtils.scala:85)
at org.apache.spark.util.HadoopFSUtils$.parallelListLeafFiles(HadoopFSUtils.scala:69)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.bulkListLeafFiles(InMemoryFileIndex.scala:158)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.listLeafFiles(InMemoryFileIndex.scala:131)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.refresh0(InMemoryFileIndex.scala:94)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.<init>(InMemoryFileIndex.scala:66)
at org.apache.spark.sql.execution.datasources.DataSource.createInMemoryFileIndex(DataSource.scala:567)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:409)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
```
Note the exception message , the root cause is `: java.io.FileNotFoundException: No such file or directory: s3a://bucket1/dt%3D2022
`, the root cause is the "=" not being handled correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on pull request #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter
Posted by GitBox <gi...@apache.org>.
neils-dev commented on PR #4127:
URL: https://github.com/apache/ozone/pull/4127#issuecomment-1381248031
Thanks @xichen01 for filing this jira and this well described PR.
I have a general comment. Using the current ozone codebase from the master, I tried your example illustrating the need for the patch with the aws cli as follows:
```
$ aws s3api --endpoint http://localhost:9878 create-bucket --bucket=wordcount
$ aws s3api --endpoint http://localhost:9878 put-object --bucket=wordcount --key file=123 ./README.md
```
I expect to see the `list-object-v2` return just "_file=123_" for both _encoding=default_ and _encoding=url_, however when encoding is specified, the response is indeed encoded.
```
$ aws s3api --endpoint http://localhost:9878 list-objects-v2 --bucket wordcount --encoding-type url
{
"Key": "file%3D123",
"LastModified": "2023-01-13T01:33:38.697000+00:00",
"ETag": "2023-01-13T01:33:38.697Z",
"Size": 4263,
"StorageClass": "STANDARD"
},
```
It seems to encode the response as expect with aws-cli. Have you seen the same? Does that not satisfy what aws supports with https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #4127:
URL: https://github.com/apache/ozone/pull/4127#issuecomment-1370460456
Thank you @xichen01 for working on this.
@duongkame @aswinshakil @neils-dev can you'll please help review this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xichen01 commented on pull request #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter
Posted by GitBox <gi...@apache.org>.
xichen01 commented on PR #4127:
URL: https://github.com/apache/ozone/pull/4127#issuecomment-1382129399
> Thanks @xichen01 for filing this jira and this well described PR. I have a general comment. Using the current ozone codebase from the master, I tried your example illustrating the need for the patch with the aws cli as follows:
>
> ```
> $ aws s3api --endpoint http://localhost:9878 create-bucket --bucket=wordcount
> $ aws s3api --endpoint http://localhost:9878 put-object --bucket=wordcount --key file=123 ./README.md
> ```
>
> I expect to see the `list-object-v2` return just "_file=123_" for both _encoding=default_ and _encoding=url_, however when encoding is specified as `url`, the response is indeed encoded.
>
> no encoding
>
> ```
> $ aws s3api --endpoint http://localhost:9878 list-objects-v2 --bucket wordcount
> {
> "Key": "file=123",
> "LastModified": "2023-01-13T01:33:38.697000+00:00",
> "ETag": "2023-01-13T01:33:38.697Z",
> "Size": 4263,
> "StorageClass": "STANDARD"
> },
> ```
>
> encoding=url
>
> ```
> $ aws s3api --endpoint http://localhost:9878 list-objects-v2 --bucket wordcount --encoding-type url
> {
> "Key": "file%3D123",
> "LastModified": "2023-01-13T01:33:38.697000+00:00",
> "ETag": "2023-01-13T01:33:38.697Z",
> "Size": 4263,
> "StorageClass": "STANDARD"
> },
> ```
>
> It seems to encode the response as expect with aws-cli. Have you seen the same? Does that not satisfy what aws supports with https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html ?
This issue can not be reproduced in the AWS client, I found two scenarios that would be reproduced, one using go-SDK, and the other using spark to access Ozone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter
Posted by "DaveTeng0 (via GitHub)" <gi...@apache.org>.
DaveTeng0 commented on PR #4127:
URL: https://github.com/apache/ozone/pull/4127#issuecomment-1418589022
> > Thanks @xichen01 for filing this jira and this well described PR. I have a general comment. Using the current ozone codebase from the master, I tried your example illustrating the need for the patch with the aws cli as follows:
> > ```
> > $ aws s3api --endpoint http://localhost:9878 create-bucket --bucket=wordcount
> > $ aws s3api --endpoint http://localhost:9878 put-object --bucket=wordcount --key file=123 ./README.md
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > I expect to see the `list-object-v2` return just "_file=123_" for both _encoding=default_ and _encoding=url_, however when encoding is specified as `url`, the response is indeed encoded.
> > no encoding
> > ```
> > $ aws s3api --endpoint http://localhost:9878 list-objects-v2 --bucket wordcount
> > {
> > "Key": "file=123",
> > "LastModified": "2023-01-13T01:33:38.697000+00:00",
> > "ETag": "2023-01-13T01:33:38.697Z",
> > "Size": 4263,
> > "StorageClass": "STANDARD"
> > },
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > encoding=url
> > ```
> > $ aws s3api --endpoint http://localhost:9878 list-objects-v2 --bucket wordcount --encoding-type url
> > {
> > "Key": "file%3D123",
> > "LastModified": "2023-01-13T01:33:38.697000+00:00",
> > "ETag": "2023-01-13T01:33:38.697Z",
> > "Size": 4263,
> > "StorageClass": "STANDARD"
> > },
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > It seems to encode the response as expect with aws-cli. Have you seen the same? Does that not satisfy what aws supports with https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html ?
>
> This issue can not be reproduced in the AWS client, I found two scenarios that would be reproduced, one using go-SDK, and the other using spark to access Ozone.
Interesting~ Maybe there's some encoding/decoding logic in the s3 gateway?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4127:
URL: https://github.com/apache/ozone/pull/4127#issuecomment-1437204367
Thanks @xichen01 for the patch, @neils-dev for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #4127:
URL: https://github.com/apache/ozone/pull/4127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org